GPA: Learning GUI Process Automation from Demonstrations

🎯 Pitch

Record a GUI task once and replay it with 100% reliability—no coding, no brittle selectors, and no cloud API calls. GPA replaces the randomness of VLM agents with a particle-filtering geometric matcher, achieving a 10× speedup over Gemini 3 Pro while running fully on-device.

1. Executive Summary

This paper introduces GUI Process Automation (GPA), a record-and-replay system that learns deterministic GUI workflows from a single user demonstration and replays them using robust visual graph matching rather than generative AI. The core technical mechanism is Sequential Monte Carlo localization (a particle-filtering procedure that infers a target element's position by jointly reasoning over the target's appearance and the geometric relationships of its neighboring nodes in a UI graph), combined with readiness calibration (a gating mechanism that executes actions only when a statistically calibrated confidence score exceeds a threshold, preventing the system from acting on ambiguous or low-quality matches) and fully local execution using lightweight models like IconCLIP and a finetuned UI detector. In a pilot study of 16 desktop GUI tasks against Gemini 3 Pro (with CUA tools), GPA achieves 100% success rate versus 89.38% for Gemini while running roughly 10× faster on average (33.74 s vs. 329.31 s), establishing that deterministic demonstration-based replay can outperform cloud-based VLM agents on structured GUI workflows — provided the task can be recorded once and replayed without requiring runtime reasoning.

2. Context and Motivation

The Core Problem: Bridging the Gap Between RPA and VLM Agents for GUI Automation

This paper addresses a specific and practically pressing gap: how to automate repetitive GUI workflows reliably, quickly, and privately, without the fragility of traditional Robotic Process Automation (RPA) scripts or the non-deterministic risks of vision language model (VLM)-based GUI agents. The problem sits at the intersection of enterprise productivity, computer vision, and probabilistic inference, and it is motivated by a genuine engineering tension that no existing solution resolves well.

The paper frames this tension directly in Table 1 and the opening of Section 1. On one side, traditional RPA is deterministic and fast — once scripts are written, they execute the same sequence of actions every time. But RPA imposes a heavy upfront implementation cost: developers must manually inspect application internals to define selectors (HTML id attributes, accessibility metadata, CSS paths) and hard-code logic for edge cases like loading delays or unexpected popups. Worse, these scripts are brittle under interface drift: a website redesign that changes a div hierarchy, or a screen resolution change that shifts pixel coordinates, can silently break a workflow that previously worked perfectly. This brittleness is well-documented in the RPA literature the paper cites — Eikebrokk and Olsen (2020) and Haerens and Mannaert (2020) both emphasize that coupling automation logic to application structure makes maintenance costly and evolution difficult.

On the other side, the emerging class of VLM-based GUI agents (Anthropic's Computer Use, OpenAI's Operator and CUA, Google's Gemini with computer-use tools, and open-source frameworks like UFO, OS-ATLAS, and Mobile-Agent) offer a different tradeoff. These agents interpret screenshots and high-level instructions, then generate actions autoregressively. They require no selector inspection or scripting: the user provides a natural-language goal, and the agent decides what to click, type, or scroll. This flexibility is appealing for workflows that span multiple applications, involve dynamic content, or require some degree of runtime judgment. However, the paper identifies three critical shortcomings that make VLM agents unsuitable for many enterprise settings:

Non-determinism. Because VLM agents generate actions via next-token prediction, they are inherently stochastic. The paper notes (Section 1): "The stochastic nature of probabilistic next-token prediction means that an agent may perform correctly nine times but hallucinate an action on the tenth." For mission-critical enterprise workflows — processing invoices, submitting regulatory filings, entering HR data — a 90% success rate is not "90% productive"; it is 10% manual intervention and audit overhead. The paper argues this uncertainty is "difficult to bound in practice," a claim that reflects the broader challenge of reliability guarantees for generative systems.
Latency and cost. Each action requires streaming a screenshot to a cloud API, running a large VLM inference, and waiting for the response. The paper's own measurements in Section 3 bear this out: Gemini 3 Pro averages 329 seconds for a 22-step workflow — roughly 15 seconds per action, most of which is network round-trip and VLM inference time. For interactive use or high-throughput batch processing, this latency accumulates to unacceptable levels.
Privacy. Streaming screenshots to external cloud providers means sensitive visual data — internal dashboards, customer records, financial software interfaces — leaves the organization's network. The paper flags this as a fundamental concern for enterprise adoption (Section 1, Table 1), and it is particularly acute in regulated industries (healthcare, finance, government) where data residency requirements may prohibit cloud transmission entirely.

The paper's central observation is that these two approaches sit at opposite extremes of a tradeoff curve — RPA gives you determinism and speed at the cost of rigidity and implementation effort; VLM agents give you flexibility and ease of use at the cost of reliability, latency, and privacy — and neither is satisfactory for the common case: repetitive, structured workflows that need to run reliably, quickly, and privately, but that also need to tolerate minor UI variations (window resizing, font changes, rendering differences) without breaking.

Why This Matters: The Practical Stakes

The problem matters for reasons that extend beyond academic interest in UI grounding. The paper's framing (Section 1, Table 1, and the conclusion in Section 5) makes clear that this is an enterprise infrastructure problem:

Productivity at scale. Organizations deploy RPA to automate millions of repetitive interactions with legacy applications that lack APIs. When these RPA scripts break due to UI updates, the cost is not just the immediate failure — it is the maintenance burden of diagnosing and repairing brittle selector logic, often by developers who did not write the original scripts. A system that retains RPA's determinism and speed while eliminating selector-level brittleness would substantially reduce this total cost of ownership.
The reliability-requirement gap for VLM agents. Even as VLM capabilities improve, the paper argues (Section 1, implicitly) that probabilistic action generation is fundamentally the wrong mechanism for replaying deterministic workflows. If the user has demonstrated exactly what to do — click this button, type this text, press this hotkey — then the execution engine should not need to "reason" about what to click at runtime. It should locate the previously-demonstrated element and act. Injecting generative uncertainty into a deterministic replay task creates risk without adding value.
Privacy and compliance. The paper's emphasis on fully local execution (Table 1, Section 1) is not merely a performance optimization — it is a deployment requirement for many potential users. Government agencies, financial institutions, and healthcare organizations cannot legally route screenshots of internal applications through external APIs. A system that runs entirely on-device, using only lightweight local models, eliminates this barrier entirely.
Latency as a user-experience constraint. For desktop automation, the difference between sub-second action execution (GPA) and multi-second VLM inference per action (Gemini) is the difference between an automation that feels instantaneous — like a keyboard shortcut — and one that feels sluggish and intrusive. The paper's 10× speedup (33.74 s vs. 329.31 s on 22-step workflows) is not just a benchmark statistic; it represents the perceptual threshold between acceptable and unacceptable automation latency.

Prior Approaches and Their Shortcomings

The paper situates itself against a well-mapped landscape of prior work (Section 4, Related Work). The shortcomings it identifies are concrete and motivate specific design choices in GPA.

Traditional RPA: Fragile Under UI Drift

Classical RPA systems (UiPath, Automation Anywhere, Blue Prism) operate by identifying UI elements through application-level metadata: HTML id attributes, CSS selectors, accessibility tree paths, window handle hierarchies. The paper cites Eikebrokk and Olsen (2020) and Haerens and Mannaert (2020) to establish the known limitations: "coupling automation logic to application structure or surface behavior makes such systems brittle under software evolution and interface changes." When a website redesign changes a div structure, or a desktop application update renumbers window handles, RPA selectors break silently. Fixing them requires a developer to re-inspect the application and update the selector logic — a maintenance cost that compounds across large automation portfolios.

The paper notes that earlier work attempted to reduce this brittleness through visual GUI testing and automation (Section 4). Chang et al. (2010) introduced screenshot matching for GUI testing, replacing DOM-based selectors with pixel-level correspondence. Alégroth et al. (2015) surveyed visual GUI testing in practice. These approaches eliminated dependency on application internals by matching screenshots directly. However, the paper identifies their core limitation: "these methods still rely on low-level visual correspondence and offer limited semantic understanding, which leaves them sensitive to appearance changes and ambiguous interface states." If a button changes color or a surrounding text label shifts slightly, pixel-level matching degrades. Visual testing tools lack the structured geometric context — knowing that a checkbox sits to the left of the "Edit" label, regardless of exact pixel coordinates — that would make matching robust against common visual variations.

The paper's innovation is not to abandon visual matching, but to elevate it from pixel-level to graph-level: represent the UI as a graph of detected elements with spatial relationships, then match subgraphs rather than templates. This preserves the determinism of RPA while gaining robustness against the visual variations that break pixel-level matching.

VLM-Based GUI Agents: Non-Deterministic and Unbounded

The paper provides an extensive survey of the rapidly growing VLM GUI agent literature (Section 4, "Foundation-model GUI agents"). It traces the evolution from early web agents like Mind2Web (Deng et al., 2023) and WebArena (Zhou et al., 2024), through end-to-end web navigators like WebVoyager (He et al., 2024) and AutoWebGLM (Lai et al., 2024), to full computer-use systems spanning desktop, mobile, and cross-application tasks (OSWorld, AndroidWorld, WindowsAgentArena, ScreenSpot-Pro). It catalogues system-level agents (CogAgent, Ferret-UI, AppAgent, Mobile-Agent, UFO) and increasingly capable execution frameworks (AutoGLM, Agent S/S2, UI-TARS, PC-Agent, UFO2), as well as productized offerings (Anthropic Computer Use, OpenAI Operator/CUA, Google Project Mariner/Gemini Computer Use).

The paper's critique is not that these systems fail to work — many achieve impressive benchmark results — but that their core mechanism is mismatched to the task of replaying known workflows:

"Their reliance on generative probability introduces a fundamental flaw for mission-critical workflows: they are non-deterministic."

This is a subtle but important distinction. VLM agents are designed for open-ended task completion: given a goal ("book a flight to Chicago next Tuesday"), they must reason about which website to use, which fields to fill, which buttons to click, and in what order. That reasoning task genuinely requires generative flexibility — the sequence of actions is not predetermined. But when the task is to replay a demonstrated workflow ("click the same Compose button, type in the same recipient field, press the same send hotkey"), autoregressive action generation is overkill. The agent can still make mistakes — misidentifying a button, clicking an adjacent element, generating an action that doesn't correspond to any demonstrated step — but gains no benefit from its reasoning capability, because the reasoning has already been done (by the human demonstrator). The paper's framing (Table 1, Section 5 conclusion) implies that for replay tasks, deterministic matching is both more reliable and more efficient than generative reasoning.

The paper further notes two secondary shortcomings of VLM agents that compound the reliability concern:

Limited controllability. "The same high-level instruction can yield different action sequences across runs, making it hard to audit, constrain, or predict behaviour." For enterprise workflows that must comply with documented procedures, this unpredictability is a compliance risk — you cannot guarantee that the agent followed the approved process, because you cannot predict exactly what sequence of actions it will take.
High latency and privacy risks (covered above in "Why This Matters"). These are practical deployment barriers that affect adoption independently of accuracy.

Demonstration-Based GUI Automation: The Missing Piece

The paper identifies a third strand of prior work that is closest in spirit to GPA: learning GUI tasks from demonstrations (Section 4, "Learning from demonstration"). Early work by Intharah et al. (2017) with HILC showed that users could teach GUI tasks through demonstration, with follow-up questions resolving ambiguity. More recently, LearnAct (Liu et al., 2025) introduced a demonstration-based framework for mobile GUI agents with the LearnGUI benchmark, using dedicated modules to parse demonstrations, retrieve relevant prior experience, and execute actions in new contexts. Instruction Agent (Li et al., 2025) similarly leverages a single expert demonstration to extract step-by-step instructions and constrains execution with verification and backtracking.

The paper acknowledges this lineage but does not position GPA as a direct improvement over LearnAct or Instruction Agent. Rather, GPA represents a different design point on the demonstration-based spectrum. The approaches cited in the related work still embed VLM reasoning in their execution pipeline — they parse demonstrations into instructions, then use generative models to decide how to follow those instructions in new contexts. GPA takes a more radical approach: eliminate the generative model from the execution path entirely. Instead of "parse the demo, then reason about what to do," GPA's philosophy is "parse the demo, then match."

This is why the paper emphasizes that GPA "does not have reasoning or decision-making capabilities" (Section 5, Limitations) — it is a deliberate design constraint, not an oversight. By removing generative reasoning from the execution loop, GPA eliminates the primary source of non-determinism. The cost, which the paper acknowledges, is that GPA cannot adapt to situations requiring judgment (e.g., calendar date selection where the current month state determines how many clicks are needed). The paper frames this as an acceptable tradeoff for the target use case: structured, repetitive enterprise workflows where the human demonstrator has already made all necessary decisions.

How GPA Positions Itself

The paper positions GPA not as a competitor to VLM agents in general, but as a complementary tool for a specific, high-value subset of GUI automation tasks. This is reflected in several design and framing choices:

Design philosophy: "record once, replay reliably." GPA's architecture (Sections 2.1–2.3) is built around the assumption that the task can be demonstrated once and replayed deterministically. The technical challenge is not deciding what to do (the demonstration encodes that), but where to do it under visual variation. This reframes the problem from "autonomous GUI agent" to "robust visual localization for replay" — a narrower, more tractable problem that admits deterministic solutions.

Determinism as a feature, not a limitation. Where VLM agent papers emphasize flexibility and generalization, GPA emphasizes reliability, privacy, and speed (Table 1). The paper's repeated use of phrases like "deterministic, reliable replay," "safeguarded by readiness calibration," and "guarantee that sensitive visual data never leaves the local machine" signals that these are the primary value propositions, not secondary benefits. The paper is arguing that for a large class of enterprise workflows, these properties matter more than generative flexibility.

The MCP/CLI integration foresight. Section 1 and Section 5 conclude with an architectural vision: GPA can serve as an MCP (Model Context Protocol) or CLI tool for VLM agents, where "the agent only reasons and orchestrates while GPA handles the GUI execution." This positioning is strategically important: it suggests that GPA is not trying to replace VLM agents, but to become the execution substrate that gives them deterministic reliability for the GUI interaction parts of their workflows. The agent decides what to do (high-level planning, handling exceptions, deciding between workflows), and GPA handles how to do it (robustly clicking the right button every time).

The SMC formulation as a principled response to uncertainty. The paper's technical centerpiece (Section 2.3, Appendix A) — Sequential Monte Carlo localization with neighbor context — is a direct response to the brittleness of prior approaches. Where traditional RPA would fail if a target element's selector changed, and where VLM agents might hallucinate if the target is visually ambiguous, GPA explicitly models uncertainty through a particle filter over the target's location and scale. The SMC procedure doesn't just return a best guess; it returns a calibrated confidence score that gates execution. This is the paper's answer to the question "how do you achieve RPA-style determinism without the fragility?": by using probabilistic inference to estimate location, but then applying a strict deterministic threshold before acting.

Limitations as honest scoping. The paper's deliberate enumeration of limitations in Section 5 — no reasoning capability, date pickers only work for the demonstrated date, cannot adapt to situations requiring judgment — serves to define the boundary of the approach's applicability. This is not self-deprecation; it is a precise scoping claim: within this boundary, GPA is the right tool; outside it, use something else (possibly a VLM agent, possibly a hybrid system). The MCP/CLI integration idea makes this boundary operational — the VLM agent handles the "outside" cases, and GPA handles the reliable replay of known interaction patterns.

In summary, the paper addresses a gap that is simultaneously technical (how to robustly ground UI elements under visual variation without application-level metadata), practical (how to automate enterprise GUI workflows reliably, quickly, and privately), and architectural (how to combine the determinism of RPA with the visual robustness of computer vision, without incurring the non-determinism of generative AI). It positions GPA as a purpose-built solution for the common case of structured replay tasks, with an explicit design philosophy that determinism, speed, and privacy are more valuable than generative flexibility when the task is already known.

3. Technical Approach

3.1 Reader Orientation

GPA is a record-and-replay system that watches a user perform a GUI task once, compiles that demonstration into a structured workflow with a graph of UI elements for each step, and then re-executes that workflow deterministically by matching the recorded UI graphs against the live screen. The core technical problem it solves is robust UI grounding under visual variation — given a recorded screenshot showing "the checkbox next to the word Edit," how do you locate that same checkbox on a new screenshot where the window may have been resized, fonts may render differently, or some elements may look slightly different? The solution's shape is a two-phase pipeline: a demonstration phase that builds a reusable workflow template from a single recording, and an execution phase that uses particle-filter-based localization (Sequential Monte Carlo) with geometric context from neighboring elements to robustly locate each target, gated by a statistically calibrated confidence check that prevents the system from acting on uncertain matches.

3.2 Big-Picture Architecture (Diagram in Words)

GPA has five major components operating across two phases:

Demonstration Phase (build-time):

UI Parser — a finetuned icon detector (based on OmniParser) plus OCR that extracts all interactive elements from each screenshot, producing bounding boxes, text content, and icon embeddings.
UI Graph Builder — connects extracted elements via k-nearest-neighbors (k=5) based on spatial proximity, creating a graph where edges represent "these elements are close together on screen."
Workflow Compiler — for each demonstrated action, identifies which detected element the user clicked on (the target), extracts that target plus its neighboring nodes into a step subgraph, and (post-recording) uses an LLM to assign natural-language descriptions, extract parameterizable variables, and generate the workflow template.

Execution Phase (runtime): 4. SMC Retriever — the core localization engine. Given a step subgraph from the demo, it runs a particle filter (Sequential Monte Carlo sampler) that jointly reasons over the target element's appearance and the geometric arrangement of its neighbors to estimate where the target is on the current screen, outputting a predicted location and confidence score. 5. Finite State Machine (FSM) Controller — orchestrates execution: for each step, it parses the current screen into a UI graph, invokes the SMC retriever, checks the readiness confidence score, and decides whether to execute, retry (after re-observing the screen), or fail the step. A precheck pipeline speculatively processes upcoming steps in a background thread while the environment settles after each action.

Information flows as follows: a user demonstration is recorded as (screenshot, action) pairs → the UI Parser extracts elements from each screenshot → the Graph Builder connects them → the Workflow Compiler identifies targets and neighbors, producing a workflow template. At runtime, the FSM Controller steps through the template → for each step, it captures a new screenshot → the UI Parser and Graph Builder produce a fresh runtime graph → the SMC Retriever matches the demo subgraph against the runtime graph → the confidence score gates execution → the action is dispatched to the environment.

3.3 Roadmap for the Deep Dive

First, the UI graph representation and similarity computation, because every subsequent mechanism — from fast-path matching to SMC localization to readiness checking — depends on understanding what a node is, what features it carries, and how similarity between nodes is measured across screens.
Second, the demonstration phase and workflow building, because this explains how the recorded interaction gets converted into the structured step subgraphs that the execution phase matches against.
Third, the SMC localization procedure, because this is the paper's central technical contribution: how particle filtering with geometric context from neighbor nodes achieves robust grounding under rescaling and detection uncertainty.
Fourth, readiness checking and confidence calibration, because this is the mechanism that converts the SMC's probabilistic output into the deterministic reliability guarantee that distinguishes GPA from VLM agents.
Fifth, the execution control FSM and precheck pipeline, because this shows how the localization and confidence components are orchestrated into a complete step-by-step replay engine with bounded retries and speculative preprocessing.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems paper whose core idea is that GUI replay tasks can be solved deterministically and robustly by representing UI states as graphs, formulating grounding as a subgraph-matching problem under geometric constraints, and using particle filtering (Sequential Monte Carlo) to jointly infer target location and window-scale parameters from both target-appearance and neighbor-geometry evidence, with a calibrated confidence gate preventing execution on ambiguous matches.

The UI Graph: Nodes, Features, Edges, and Similarity

What goes into a node. Every detected UI element becomes a graph node $v$ storing three pieces of information: a bounding box $b_v = [x, y, w, h]$ (pixel coordinates and dimensions), extracted OCR text $t_v$ (if the element has readable text — a button label, a field name, a menu item), and an icon embedding $e_v \in \mathbb{R}^{512}$ produced by IconCLIP (ViT-B-32, a CLIP variant fine-tuned for icon recognition). The paper uses a finetuned version of OmniParser for icon detection, meaning the system can identify non-text interactive elements (buttons, checkboxes, dropdown arrows, icons) that OCR alone would miss. OCR handles text extraction. Together, these two detectors produce a comprehensive set of UI elements — text labels, input fields, buttons with text, and purely visual icons.

How edges are formed. After extracting all elements from a screenshot, the system builds a graph $G = (V, E)$ by connecting each element to its $k = 5$ nearest neighbors based on Euclidean distance between element centers. This produces a spatial proximity graph — edges represent "these elements are physically close on screen," not semantic relationships. The paper explicitly notes this: "edges represent spatial proximity rather than semantic relationships." This design choice is important for robustness: spatial relationships are largely invariant to the kinds of visual changes that break semantic selectors (a button remains next to its label even if the button's CSS class changes or the label's font renders differently).

How node similarity is computed. The system uses a hybrid approach that differentiates between textual and non-textual elements, defined in Appendix B.2. For textual elements (buttons with text labels, text fields, menu items), similarity is a weighted combination:

$s(v_{\text{demo}}, v_{\text{candidate}}) = 0.9 \cdot s_{\text{text}}(v_{\text{demo}}, v_{\text{candidate}}) + 0.1 \cdot \cos(e_{\text{demo}}, e_{\text{candidate}})$

where $s_{\text{text}}$ is a fuzzy string matching score using the Levenshtein distance ratio (implemented via RapidFuzz) with length-adaptive tolerance that provides more forgiveness for OCR errors in short strings, returning 1.0 for exact matches and smoothly degrading for partial matches; and $\cos(e_{\text{demo}}, e_{\text{candidate}})$ is the cosine similarity between the two IconCLIP icon embeddings. The 0.9/0.1 weighting reflects that text is the primary matching signal when available, with icon similarity providing a small robustness boost against OCR errors.

For non-textual elements (pure icons, images with no extractable text), similarity is purely visual: $s(v_{\text{demo}}, v_{\text{candidate}}) = \cos(e_{\text{demo}}, e_{\text{candidate}})$ .

Why this design. Fuzzy string matching with adaptive tolerance handles OCR noise — a "Compose" button might be read as "Compose" on one run and "Compos" on another, and the Levenshtein ratio gracefully degrades rather than producing a hard 0/1 mismatch. IconCLIP embeddings provide a fallback signal when OCR is unavailable (purely visual elements) and a robustness boost when OCR is available but potentially noisy. The combination means GPA does not require pixels to match exactly (as in template matching) or metadata to remain stable (as in RPA selectors) — it only requires that the same element looks and reads similarly enough to its recorded version.

Demonstration Phase: From User Interaction to Workflow Template

Recording. During the demonstration phase, the user performs the task once while the system captures each action (click, type, hotkey, scroll) and the corresponding screenshot as a keyframe. The paper emphasizes that this is a single demonstration — the user does not need to repeat the task or provide multiple examples. A background preprocessing thread immediately parses each incoming screenshot into a UI graph as keyframes arrive, overlapping processing with the user's natural interaction so that "the workflow is ready to compile as soon as the recording ends."

Target identification. For each step, GPA identifies which detected UI element the user interacted with by checking which element's bounding box contains the recorded click coordinates. This becomes the target node $v_{\text{target}}$ for that step. The system then selects nearby nodes using k-nearest-neighbor graph traversal starting from the target, forming a step subgraph $G_d$ consisting of the target node plus $\{v_i\}_{i=1}^{M}$ neighbor nodes with their bounding boxes, text, and icon embeddings.

Scroll handling. For steps following a scroll action, the screenshot is preprocessed to mask regions outside the scrollable container before node matching. This is a subtle but important detail: if the user scrolled down to find a button, the neighbor nodes in the recorded screenshot are drawn from the scrolled viewport, not from the entire application window. Masking ensures that the step subgraph only contains elements from within the relevant scrollable area, preventing the system from matching against unrelated elements outside the container.

LLM post-processing. After recording completes, an LLM analyzes the full sequence of steps (screenshots + actions) to perform three tasks: (1) assign a natural-language description to each step (e.g., "Click on Compose button"), (2) identify parameterizable fields (e.g., email addresses, subject lines, form values) and extract them as typed workflow variables with default placeholder values, and (3) generate a workflow name and title. The output is a workflow template — a structured YAML file containing metadata, variable definitions, and an ordered list of steps, each with an action type, description, and reference to its step subgraph data stored in a companion JSON file. Appendix D provides a complete example: a 10-step "Draft Email" workflow with variables for recipient_email, subject, and email_content, and steps for clicking icons, typing text, pressing tab, and saving.

Why separate recording from variable extraction. The paper separates interaction recording (which requires the user to perform actions) from variable extraction (which requires reasoning about which text strings are parameterizable). The recording phase captures the raw demonstration — "the user typed alice@example.com in this field" — and the LLM post-processing reasons that alice@example.com is an email address that should be a variable, not a hard-coded literal. This means the user does not need to annotate variables during the demonstration; they simply perform the task naturally, and the LLM infers the parameterization afterward. The LLM is used only at build-time (not at runtime), so its latency and non-determinism do not affect execution.

Step subgraph storage. Each step's subgraph is stored as a JSON object containing the target element ID, the full UI graph for that step (with node positions, icon embeddings, text embeddings, and k-NN edge list), window bounds, the recording scale factor, and click coordinate offsets. This is the data structure that the execution phase's SMC retriever matches against the runtime screen.

Execution Phase: Grounding as Sequential Monte Carlo Localization

This is the paper's central technical contribution — a particle-filtering procedure that jointly estimates the target element's location and the window-scale transformation between the demonstration and runtime screens, using both the target's own appearance and the geometric arrangement of its neighbors as evidence.

The localization problem formalized. The paper models the problem as inferring a latent variable $\theta = [x, y, s_x, s_y]$ where $(x, y)$ is the predicted target center on the runtime screen, and $(s_x, s_y)$ are horizontal and vertical scale factors between the demo and runtime windows. For each neighbor node $i$ in the demo graph, the system precomputes a displacement vector $r_i = c_i^{\text{demo}} - t_{\text{demo}} = [r_{ix}, r_{iy}]^\top$ from the target center to that neighbor's center. Under a hypothesized $\theta$ , the predicted position of neighbor $i$ on the runtime screen is:

$\hat{c}_i(\theta) = [x, y] + [s_x r_{ix}, s_y r_{iy}]$

where $\hat{c}_i(\theta)$ is the predicted runtime center for neighbor $i$ and $(s_x, s_y)$ account for horizontal and vertical rescaling of the window between recording and execution.

What this equation means operationally. Given a guess $\theta$ for where the target is and how much the window has been resized, you can predict where each neighbor should appear based on its known displacement from the target in the demo. For example, if the "Edit" text was 50 pixels to the right of the target checkbox in the demo ( $r_i = [50, 0]$ ), and your hypothesis says the target is at $(300, 200)$ with no rescaling ( $s_x = s_y = 1$ ), then you predict the "Edit" text should appear at $(350, 200)$ on the runtime screen.

The posterior. The goal is to compute $p(\theta \mid Z)$ , the posterior distribution over target location and scale given all observations. The observations $Z = Z_{\text{target}} \cup Z_{\text{neighbor},1} \cup Z_{\text{neighbor},2} \cup \dots$ include both direct appearance matches to the target element and appearance matches to each neighbor element at their predicted positions. The posterior decomposes via Bayes' rule as $p(\theta \mid Z) \propto p(Z \mid \theta) \, p(\theta)$ , where $p(Z \mid \theta)$ is the likelihood of observing matching elements at the predicted positions (the evidence from the runtime screen) and $p(\theta)$ is a prior over plausible locations and scales (the scale prior, described below).

Why a probabilistic formulation. The paper uses a probabilistic formulation rather than a deterministic one because correspondences between demo nodes and runtime candidates are ambiguous — multiple runtime elements might look like the target, and multiple neighbor elements might have plausible matches at slightly different positions. A deterministic "pick the best match" approach would commit to a single correspondence and fail when the best match (by appearance alone) is the wrong one. The probabilistic formulation instead maintains uncertainty over multiple possible correspondences and lets the geometric constraints from neighbors disambiguate them.

The per-node likelihood. For each node $v$ in the demo graph, the likelihood of observing it on the runtime screen given hypothesis $\theta$ combines two possibilities: either the node has a matching candidate on the runtime screen (weighted by appearance similarity and geometric proximity to the predicted position), or the node is missing/occluded. Formally:

$p(Z_v \mid \theta) = p_{\text{miss}} + \max_{c \in C_v} \left[ w_{\text{app}}(v, c) \cdot \mathcal{N}\!\left(p_c \mid \hat{c}_v(\theta), \sigma_v^2 I\right) \right]$

where $C_v$ is the set of candidate runtime nodes that pass an appearance filter (text match for textual elements, icon embedding similarity for visual elements), $w_{\text{app}}(v, c)$ is the appearance similarity score between demo node $v$ and candidate $c$ , $p_c$ is the candidate's center position, $\hat{c}_v(\theta)$ is the predicted center under hypothesis $\theta$ , $\sigma_v^2$ is a size-aware geometric tolerance (explained below), and $p_{\text{miss}}$ is a small constant probability that the node is simply not present on the runtime screen (handles occlusions, detector failures, and genuinely changed UIs). The Gaussian $\mathcal{N}(p_c \mid \hat{c}_v(\theta), \sigma_v^2 I)$ evaluates to a value between 0 and 1 that is high when the candidate is close to the predicted position, and low when it is far away.

Why the max over candidates. For each demo node, the system considers all runtime candidates that pass the appearance filter, not just the single best-looking one. The max takes the candidate that is both visually similar and geometrically consistent with the hypothesis — a candidate that looks perfect but is in the wrong place gets a low Gaussian weight, while a candidate with slightly degraded appearance but perfect position can still contribute strong evidence.

Why $p_{\text{miss}}$ . Without a missing-node term, a demo node with no good runtime match would force the likelihood to near-zero for all hypotheses, even if other nodes match well. $p_{\text{miss}}$ provides a floor: if no candidate matches, the node contributes $p_{\text{miss}}$ regardless of $\theta$ , effectively saying "this node provides no information about the target location." This prevents single-node detection failures from collapsing the entire posterior. The paper sets $p_{\text{miss}}$ to a small constant whose exact value is not specified in the main text but is calibrated to be lower than the contribution of a moderately good match, ensuring that matched nodes dominate the likelihood when available.

Locality weighting. Neighbors closer to the target are more reliable for geometric inference because their relative positions are less affected by layout changes in distant parts of the UI. The joint log-likelihood weights each node by a locality factor:

$\log p(Z \mid \theta) + \log p(\theta) = \sum_{v \in V_{\text{demo}}} w_{\text{loc}}(v) \log p(Z_v \mid \theta) + \log p(s_x) + \log p(s_y)$

where the locality weight is:

$w_{\text{loc}}(i) = \exp\left(-\frac{\|v_i\|^2}{2 \sigma_{\text{loc}}^2}\right)$

with $v_i = c_i^{\text{demo}} - t_{\text{demo}}$ being the demo displacement of node $i$ from the target, and $\sigma_{\text{loc}}$ is an adaptive bandwidth computed using Silverman's rule on the root-mean-square distance from the target to its neighbor nodes. The formula (Appendix A.1) is:

$\sigma_{\text{loc}} = \text{clamp}\left(1.06 \, \hat{\sigma} \, n^{-1/5}, \sigma_{\text{min}}, \sigma_{\text{max}}\right)$

where $\hat{\sigma} = \sqrt{\frac{1}{n}\sum_{i=1}^{n} \|v_i\|^2}$ is the RMS distance from target to neighbor centers, $n$ is the number of neighbor nodes, $n^{-1/5}$ is the asymptotic optimal bandwidth rate from Silverman's rule (which makes the bandwidth narrower when there are more nodes), and clamping bounds are $\sigma_{\text{min}} = 30$ pixels and $\sigma_{\text{max}}$ determined by the screen dimensions.

What adaptive bandwidth achieves. In dense UIs (many elements close to the target, $\hat{\sigma} \approx 70$ pixels, 20 nodes), $\sigma_{\text{loc}} \approx 1.06 \times 70 \times 20^{-0.2} \approx 41$ pixels — only nearby nodes receive significant weight, because dense interfaces provide abundant close neighbors. In sparse UIs (few elements, $\hat{\sigma} \approx 500$ pixels, 3 nodes), $\sigma_{\text{loc}} \approx 1.06 \times 500 \times 3^{-0.2} \approx 425$ pixels — even distant nodes receive meaningful weight because there simply aren't enough close neighbors to constrain the target. When the target is at the edge of the neighbor cluster, $\hat{\sigma}$ is larger, giving the bandwidth room to include nodes in the far part of the cluster. Edge cases: fewer than 2 nodes triggers a fallback constant of 1000 pixels; all nodes at the target position ( $\hat{\sigma} \to 0$ ) is clamped to 30 pixels.

The scale prior. The prior over scale factors $p(s_x)$ and $p(s_y)$ is a mixture of two log-normal distributions per axis:

$p(s_x) = w \cdot \text{LogNormal}(s_x; 0, \sigma_{s,x}) + (1-w) \cdot \text{LogNormal}(s_x; \mu_x, \sigma_{s,x})$

where $w$ is a mixing weight (not explicitly specified, but conceptually balanced to handle both unscaled and proportionally rescaled windows), $\mu_x = \log(W_{\text{live}} / W_{\text{demo}})$ is the observed horizontal window-size ratio between execution and demonstration time, and $\sigma_{s,x}$ controls the spread around each mode. The identity component (mode at $s_x = 1$ , i.e., no resizing) captures the common case where the window dimensions remain unchanged. The ratio component (mode at the observed window-size ratio) captures the case where the user has resized the application window proportionally. The $y$ -axis uses the same structure but with a tighter $\sigma_{s,y}$ since vertical layout is typically more stable than horizontal. Per-axis ratios (rather than a single isotropic scale) account for independent horizontal and vertical rescaling. Penalties are applied symmetrically in log-space so that shrinking and expanding are equally penalized.

Geometric tolerance. The Gaussian variance $\sigma_v^2$ in the per-node likelihood is size-aware:

$\sigma_v = \sigma_{\text{base}} + \alpha \cdot d_v + \beta \cdot \min(w_v, h_v)$

where $d_v = \|c_v^{\text{demo}} - t_{\text{demo}}\|_2$ is the distance from the demo node to the target, $(w_v, h_v)$ is the node's element size, $\sigma_{\text{base}} = 50$ pixels is the base tolerance, and $\alpha$ and $\beta$ are scaling coefficients. The $\alpha \cdot d_v$ term makes the tolerance grow with distance — distant nodes provide inherently less precise geometric constraints because small angular errors in displacement produce large positional errors at distance, so the system is more forgiving of distant-node mismatches. The $\beta \cdot \min(w_v, h_v)$ term makes the tolerance proportional to element size — large elements (dialogs, images) have more ambiguous centers, so the system is more tolerant of positional variation.

The SMC sampler. Direct sampling from the posterior $p(\theta \mid Z)$ is difficult because the posterior can be sharp (strong evidence from many matching neighbors) and multi-modal (multiple plausible target locations, e.g., when the same visual pattern appears in several places on screen). The paper uses a tempered Sequential Monte Carlo (SMC) sampler with $N$ weighted particles (the exact number is not specified, but the method is described as running "in less than 0.2 sec" with vectorized operations). The tempering schedule introduces a sequence of intermediate distributions:

$\pi_\beta(\theta) \propto p(Z \mid \theta)^\beta \, p(\theta)$

where $\beta$ increases from 0 (the prior) to 1 (the full posterior) through adaptive steps. At $\beta = 0$ , particles are distributed according to the prior alone — they cover the screen broadly. As $\beta$ increases, the likelihood exerts more influence, and particles concentrate around locations where both the target's appearance and the neighbor geometry align with the runtime observation.

Particle initialization. Particles are initialized by back-projecting top-K context-candidate matches into target proposals with Gaussian jitter. This means: for each plausible correspondence between a demo node and a runtime candidate, the system computes where the target would be if that correspondence is correct (by applying the inverse displacement), and places a particle near that location with added Gaussian noise. This provides intelligent initialization that covers the plausible modes of the posterior rather than blind uniform sampling over the entire screen.

SMC steps. At each tempering stage $s$ :

Incremental reweighting: particle weights are updated by $\tilde{w}^{(j)} \propto w^{(j)} \cdot p(Z \mid \theta^{(j)})^{\beta_{s+1} - \beta_s}$ , where the temperature increment $\beta_{s+1} - \beta_s$ is chosen adaptively to maintain a target effective sample size (ESS), with an optional cap on $\Delta \beta$ to ensure gradual annealing. The ESS measures how many particles effectively contribute to the approximation — when ESS drops too low, the tempering step was too aggressive and fewer particles carry meaningful weight.
Resampling: particles are resampled with replacement according to their normalized weights, optionally gated by ESS (resampling only when ESS falls below a threshold). This culls low-weight particles and duplicates high-weight ones, focusing computational resources on promising regions.
Rejuvenation (MCMC): each particle undergoes $L$ Metropolis-Hastings steps targeting $\pi_{\beta_{s+1}}(\theta)$ . The proposal is a symmetric Gaussian random walk, with step sizes scaled by the current particle spread and modulated by an intensity factor. During tempering ( $\beta < 1$ ), intensity decays to progressively focus particles; after $\beta = 1$ , intensity resets with a floor so that the MCMC retains enough step size to merge surviving sub-clusters. This prevents particle degeneracy (all particles collapsing to the same point prematurely) and improves mixing between modes.

Early exit. The sampler may exit early when confidence exceeds a threshold $c_{\text{min}}$ or when $\beta = 1$ and confidence has stabilized. An optional refinement phase at $\beta = 1$ runs additional resample+MCMC steps to tighten concentration.

Output. The final prediction $\hat{t}$ is the mean of the densest particle cluster, identified by grid-based clustering. This is important: the SMC posterior may be multi-modal (e.g., two plausible checkboxes on screen), and simply taking the global weighted mean would return a point between modes that corresponds to neither. The densest-cluster mean instead selects the most strongly supported mode.

Why SMC rather than simpler alternatives. The paper's choice of SMC over alternatives is motivated by the multi-modal nature of the posterior. A simple optimization (gradient ascent on the log-posterior) would find a local mode but could miss the correct one. A single Gaussian approximation would collapse multi-modal structure. Grid search over the 4-dimensional space would be prohibitively expensive. SMC provides a middle ground: it maintains multiple hypotheses (particles) through the inference process, uses tempering to smoothly transition from broad exploration to focused exploitation, and naturally produces both a point estimate and uncertainty quantification (via the particle spread).

The fast-path bypass. Before running the full SMC procedure, GPA attempts a direct appearance match using only the target node. It ranks runtime candidates by similarity to the demo target, and if the top candidate satisfies two conditions — (1) score exceeds $s_{\text{min}} = 0.9$ and (2) the normalized entropy of the softmax distribution over top- $k$ candidates is below $H_{\text{thr}} = 0.5$ — the system returns that candidate directly, skipping SMC entirely. The normalized entropy is:

$H = -\frac{1}{\log k_{\text{eff}}} \sum_{j=1}^{k} p_j \log p_j, \quad p_j = \frac{\exp(s_j / \tau)}{\sum_{\ell=1}^{k} \exp(s_\ell / \tau)}$

where $s_j$ are the top- $k$ similarity scores, $\tau = 0.02$ is a temperature that amplifies score differences, and $k_{\text{eff}} = \max(|\{j : p_j > 0.01\}|, 2)$ is the effective candidate count rather than the raw $k$ . The normalization by $\log k_{\text{eff}}$ (not $\log k$ ) prevents dilution from irrelevant tail candidates: if only the top two candidates carry meaningful probability mass, $k_{\text{eff}} = 2$ regardless of how many weak candidates exist. At $\tau = 0.02$ , the distinction between scores of 0.95 and 0.40 produces a sharply peaked distribution ( $H \approx 0.00$ — unambiguous), while scores of 0.92 and 0.90 produce $H \approx 0.84$ — ambiguous because the top two are too close.

Why this fast path. The SMC procedure, while fast (under 0.2 seconds), is still slower than a direct lookup. The fast path handles the common case where the UI is largely unchanged — the target element looks the same, and no other element is a close visual match — without incurring the particle-filter overhead. Only when appearance matching is ambiguous (low score or high entropy) does the system invoke the more expensive geometric reasoning.

Readiness Checking: Converting Probabilistic Inference into Deterministic Confidence

The SMC retriever outputs a probability distribution over the target location — but GPA must make a binary decision: act or don't act. The readiness checker converts the SMC's probabilistic output into a single confidence score $C$ and gates execution on a threshold.

The confidence decomposition. The final confidence is the product of two independent measures:

$C = \tilde{p}(Z \mid \theta) \times C_{\text{spatial}}$

where $\tilde{p}(Z \mid \theta)$ is likelihood confidence — how well the predicted target position explains the observed demo-runtime node matches — and $C_{\text{spatial}}$ is spatial confidence — how tightly the SMC particles agree on the target location.

Likelihood confidence. This measures whether the predicted position $\hat{\theta}$ makes the observed matches likely under the model, independent of particle spread. For each node $v$ in the demo graph, the system computes three quantities at $\hat{\theta}$ :

$\log p_{\text{match},v} = \max_{c \in C_v} \left[ -\frac{\|p_c - \hat{c}_v(\hat{\theta})\|^2}{2\sigma_v^2} + \log w_{\text{app}}(v, c) \right]$ — the best joint score (geometry + appearance) among all candidates.
$\log p_{\text{best\_sim},v} = \max_{c \in C_v} \log w_{\text{app}}(v, c)$ — the best possible appearance score if geometry were perfect.
$\log p_{\text{miss},v} = \log p_{\text{miss}}$ — the baseline if the node is missing.

The per-node confidence normalizes the match quality relative to the best achievable:

$c_v = \text{clip}\left( \frac{\log p_{\text{match},v} - \log p_{\text{miss},v}}{\log p_{\text{best\_sim},v} - \log p_{\text{miss},v}}, 0, 1 \right)$

Operational meaning. When $\log p_{\text{match},v} \geq \log p_{\text{best\_sim},v}$ (the match is as good as or better than the best possible appearance match, meaning the geometry is near-perfect), $c_v \to 1$ . When $\log p_{\text{match},v} \leq \log p_{\text{miss},v}$ (the match is worse than assuming the node is missing), $c_v = 0$ . When the best candidate's similarity itself falls below the missing baseline ( $\log p_{\text{best\_sim},v} < \log p_{\text{miss},v}$ ), the denominator is non-positive and $c_v = 0$ — matching to a very weak candidate should not contribute positively even if geometry is perfect.

Aggregation. The overall likelihood confidence is a locality-weighted average:

$\tilde{p}(Z \mid \theta) = \frac{\sum_{v \in V_{\text{demo}}} w_{\text{loc}}(v) \, c_v}{\sum_{v \in V_{\text{demo}}} w_{\text{loc}}(v)}$

This gives more influence to nearby nodes whose spatial predictions are more reliable. A high $\tilde{p}(Z \mid \theta)$ (close to 1) means the predicted position is geometrically consistent with multiple context-node observations — nodes that should appear near the target do appear near the predicted location. A low value means nodes cannot find their expected matches near the prediction, signaling that the UI layout has changed substantially.

Spatial confidence. This measures posterior certainty — do the particles agree on where the target is? Given $N$ weighted particles with positions $x^{(j)} = (\theta^{(j)}_x, \theta^{(j)}_y)$ , the system computes the weighted mean $\mu$ and covariance $\Sigma$ , then the average variance $\bar{\sigma}^2 = \frac{1}{2} \text{tr}(\Sigma)$ . Under an isotropic Gaussian approximation, the probability that the true position lies within radius $r$ of the mean follows a Rayleigh CDF:

$C_{\text{spatial}} = 1 - \exp\left(-\frac{r^2}{2\bar{\sigma}^2}\right)$

The adaptive acceptance radius. The radius $r$ adapts to the spatial scale of the problem:

$r = r_{\text{base}} + \alpha \cdot \sigma_{\text{loc}}$

where $r_{\text{base}} = 50$ pixels and the $\alpha \cdot \sigma_{\text{loc}}$ term (with the same $\sigma_{\text{loc}}$ used in locality weighting) ensures that the acceptance region grows when context nodes are sparse. In dense UIs ( $\sigma_{\text{loc}} \approx 40$ pixels), $r \approx 58$ pixels — the system demands tight particle convergence because abundant neighbors should provide strong constraints. In sparse UIs ( $\sigma_{\text{loc}} \approx 500$ pixels), $r \approx 150$ pixels — the system tolerates wider spread because limited evidence naturally produces more uncertainty.

Behavior at extremes. When $\bar{\sigma} \approx 5$ pixels (particles tightly clustered) and $r = 50$ , $C_{\text{spatial}} \approx 1.0$ . When $\bar{\sigma} \approx 100$ pixels (wide scatter) and $r = 50$ , $C_{\text{spatial}} \approx 0.12$ . When all particles collapse to a single point ( $\text{tr}(\Sigma) = 0$ ), $C_{\text{spatial}} = 1$ . This last case is important: the spatial confidence formula handles degeneracy gracefully rather than dividing by zero.

Why multiplicative combination. The product structure $C = \tilde{p}(Z \mid \theta) \times C_{\text{spatial}}$ requires both factors to be high for the system to act. The paper explicitly enumerates the failure modes this catches:

High $\tilde{p}(Z \mid \theta)$ , low $C_{\text{spatial}}$ : nodes match well individually, but particles are scattered across multiple modes — the posterior is ambiguous (e.g., two identical checkboxes on screen). Low confidence is correct because the prediction is unreliable.
Low $\tilde{p}(Z \mid \theta)$ , high $C_{\text{spatial}}$ : particles converge tightly, but to a position where nodes do not match — the system is confidently wrong. Low confidence is correct because the prediction is likely incorrect.
High both: strong context evidence and tight particle agreement — proceed.
Low both: poor evidence and scattered particles — do not proceed.

The threshold. GPA proceeds only when $C$ exceeds a threshold. The exact threshold value is not specified numerically in the main text, but the paper describes it as "strictly defined" — the calibration is based on "a pre-computed null distribution" (mentioned in the abstract and introduction), implying that the threshold is set based on the distribution of confidence scores observed on known-correct and known-incorrect matches during development. If confidence is below threshold, the step is retried (after re-observing the screen).

Execution Control: The Finite State Machine and Precheck Pipeline

The SMC retriever and readiness checker determine where and whether to act. The FSM controller determines when to act, when to retry, and when to fail.

Top-level FSM (Figure 6a). For each workflow step, the controller cycles through four states:

CHECK readiness & generate action: parses the current screen into a UI graph, invokes the SMC retriever for the current step's demo subgraph, computes the confidence score $C$ , and generates the action (click coordinates, text to type, hotkey to press) if $C > $ threshold.
DECIDE: chooses among three outcomes based on readiness and retry budget. If ready, finish the step and advance. If not ready and retries remain, sleep for 1 second, re-observe the screen, and return to CHECK. If retries exhausted, fail the step.
EXECUTE: dispatches the generated action to the environment and captures the next observation.
DONE / FAILED: terminal states — DONE advances to the next step, FAILED aborts the workflow.

Bounded retries. The retry budget (not specified numerically, but from context it is small — likely 3–5 attempts) handles transient conditions: a loading spinner that hasn't resolved, a dialog that hasn't appeared yet, a brief animation. Each retry re-observes the screen, which may have changed, and re-runs the SMC retriever on the fresh observation. If the UI eventually reaches the expected state, confidence rises above threshold and execution proceeds. If the UI never reaches the expected state, the workflow fails explicitly rather than guessing.

Scroll-to-find logic (Figure 6b). The DECIDE state contains a small step-specific branch for scroll-to-find steps — scroll actions whose purpose is to reveal a target that is currently off-screen. For these steps, DECIDE first checks whether the target is already visible. If visible, GPA skips the scroll action entirely and completes the step immediately (the scroll was unnecessary because the target is already in view). If the target is not yet visible, GPA checks whether the scroll action itself is ready (the scrollbar or scroll target element is confidently located). A ready scroll action is executed, and on the next iteration the newly scrolled screen is evaluated — potentially revealing the target. A not-ready scroll follows the same bounded retry-or-fail path as regular steps.

Why scroll-specific logic in DECIDE rather than a separate FSM. The paper explicitly notes that "scroll-specific logic refines the decision state without changing the top-level FSM." This is a clean architectural choice: the core FSM's CHECK → DECIDE → EXECUTE cycle is shared across all steps, and only the DECIDE state's internal logic branches on step type. This keeps the controller simple and maintainable while handling the most common special case (scrolling to reveal content).

Precheck pipeline (Appendix C, Figure 7). After executing an action, the environment needs time to settle — a page loads, an animation plays, a dialog appears. The precheck pipeline exploits this idle time by speculatively processing upcoming steps in a background thread. When the runner finishes step $N$ , it submits the current observation to the precheck module and dispatches the action to the environment. While the environment executes, the precheck pipeline processes step $N+1$ (and optionally step $N+2$ ) using the last available observation — which still shows the screen before the action took effect. When the environment returns a new observation, the runner calls collect() to retrieve any completed precheck results. If the precheck result for step $N+1$ has sufficiently high confidence, the runner uses it directly, skipping redundant UI parsing and SMC retrieval. Otherwise, the cached result is discarded and the step is processed normally with the fresh observation.

Why this works. The precheck uses a stale observation (before the action), so it won't always find the target, especially if the target only appears after the action (e.g., a dialog that opens on click). But for steps where the target is already visible before the action (e.g., a sequence of form fields where all fields are already rendered), the precheck can process ahead and have a result ready by the time the runner advances. When confidence is low on the stale observation (because the UI hasn't updated yet), the result is discarded and the step is reprocessed with the fresh observation — so the precheck is a pure performance optimization with no correctness impact.

The complete step lifecycle. Putting it all together: for each step in the workflow template, the FSM captures a new screenshot → the UI Parser extracts elements and builds a runtime graph → the SMC Retriever (or fast-path direct match) locates the target using the demo subgraph → the Readiness Checker computes $C = \tilde{p}(Z \mid \theta) \times C_{\text{spatial}}$ → if $C >$ threshold, the action is generated (click at predicted coordinates, type variable-substituted text, press recorded hotkey) and executed → the environment is observed, and the cycle repeats for the next step. If $C \leq$ threshold, the system sleeps, re-observes, and retries up to the budget. Meanwhile, the precheck pipeline speculatively processes step $N+1$ on the pre-action observation in the background.

Summary of Design Choices and Their Justifications

UI graph with spatial edges over pixel templates or DOM selectors: spatial relationships are invariant under font changes, rendering differences, and minor layout shifts, unlike pixel values or application-internal identifiers.
Hybrid text+icon similarity over pure OCR or pure visual matching: fuzzy text matching handles OCR noise, while icon embeddings provide a signal for purely visual elements and robustness when OCR fails.
SMC with neighbor context over single-element appearance matching or grid search: jointly modeling target appearance and neighbor geometry disambiguates visually identical elements (e.g., the same checkbox icon appearing in multiple rows) that a naive appearance scorer would confuse.
Tempered SMC over direct optimization: maintains multiple hypotheses (particles) through inference, naturally handling multi-modal posteriors that arise from ambiguous context-candidate correspondences.
Adaptive locality bandwidth (Silverman's rule) over fixed bandwidth: automatically adjusts to the spatial density of the UI, trusting only nearby nodes in dense interfaces while leveraging distant nodes in sparse ones.
Two-component confidence (likelihood × spatial) over a single score: catches both "evidence is weak" failures (low likelihood confidence) and "evidence is ambiguous" failures (low spatial confidence), which require different detection mechanisms.
Mixture log-normal scale prior over a single fixed-scale assumption: handles both unchanged and rescaled windows by placing modes at $s = 1$ and at the observed window-size ratio, with symmetric log-space penalties.
Size- and distance-aware geometric tolerance over fixed $\sigma$ : prevents distant nodes and large elements from being overly penalized for small positional errors that are expected given their geometry.
Fast-path direct match with entropy gate over always running SMC: handles the common unchanged-UI case efficiently (direct lookup) while only invoking SMC when appearance matching is genuinely ambiguous.
Bounded-retry FSM with sleep-and-reobserve over single-shot matching or unbounded retries: handles transient UI states (loading, animations) without risking infinite loops or acting on bad matches.
Offline LLM for variable extraction and descriptions over runtime LLM calls: moves all generative uncertainty to build-time, where it can be reviewed and corrected, keeping the execution path purely deterministic.
Precheck pipeline over serial processing: exploits environment-settling idle time for speculative computation, reducing effective per-step latency with zero correctness risk (low-confidence precheck results are discarded).
Scroll-to-find target visibility check over blindly executing scrolls: prevents unnecessary scroll actions when the target is already in view, and correctly handles the case where a single scroll reveals the target early.

4. Key Insights and Innovations

Innovation 1: Generative Reasoning Is Mismatched to Deterministic Replay — and That's a Feature Decision, Not a Capability Gap

This paper's most conceptually disruptive move is not a technical method but a design philosophy argument: for the specific, high-volume class of tasks where a user can demonstrate the correct workflow once, using a generative model to decide what to click at runtime is actively harmful — it introduces failure modes without adding value, because the reasoning has already been done by the human demonstrator.

The field's default assumption, visible across the entire VLM GUI agent literature the paper surveys (Section 4), is that more reasoning capability = better GUI automation. Systems like WebVoyager, OS-ATLAS, Aguvis, and UI-TARS attack the problem by scaling model size, improving grounding, and adding planning modules — all aimed at making the agent smarter about what action to take. The paper does not dispute that this is valuable for open-ended tasks. But it draws a sharp boundary: when the task is replay, generative action selection converts a deterministic capability (matching what was demonstrated) into a stochastic one (generating what seems right given the context). The agent can misread a label, click an adjacent element, or miss a page load — errors that a deterministic matcher, gated by confidence, simply does not make.

The evidence that makes this argument concrete is Table 2: GPA achieves 100% success on both simple and hard tasks versus Gemini's 87.64% on hard tasks, with the gap widening as task length grows. The paper's explanation — "Gemini must infer the correct action at every step, and each inference carries a small misidentification risk that compounds over the trajectory" — encodes the core insight: per-step generative error is a structural property of the approach, not a model-quality issue. A better VLM reduces the per-step error rate but cannot eliminate it, because the mechanism itself is probabilistic. GPA eliminates it entirely by refusing to generate actions — it only matches.

This framing is significant beyond the specific system because it challenges the increasingly dominant narrative that "LLMs/VLMs are the future of GUI automation." The paper is not saying VLMs are bad — it explicitly carves out a role for them in the MCP/CLI integration (Section 5) where they handle orchestration and reasoning — but it argues that the execution substrate for known interaction patterns should be deterministic. This is a fundamental architectural claim with implications for how AI-augmented automation systems should be composed: separate the what from the how, and use the appropriate mechanism for each.

Innovation 2: UI Grounding as Graph Matching with Geometric Context — Moving Beyond the Single-Element Localization Paradigm

The dominant approach in GUI grounding — visible in SeeClick (Cheng et al., 2023), ScreenSpot (Li et al., 2025), and the grounding modules of most VLM agents — treats UI element localization as a single-element retrieval problem: given a description or reference image of the target element, find the matching element on the current screen. This works well when the target is visually distinctive, but fails in a common and practically important case: visually identical elements that are disambiguated only by their surrounding context.

Consider the paper's running example: multiple identical checkboxes in a settings panel, distinguished only by the text labels next to them ("Edit", "Delete", "Share"). A single-element matcher — whether based on icon embeddings, template matching, or even an attention-based VLM — sees the same checkbox icon at multiple locations and must pick one. The paper's entropy-based ambiguity detection (Appendix A.2) explicitly quantifies this failure mode: when the top- $k$ candidate scores are near-identical, entropy is high and the system cannot confidently select. Prior work either picks the top-scoring match and accepts a random failure rate, or relies on the VLM's attention to implicitly model context — which is unreliable precisely because attention is distributed and not guaranteed to latch onto the correct spatial relationship.

GPA's move is to elevate grounding from single-element matching to subgraph matching under geometric constraints. The per-node likelihood (Equation 2) jointly models the target and its neighbors, but the innovation is not the equation — it's the idea that neighbor geometry is evidence for target location. When the target checkbox is visually ambiguous, the system does not guess among the identical candidates. Instead, it asks: which candidate is positioned correctly relative to the "Edit" label, the "Delete" label, and other nearby elements, given their recorded spatial relationships? The SMC procedure aggregates this geometric evidence across multiple neighbors, weighted by locality (nearby nodes provide stronger constraints), to resolve ambiguity that appearance alone cannot.

This matters conceptually because it reframes grounding as a relational inference problem rather than a retrieval problem. The target element is not identified in isolation; it is identified by its position in a spatial neighborhood. This mirrors how humans locate ambiguous UI elements — "the checkbox next to the word 'Edit'" — and provides robustness against the exact failure mode that plagues appearance-only matchers. The evidence for this working is implicit in GPA's 100% success rate (Table 2): many of the 16 pilot tasks likely contain visually repetitive elements (form fields, checkboxes, buttons in list views), and the SMC+context mechanism handles them without error.

Innovation 3: Calibrated Readiness as a Mechanism for Deterministic Reliability — Separating "Where" from "Whether"

VLM-based GUI agents collapse two distinct decisions into a single generative step: where to click and whether to click. The agent generates an action — a coordinate, a button reference — and executes it, regardless of confidence. Even when VLM agents output confidence scores or use self-consistency checks, these are post-hoc filters on a generated decision, not structural gates that prevent uncertain actions from being generated in the first place.

GPA introduces a structural separation between localization (estimating where the target is) and readiness (deciding whether the evidence is strong enough to act). The SMC retriever produces a posterior distribution over target location and a likelihood model of how well the evidence supports that location. The readiness checker then computes two independent confidence components — likelihood confidence $\tilde{p}(Z \mid \theta)$ (how well does the predicted position explain the observed matches?) and spatial confidence $C_{\text{spatial}}$ (how tightly do particles agree?) — and gates execution on their product. This decomposition is not an engineering detail; it encodes a specific diagnostic insight: there are two distinct ways a match can be bad, and they require different detection mechanisms.

Observation mismatch (low $\tilde{p}(Z \mid \theta)$ , possibly high $C_{\text{spatial}}$ ): the particles agree on a location, but the context nodes don't match well there. This happens when the UI layout has changed — the target looks correct in isolation, but its neighbors have moved. A system that only checked particle agreement would be "confidently wrong" and would execute. GPA catches this via the likelihood component.
Posterior ambiguity (high $\tilde{p}(Z \mid \theta)$ , low $C_{\text{spatial}}$ ): individual nodes match well at multiple plausible locations, and particles are scattered across these modes. This happens with visually repetitive UIs where context can't fully resolve ambiguity. A system that only checked node-match quality might pick one mode arbitrarily. GPA catches this via the spatial confidence component, and the adaptive acceptance radius (larger in sparse UIs, tighter in dense UIs) ensures the system isn't overly conservative or permissive in different UI layouts.

The significance of this innovation is that it makes determinism a property of the execution mechanism, not a dataset-level statistic. VLM agents report success rates — 93% on simple tasks, 87% on hard tasks — which are averages over trials. A 10% failure rate means you don't know which tasks will fail, and you certainly don't know when within a task the failure will occur. GPA's readiness checker ensures that the system never acts on ambiguous or low-quality matches; it either acts with high confidence or retries/reports failure explicitly. The 100% success rate (Table 2) is not a statistical claim about a distribution — it is a property of the gating mechanism: on these 16 tasks, the readiness checker never let through a bad match, and the retry logic resolved all transient issues.

This represents a shift from probabilistic reliability to structural reliability — from "this works 95% of the time" to "this works when the evidence is sufficient, and refuses when it isn't." It's a reframing of the reliability problem in GUI automation, and it connects to broader discussions in AI safety about when and how to deploy systems with bounded uncertainty guarantees.

Innovation 4: The Compute- vs. Inference-Time Tradeoff Reframed for GUI Automation — Lightweight Local Models Can Outperform Cloud VLMs When the Problem Is Scoped Correctly

The paper's pilot results (Table 2) contain a finding that is easy to overlook amid the SMC technical content but carries significant practical implications: a system built entirely on lightweight local models (finetuned OmniParser for detection, IconCLIP for embeddings, fuzzy string matching for text, and a vectorized SMC sampler) achieves 10× faster execution and higher success than Gemini 3 Pro with CUA tools. This is not an incremental improvement — it inverts the standard assumption that more capable models (larger, cloud-hosted, more parameters) produce better results for complex tasks.

The dominant narrative in the GUI agent literature is one of scaling up: CogAgent → OS-ATLAS → Aguvis → UI-TARS, with each generation using larger models and more sophisticated reasoning. The implicit assumption is that desktop automation is hard enough to require frontier-model-level visual understanding and planning. GPA demonstrates that, when the task is correctly scoped to replay rather than planning, tiny specialist models outperform massive generalist ones. This is not because the local models are "better" in any absolute sense — IconCLIP is dramatically less capable than Gemini — but because the problem has been restructured so that the models only need to do what they're good at (visual similarity matching, text comparison) and never need to do what they're bad at (reasoning, planning, multi-step decision-making under uncertainty).

The practical significance is hard to overstate. The paper identifies privacy, latency, and cost as the three barriers to enterprise adoption of VLM-based GUI agents (Section 1, Table 1). GPA eliminates all three simultaneously: local execution solves privacy, lightweight models solve latency (10× speedup), and no per-action API calls solve cost. The 100% success rate is the reliability guarantee that makes the other three properties commercially viable — a fast, private, cheap system that fails 12% of the time (like Gemini on hard tasks) is not a production automation solution; it's a semi-automated tool requiring human oversight.

This is not a claim about model architecture or training methodology — it's a system-design insight about how task scoping changes the capability requirements. By restricting to replay (no planning), limiting actions to what was demonstrated (no open-ended decision-making), and using confidence gating (no action on uncertainty), GPA reduces the GUI automation problem to a visual matching + geometric inference problem that can be solved with mature computer-vision techniques and tiny models. The comparison to Gemini is not about whose technology is better — it's about matching the solution mechanism to the problem structure, and the evidence shows that for replay tasks, the deterministic local approach dominates on every axis that matters for enterprise deployment.

The paper's choice of SMC for localization might appear to be a standard Bayesian inference implementation detail, but it represents a diagnostic insight about the structure of the UI grounding posterior. The posterior over target location $p(\theta \mid Z)$ is not just uncertain — it is potentially multi-modal, because the same visual pattern (a checkbox, a button icon, a text label) can appear at multiple locations on screen, and each instance is geometrically consistent with some subset of the neighbor observations. This multi-modality is not a bug to be smoothed over with a stronger prior or a better detector; it is a fundamental property of ambiguous visual interfaces.

Prior approaches handle this implicitly or poorly. Single-element matchers pick the best-scoring candidate and ignore multi-modality entirely — they commit to a single mode and fail when it's the wrong one. VLM agents rely on attention mechanisms to implicitly select among modes, but attention is soft and distributed, and there is no guarantee that the model's internal representation correctly tracks which candidate is actually the target. Even within the SMC/particle-filter literature, many applications use a single Gaussian approximation (Kalman filtering or its extensions) that collapses multi-modal structure.

GPA's SMC formulation is specifically designed to maintain and resolve multi-modality:

Particle initialization via back-projection from context-candidate matches seeds particles in the vicinity of each plausible target location, ensuring the sampler doesn't miss modes.
Tempered transitions (Equation 12) gradually increase the influence of the likelihood, allowing particles to explore broadly at low $\beta$ and concentrate at high $\beta$ , rather than collapsing immediately to the nearest local mode.
MCMC rejuvenation with step sizes scaled to particle spread prevents particle degeneracy while allowing sub-clusters to merge when the evidence favors a single mode.
The final prediction uses the densest cluster mean, not the global weighted mean, explicitly selecting a mode rather than averaging across modes (which would produce a point between targets — a classic particle-filter failure mode).

The choice of SMC is therefore not just a probabilistic implementation — it is an architectural commitment to handling the multi-modal nature of the problem explicitly rather than implicitly or not at all. The fact that the entire SMC procedure runs in under 0.2 seconds (Section 2.3) with vectorized operations demonstrates that this commitment is practical, not just theoretically elegant.

This insight generalizes beyond GPA: any visual grounding system operating in environments with repetitive visual elements (which is most real-world UIs — spreadsheets, forms, settings panels, file managers) needs to handle posterior multi-modality explicitly. The paper's SMC formulation provides a template for how to do this: initialize broadly using geometric back-projection, temper the likelihood to avoid premature mode collapse, maintain particle diversity through MCMC, and select the dominant mode at the end. It's a transferable architectural pattern, not a one-off implementation choice.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation uses a custom set of 16 desktop GUI tasks collected by the authors. There is no mention of using a public benchmark such as OSWorld, ScreenSpot, or Mind2Web. The tasks are categorized by the length of the recorded demonstration: 5 "simple" tasks averaging 10.8 steps (including drafting an email, downloading a receipt, flight searching and booking) and 11 "hard" tasks averaging 27.27 steps (including Google Calendar event creation, Agentforce tasks, SAP ERP form-filling, reimbursement submission, and HR workflows for interview scheduling and candidate information entry). This task set is described as a "pilot study" and covers "a mix of common desktop productivity and enterprise workflows" (Section 3).
Base model(s). GPA itself uses no large generative model at runtime. The components that are models are: (1) a finetuned OmniParser variant for icon detection (available at https://huggingface.co/Salesforce/GPA-GUI-Detector), described as a lightweight detector trained by the authors; (2) IconCLIP (ViT-B-32) for computing 512-dimensional icon embeddings; and (3) OCR for text extraction (the specific OCR engine is not named, but Appendix D's metadata references ocrmac_detector, suggesting macOS-native OCR). An LLM is used only at build-time for post-processing (variable extraction, step description generation). The comparison baseline is Gemini 3 Pro with computer-use (CUA) tools, a cloud-based VLM agent from Google (Gemini 2.5/3 family, referenced as the "Gemini computer-use agent" in Section 3).
Metrics. Two metrics are reported: wall-clock runtime measured in seconds (presumably end-to-end from workflow start to completion, including any retries and environment settling delays) and success rate (the fraction of tasks for which the workflow completes correctly). The paper does not define "correctly" with a formal success criterion, but from context it implies the workflow's intended outcome is achieved (the email is drafted, the form is submitted, the calendar event is created). For Gemini, "success" presumably means the agent completed the described task without intervention. The paper does not report metrics like per-step accuracy, number of retries, or confidence-score distributions.
Baselines. There is exactly one baseline: Gemini 3 Pro with CUA tools. The paper describes Gemini's protocol as: "given the same demonstrated video and process them into textual form, and use it to guide its action autoregressively during execution" (Section 3). This means Gemini receives the demonstration as a video → text conversion (presumably describing what the user did) and then generates actions step-by-step autoregressively, rather than receiving a natural-language goal as in standard VLM-agent benchmarks. The paper does not include baselines like: a simpler template-matching replay system, an RPA system with selectors, a different VLM agent (Claude Computer Use, OpenAI Operator), or a demonstration-based method like LearnAct or Instruction Agent. There is also no baseline comparing GPA-without-SMC (i.e., pure appearance matching without neighbor context) to measure the SMC component's contribution.
Generation budget / compute accounting. For GPA, compute is not measured in FLOPs or tokens; it is measured indirectly by wall-clock runtime. The latency breakdown (per-step SMC time of "less than 0.2 sec," OCR parsing, graph construction) is described qualitatively but not reported with per-component timing in the main experiments. For Gemini, the cost model is API latency — each step incurs network round-trip and VLM inference time. The paper reports aggregate runtime but does not normalize for hardware differences (GPA runs locally; Gemini runs on Google's cloud infrastructure) or attempt a FLOPs-matched comparison.
Cross-validation / statistical protocol. There is no cross-validation, no statistical significance testing, no confidence intervals, and no multiple-run averaging reported. The pilot study appears to be a single run per task per method. The paper does not describe whether tasks were repeated with different variable values (e.g., different email recipients, different flight dates) or whether the reported numbers are from a single execution. The task set of 16 is too small for meaningful statistical analysis, which the authors acknowledge implicitly by calling this a "pilot study" (Section 3) and explicitly in the paper's title/abstract framing as a "proof of concept project for research."

Main Quantitative Results

Aggregate Performance Comparison (Table 2)

The headline result is a clean sweep: GPA achieves 100% success on both simple and hard tasks, while Gemini 3 Pro achieves 93.2% on simple tasks and 87.64% on hard tasks, averaging 89.38% across all 16 tasks. The runtime gap is dramatic: GPA averages 33.74 seconds across all tasks (17.84 s for simple, 40.96 s for hard), while Gemini averages 329.31 seconds (210.66 s for simple, 383.24 s for hard), making GPA roughly 10× faster on average.

The hard-task breakdown is particularly informative: on the 11 tasks averaging 27.27 steps, Gemini's success rate drops by 5.56 percentage points relative to simple tasks (from 93.2% to 87.64%), while GPA remains at 100%. The paper attributes this gap to compounding per-step failure probabilities in Gemini: "The generative agent may misread a label, click an adjacent element, or miss a page load at each step — over 27 steps these per-step failure probabilities compound" (Section 3). GPA's fixed-plan execution "adds only a cheap matching operation per step with no additional drift risk."

The runtime scaling with task length shows GPA's advantage widening: on simple tasks (10.8 steps), GPA is 11.8× faster (17.84 vs. 210.66 s); on hard tasks (27.27 steps), GPA is 9.4× faster (40.96 vs. 383.24 s). The slightly lower speedup ratio on hard tasks likely reflects that Gemini's per-step latency is relatively constant (~14 s per step for simple, ~14 s per step for hard), while GPA's per-step time includes some fixed overhead (screenshot capture, graph construction) that doesn't scale linearly with the number of elements.

The Three Explanatory Factors (Section 3, Results Paragraphs)

The paper decomposes the performance gap into three factors, though only the first two are supported by direct measurements:

Reliability. The paper claims that Gemini "must infer the correct action at every step, and each inference carries a small misidentification risk that compounds over the trajectory," while "GPA follows a fixed demonstrated procedure and acts only when the readiness checker confirms a confident match." This is a mechanistic explanation rather than an experimental finding — there is no ablation comparing GPA with and without readiness checking, or Gemini with a stronger grounding module. The evidence is the success-rate differential itself, which is consistent with the explanation but does not isolate readiness checking as the causal factor.
Latency. The paper attributes Gemini's slowness to "network round-trip and VLM inference latency per step," contrasted with GPA's "on-device screenshot capture, OCR parsing, and local subgraph retrieval." This is directly observable in the runtime numbers (329.31 s vs. 33.74 s average). However, the paper does not provide a latency breakdown for either system — we don't know what fraction of GPA's 33.74 seconds is screenshot capture, OCR, graph construction, SMC retrieval, or environment settling, and we don't know what fraction of Gemini's 329.31 seconds is network latency, VLM inference, or screenshot encoding. This makes it difficult to assess whether the gap would persist under different network conditions or with different VLM hosting configurations.
Scaling with length. The paper argues that "GPA's fixed-plan execution adds only a cheap matching operation per step with no additional drift risk." The evidence is the 100% success on hard tasks vs. Gemini's 87.64% — but with 16 total tasks and unknown per-task variance, it's not possible to determine whether the 5.56 percentage-point drop from simple to hard for Gemini is statistically reliable or driven by one or two particularly difficult tasks.

Result Granularity and Missing Analyses

Several analyses that would strengthen the results are absent:

Per-task breakdown. Table 2 reports only averages within the simple/hard categories. We don't know which specific tasks Gemini failed on, whether the failures were on the same tasks across runs, or whether certain task types (SAP forms vs. HR workflows vs. email drafting) were systematically harder.
Confidence-score distributions. GPA reports a confidence score C before each action. The paper does not report the distribution of these scores across steps or tasks, the frequency of retries, or how often the fast-path direct match was used vs. the full SMC procedure. These would characterize the "difficulty" of the matching problem for the pilot tasks.
Failure mode analysis for Gemini. The paper states Gemini "may misread a label, click an adjacent element, or miss a page load" but does not report what actually happened on the failed tasks. Knowing whether the failures were grounding errors (wrong element), planning errors (wrong sequence), or timing errors (acting before page load) would clarify whether GPA's approach addresses the dominant failure mode or simply bypasses it by avoiding generative action selection entirely.
Latency breakdown. There is no component-level timing for GPA (screenshot capture, OCR, graph construction, SMC retrieval per step, env.step latency) or for Gemini (API call latency, inference time, token generation speed). The precheck pipeline's contribution to latency reduction is described architecturally (Appendix C) but not evaluated quantitatively.

Ablation Studies and Robustness Checks

The paper contains no formal ablation studies for the complete system evaluated in the pilot. The pilot experiment (Table 2) compares the full GPA system against one baseline (Gemini 3 Pro) and does not include ablated variants. However, the technical appendix contains several design-level analyses that function as component-level justifications:

Entropy-based ambiguity detection vs. score-gap heuristic (Appendix A.2): The paper argues qualitatively that normalised entropy over softmax scores provides better ambiguity detection than a simple margin test (s1 - s2 > Δ), because entropy accounts for the number of close competitors and the overall peakedness of the distribution. A worked example at τ = 0.02 shows: scores [0.95, 0.40, 0.30] → H ≈ 0.00 (unambiguous); scores [0.92, 0.90, 0.30] → H ≈ 0.84 (ambiguous); scores [0.85, 0.84, 0.83, 0.82] → H ≈ 0.90 (ambiguous with four near-equal candidates). No empirical comparison of the two methods is provided.

Adaptive locality bandwidth (Appendix A.1): The paper provides analytical justification for Silverman's rule-based adaptive σ_loc, with worked examples showing the bandwidth's behavior in dense UIs (σ_loc ≈ 41 px), sparse UIs (σ_loc ≈ 425 px), and edge cases (fewer than 2 nodes → fallback to 1000 px; all nodes at target → clamp to 30 px). No ablation comparing adaptive vs. fixed bandwidth on actual task performance is reported.

Likelihood × spatial confidence decomposition (Appendix A.3): The paper enumerates the four quadrants of the confidence space (high/low likelihood × high/low spatial) and explains why each requires a different action, but does not report how often each quadrant occurs in practice or whether the multiplicative combination outperforms alternative aggregation methods (e.g., minimum, mean, or learned weighting).

Precheck pipeline (Appendix C): The architecture is described with a sequence diagram (Figure 7), but no measurements of cache hit rate, confidence of precheck results vs. fresh results, or latency reduction are provided.

SMC vs. fast-path direct match: The paper describes a two-stage gating mechanism (direct match when score > 0.9 AND entropy < 0.5; SMC otherwise) but does not report what fraction of steps took the fast path vs. the full SMC procedure in the pilot tasks, or how much latency this saved.

PRM vs. ORM, revision model variants, etc.: Not applicable — these are components from the reference example paper, not from GPA.

The absence of ablations is a significant gap. The paper's central technical claim — that SMC with neighbor context provides robust localization — is supported only by the system-level 100% success rate, which could be attributable to any subset of GPA's components (the UI detector's quality, the readiness threshold's conservatism, the retry logic, the fast-path bypass, or the SMC itself). A minimal ablation suite would compare: (1) GPA with SMC vs. GPA with only the fast-path direct match (to isolate the SMC's contribution on ambiguous elements), (2) GPA with neighbor context vs. GPA with target-only matching (to isolate the geometric context contribution), and (3) GPA with readiness checking vs. GPA without (to isolate the gating mechanism's contribution). None of these are provided.

Critical Assessment

Central Claim 1: GPA achieves higher success rate than Gemini 3 Pro while being 10× faster.

What was tested: A pilot study of 16 desktop GUI tasks, running GPA and Gemini 3 Pro once each (presumably — the paper does not report multiple runs or variance), measuring success rate and wall-clock time.

Does this demonstrate the claim? Partially, but with major caveats:

The claim holds for these 16 tasks, assuming the numbers are from single executions. GPA's 100% success rate is a strong signal, but without multiple executions per task (with different variable values, timing conditions, or minor UI variations), we cannot assess whether the 100% is robust or lucky. A single successful run per task does not establish reliability in the statistical sense — it establishes that the system worked once on each task.
The baseline protocol may be disadvantageous to Gemini. The paper describes Gemini's input as "the same demonstrated video and process[ed] into textual form, and use it to guide its action autoregressively." This is an unusual protocol — standard VLM-agent evaluation gives the agent a natural-language goal ("draft an email to alice@example.com with subject 'Meeting' and body '...'") and lets it plan and execute. By giving Gemini a video of the demonstration converted to text, the paper is essentially asking the VLM agent to follow step-by-step instructions, which removes its planning advantage (the agent doesn't need to figure out the workflow — it's told what to do) while retaining its generative action-selection disadvantage (it still needs to generate click coordinates/target descriptions at each step). A fairer protocol might give Gemini the same high-level variables that GPA receives at execution time and let it plan the workflow autonomously, measuring whether its planning flexibility compensates for its grounding errors.
Hardware asymmetry. GPA runs entirely on-device; Gemini runs on Google's cloud infrastructure. The latency comparison is not controlled for compute — it compares local CPU/GPU processing against cloud API latency + large-model inference. A FLOPs-matched or cost-matched comparison is not attempted. If Gemini were hosted on the same machine (or GPA were run on the same cloud hardware), the latency gap would likely narrow, though it's unlikely to close entirely given the fundamental difference in model scale.
Task selection is not characterized. The 16 tasks were chosen by the authors and may be skewed toward workflows where GPA's approach works well. The paper does not describe a systematic task-collection protocol, difficulty rubric, or coverage analysis. Tasks that require runtime reasoning (calendar date selection, dynamic popup handling, conditional branching) are explicitly acknowledged as out of scope for GPA (Section 5 Limitations), but we don't know whether such tasks appeared in the pilot and contributed to Gemini's failures, or whether the task set was curated to avoid them.
No confidence intervals, no replication. With 16 tasks and (apparently) single runs, the 100% vs. 89.38% success rate comparison is not statistically testable. A single additional failure by GPA or success by Gemini would change the numbers substantially. The paper appropriately calls this a "pilot study," but the framing in the abstract ("GPA achieves higher success rate with 10× faster execution speed") presents these numbers as generalizable findings rather than preliminary observations.
No comparison to simpler deterministic baselines. The paper argues that GPA is better than both traditional RPA (fragile) and VLM agents (non-deterministic), but does not compare against either a template-matching-based replay system or a selector-based RPA system on these 16 tasks. Without these comparisons, we cannot assess whether GPA's SMC-based approach is actually better than simpler alternatives, or whether the tasks were simply easy enough that any robust matching system would succeed.

Central Claim 2: The SMC with neighbor context and readiness calibration enables robust localization under rescaling and detection uncertainty.

What was tested: The full system's end-to-end success rate on 16 tasks. No component-level evaluation or ablation is reported.

Does this demonstrate the claim? No. The pilot experiment tests the entire GPA system as a monolith. There is no experiment that: varies the amount of rescaling between demonstration and execution, systematically introduces detection failures, or compares SMC against simpler localization methods on a controlled grounding benchmark. The paper's technical appendices provide analytical justification for design choices (Silverman's rule, entropy gating, the confidence decomposition), but these are design rationales, not empirical validations.

The paper would need at minimum: (1) a controlled grounding experiment measuring localization accuracy under varying scale changes, (2) an ablation comparing SMC vs. direct matching vs. simpler alternatives on a set of annotated grounding examples (similar to ScreenSpot or SeeClick evaluation protocols), and (3) measurements of how often the readiness checker correctly rejects bad matches vs. incorrectly rejects good matches (precision/recall of the gating mechanism). None of these exist in the current paper.

Central Claim 3: GPA can serve as a reliable execution substrate for VLM agents (MCP/CLI integration).

What was tested: Nothing. This claim is purely architectural — it is mentioned in the abstract, introduction, and conclusion as a design vision but is never evaluated experimentally. There is no experiment where a VLM agent orchestrates multiple GPA workflows, handles exceptions using reasoning, or benefits from GPA's deterministic execution.

Does this demonstrate the claim? No. This is a forward-looking statement, not an evaluated contribution. The paper appropriately labels GPA as "a proof of concept project for research" and invites collaboration for "product development, or enterprise application."

Central Claim 4: GPA's fully local execution addresses privacy and latency concerns of VLM agents.

What was tested: Only runtime (the 10× speedup). Privacy is asserted as a property of the architecture ("sensitive visual data never leaves the local machine") but is not empirically evaluated — there is no measurement of what information could be extracted from intermediate representations (IconCLIP embeddings, OCR text) or a security analysis of the local pipeline.

Does this demonstrate the claim? The latency claim is supported by the runtime comparison, with the hardware-asymmetry caveat noted above. The privacy claim is an architectural property, not an experimental finding — it's true by construction (the system runs locally) but the paper does not evaluate whether this is sufficient for specific regulatory requirements or threat models.

Missing experiments that would substantially strengthen the paper:

Grounding accuracy benchmark. Evaluate GPA's SMC retriever on a public grounding benchmark (ScreenSpot, ScreenSpot-Pro, SeeClick) with known ground-truth bounding boxes, measuring localization error and success rate under controlled visual variations (scaling, font changes, layout shifts). Compare against single-element matching and VLM-based grounding methods.
Component ablations. On the 16 pilot tasks (or a larger set): GPA without SMC (direct match only), GPA without neighbor context (target-only matching), GPA without readiness checking (always execute best match), GPA without the precheck pipeline. These would isolate the contribution of each mechanism.
Multiple runs with variation. Execute each task multiple times with different variable values, different screen resolutions, and different timing conditions. Report mean success rate with confidence intervals. This would characterize the robustness of the 100% claim.
Failure analysis. For any task where GPA's confidence checker triggers retries or failures, report what happened — which step, what the confidence score was, whether the UI had genuinely changed or the matcher was being overly conservative. This would characterize the system's behavior at the boundary of its capability.
Stronger baselines. Compare against: (a) a template-matching replay system (OpenCV-based), (b) a selector-based RPA system on applications where selectors are available, (c) a simpler SMC variant (target-only, no neighbor context), and (d) the VLM agent given the goal description rather than the demonstration transcript. This would position GPA more precisely in the design space.
VLM agent + GPA integration. The simplest version: have a VLM agent decide which workflow to run based on a high-level goal, then hand off to GPA for execution. Measure whether this hybrid approach improves the VLM agent's reliability or latency.

In summary, the pilot experiment provides existence proof that GPA can successfully replay 16 specific desktop workflows with 100% success and substantially lower latency than a cloud VLM agent — but it does not establish the generality, robustness, or component-level contributions of the approach. This is consistent with the paper's own framing as a "proof of concept project for research," and the claims in the abstract should be read as preliminary findings rather than established results. The technical contributions (SMC with neighbor context, readiness calibration, the two-phase architecture) are well-motivated and analytically justified, but their empirical validation is currently at the level of a system demonstration rather than a controlled evaluation.

6. Limitations and Trade-offs

6.1 Capability Boundary: No Reasoning, No Runtime Adaptation, No Dynamic Decision-Making

The assumption or constraint. GPA is fundamentally a record-and-replay system — it executes a workflow exactly as demonstrated and cannot adapt to situations requiring judgment, reasoning, or state-dependent decision-making. The paper states this explicitly in Section 5 (Conclusions & Limitations):

"GPA is a record-and-replay system: it does not have reasoning or decision-making capabilities. It executes workflows as recorded and cannot adapt to situations that require judgment."

The paper provides a concrete example of what breaks: date pickers. Selecting a date on a calendar widget requires reasoning about the current state — which month is currently displayed, how many clicks to advance, what day of the week a date falls on. GPA cannot perform this reasoning; it replays the exact sequence of clicks from the demonstration. If the demonstration selected June 15, 2026, GPA will click the same relative positions regardless of what month the calendar widget is currently showing. The paper acknowledges this directly: "date pickers will only work correctly if the same date as in the recording is being selected."

The consequence. Any GUI task that involves conditional logic — if this dialog appears, click Cancel; if this field is pre-filled, skip to the next step; if this dropdown shows a different set of options, scroll to find the right one — is either out of scope for GPA or requires the human demonstrator to have anticipated and recorded separate workflows for each branch. This substantially limits the class of automatable workflows. Many real-world enterprise processes contain exactly this kind of conditional behavior: an expense report might route differently depending on the amount, an HR form might show different fields depending on the employee type, a customer service workflow might branch based on the issue category. GPA offers no mechanism to encode or execute such branches — it is a linear sequence of recorded actions.

More subtly, this limitation extends to UI state changes that alter the action sequence but not the visual appearance of individual elements. If a new version of an application adds an intermediate confirmation dialog, or reorders two steps, or changes the tab layout, GPA has no way to detect that the process has changed. The readiness checker will find that the expected target element is not present on screen, but it cannot diagnose why or adapt — it can only retry (hoping the element appears after a delay) or fail. A VLM agent could potentially recognize the new dialog and reason about how to dismiss it; GPA cannot.

What evidence exists in the paper. The pilot experiment (Table 2) includes task categories like "flight searching and booking" and "SAP ERP form-filling" but does not report whether these tasks contained conditional logic, dynamic content, or state-dependent steps. The 100% success rate suggests that the 16 chosen tasks either did not require runtime reasoning, or the specific parameter values used during testing happened to match the demonstration's state. The paper does not report testing tasks where the execution-time state deliberately differs from the demonstration (e.g., selecting a different date, filling a form with different field visibility, encountering a different error state).

Mitigation status. The paper does not attempt to solve this limitation. The proposed mitigation is architectural: the MCP/CLI integration vision (Sections 1, 5) suggests that VLM agents could handle the reasoning/orchestration layer, invoking GPA workflows as deterministic execution primitives for known interaction patterns. But this is not implemented or evaluated. The paper also mentions future work on "self-healing when a workflow becomes stale due to UI updates" and "precondition tracking for tool use" (Section 5), but these are forward-looking research directions, not contributions of the current system.

A practitioner evaluating GPA for deployment needs to answer: does my target workflow contain conditional logic, dynamic state, or branches? If yes, GPA in its current form cannot automate it without either (a) recording separate workflows for each branch and orchestrating them externally, or (b) restricting to cases where the demonstration's state parameters are reused exactly.

6.2 Single-Demonstration Dependency: What Happens When the Recorded UI Template Goes Stale?

The assumption or constraint. GPA's entire execution model depends on matching the current screen against a step subgraph recorded during a single demonstration. Any element that was present in the demonstration must be locatable — via appearance matching, geometric context, or both — on the execution screen. If the UI changes significantly between demonstration and execution, the match may degrade below the readiness threshold, and the system will retry and eventually fail.

The paper acknowledges this implicitly in Section 5, noting that GPA is interested in "self-healing when a workflow becomes stale due to UI updates" as future work. But the current system has no mechanism to update its stored subgraphs — the demonstration is recorded once and used indefinitely. This means that every visual change to the application — a button icon update, a text label revision, a layout reorganization, a font rendering change — increases the probability that GPA's matcher will fail to find the target with sufficient confidence.

The consequence. This is the classic RPA fragility problem, reframed for visual matching rather than selector fragility. Traditional RPA breaks when CSS selectors or DOM IDs change; GPA breaks when visual appearance or spatial layout changes enough to drop confidence below threshold. The paper argues that GPA's visual matching is more robust than selector-based RPA because spatial relationships and visual similarity are more stable than application-internal identifiers. This is plausible but unverified — the paper provides no measurement of how much visual change GPA tolerates (e.g., what percentage of element resizing, what degree of font change, what amount of layout shift) before the readiness checker rejects the match.

The practical consequence for deployment is that GPA workflows require the same kind of maintenance as RPA scripts — just triggered by different types of changes. When the target application is updated (new version, redesign, rebranding), someone needs to re-record the affected workflows. The difference is that GPA requires no coding expertise (just re-demonstrate the task), whereas RPA requires developer intervention to fix selectors. But the need for maintenance does not disappear — it shifts from code fixes to re-demonstration.

A more subtle consequence: GPA provides no signal about when a workflow is becoming stale. The readiness checker either passes or fails; it does not output a "degrading" warning that would allow proactive re-recording. If confidence drops from 0.95 to 0.60 over several months of gradual UI drift, the workflow continues to execute until the day it crosses the threshold and fails — at which point an automation that had been "working" suddenly stops, potentially disrupting dependent processes.

What evidence exists in the paper. None. The pilot experiment (Table 2) tested GPA on 16 tasks under what appear to be controlled conditions — same application versions, same screen configurations, minimal deliberate variation. The paper does not report experiments with: different screen resolutions between recording and execution, different application versions, different OS themes or accessibility settings, different font sizes, or different language/localization settings. The scale prior and adaptive geometric tolerance (Appendix B.3) are designed to handle window rescaling, but the paper never tests whether they successfully do so beyond analytical justification. There is no experiment where the demonstration was recorded in one environment (e.g., laptop screen at 1440×900) and executed in another (e.g., external monitor at 1920×1080).

Mitigation status. The paper acknowledges this implicitly by mentioning "self-healing when a workflow becomes stale due to UI updates" as a future research direction (Section 5). The MCP/CLI integration vision suggests a possible mitigation: VLM agents could detect when a GPA workflow has failed and either re-demonstrate it or switch to generative execution for that step. But this is speculative and unevaluated. In the current system, there is no mitigation — a stale workflow either succeeds (if the readiness checker still passes) or fails (if it doesn't), with no middle ground.

6.3 Task Set and Evaluation Scale: Pilot Study of 16 Tasks Cannot Support Generalization Claims

The assumption or constraint. The entire empirical evaluation of GPA consists of 16 desktop GUI tasks (5 simple, 11 hard), run apparently once each, with a single baseline (Gemini 3 Pro with CUA tools). The task set is not drawn from a public benchmark; it is a custom collection covering "a mix of common desktop productivity and enterprise workflows" (Section 3). The paper acknowledges this scale explicitly, calling the evaluation a "pilot study" (Section 3 title) and describing GPA as "a proof of concept project for research" (title page footnote).

The consequence. The 100% success rate reported in Table 2 is meaningful as existence proof — GPA can handle these 16 tasks — but cannot support claims about generality, robustness, or statistical reliability. Specific issues:

Task coverage is uncharacterized. We do not know whether the 16 tasks are representative of enterprise GUI automation workloads, whether they were chosen to favor GPA's approach (linear workflows without conditional logic, without dynamic content, with stable visual elements), or whether they deliberately include hard cases (dynamic popups, visually repetitive interfaces, applications with heavy animations). The paper lists task categories (email drafting, flight booking, SAP ERP form-filling, HR workflows) but does not report what fraction of steps involved visually ambiguous targets, required the SMC vs. the fast path, triggered retries, or benefited from neighbor context.
No statistical characterization. With 16 tasks and single executions, there are no error bars, no confidence intervals, and no assessment of within-task variability. If a task were re-executed 10 times with different timing conditions or minor environmental variations, would GPA still succeed every time? Would Gemini's success rate vary? The pilot data cannot answer these questions.
The baseline protocol may advantage GPA. Gemini is given "the same demonstrated video and process[ed] into textual form, and use it to guide its action autoregressively." This converts Gemini from an autonomous agent (given a goal, it plans and executes) into a demonstration-follower — a role that removes its planning advantage while retaining its generative action-selection disadvantage. GPA, by contrast, is purpose-built for demonstration replay. A protocol where Gemini receives only the high-level goal and variables (without the demonstration transcript) would test whether its planning capability compensates for its grounding errors — and would be a fairer comparison for the claim that "GPA achieves higher success rate than Gemini."
Hardware and cost asymmetry is unmeasured. GPA runs locally; Gemini runs on Google's cloud infrastructure. The 10× speedup conflates algorithmic efficiency with infrastructure differences. If Gemini were deployed on dedicated low-latency hardware, or if GPA were run on a resource-constrained edge device, the latency gap might narrow substantially. The paper does not report GPU/CPU specifications for the local machine, memory usage, or per-component latency.
No comparison to simpler deterministic baselines. The paper argues GPA is better than traditional RPA (fragile) and VLM agents (non-deterministic), but does not compare against a template-matching replay system (OpenCV-based screenshot matching), a selector-based RPA on applications where selectors are available, or a simpler version of GPA without SMC/neighbor context. Without these comparisons, we cannot assess whether GPA's SMC-based approach is actually necessary for these 16 tasks, or whether simpler methods would also achieve 100% success.

Mitigation status. The paper is transparent about the pilot scale and proof-of-concept framing. However, the abstract ("GPA achieves higher success rate with 10× faster execution speed in finishing long-horizon GUI tasks") and the paper's framing in Sections 1 and 5 present these findings as more general than the evidence supports. A practitioner reading the abstract might reasonably expect a larger-scale evaluation than what is provided. The paper would benefit from either (a) scaling the evaluation to a public benchmark with known difficulty characteristics, multiple runs, and standard baselines, or (b) more carefully qualifying the claims to match the evidence ("on these 16 tasks, in this configuration, GPA achieves 100% success while Gemini achieves 89%").

6.4 The Readiness Threshold Is a Black Box: No Empirical Calibration, No Sensitivity Analysis

The assumption or constraint. The readiness checker gates execution on a confidence score C = \tilde{p}(Z \mid \theta) \times C_{\text{spatial}} exceeding a threshold. The paper describes this threshold as "strictly defined" (Section 1) and references "a pre-computed null distribution" (abstract, introduction) as the basis for calibration. However, the threshold value is never specified numerically, the null distribution is never described, and there is no analysis of how the threshold was chosen or how sensitive system performance is to its value.

This matters because the readiness threshold is the single mechanism that converts GPA's probabilistic localization into the deterministic reliability guarantee that distinguishes it from VLM agents. If the threshold is too high, GPA will retry or fail on perfectly valid matches, reducing automation coverage. If it is too low, GPA will execute on ambiguous matches, reintroducing the non-deterministic errors that the paper critiques in VLM agents. The tradeoff is fundamental — there is no "correct" threshold independent of the deployment's tolerance for false positives (acting when it shouldn't) vs. false negatives (failing when it could have succeeded).

The consequence. A practitioner cannot assess, without extensive additional testing, whether GPA's default threshold is appropriate for their use case. Specific unknowns:

What is the false-positive rate? How often does GPA execute an action when the match is actually incorrect? The 100% success rate on 16 tasks suggests low false-positive rate on this task set, but 16 tasks × ~22 steps ≈ 350 confidence checks total — not enough to bound the error rate with any precision.
What is the false-negative rate? How often does GPA unnecessarily retry or fail when a human would judge the match as correct? The paper does not report the number of retries triggered, steps that failed after exhausting retries, or how many retries succeeded (the element appeared after a delay) vs. failed (the match was never confident enough). These metrics would characterize the downstream impact of threshold conservatism on automation completeness.
How transferable is the threshold across applications? The null distribution was presumably calibrated on specific applications, UI densities, and element types. Would the same threshold work for a dense spreadsheet interface (many small, similar-looking cells) and a sparse dialog box (few large, distinctive buttons)? The adaptive bandwidth and acceptance radius (Appendix A.2, A.3) partially address this by adjusting confidence components based on UI density, but the final threshold is a single scalar applied to the product C. There is no evidence that this scalar generalizes.
What is the sensitivity? If the threshold were changed by ±10%, how would success rate, latency (due to retries), and coverage (fraction of steps completing without retries) change? Without a sensitivity analysis, practitioners cannot tune the threshold to their risk tolerance.

What evidence exists in the paper. None. The threshold is never stated as a number. The null distribution is mentioned but not described (what data was used, what is its shape, what percentile was chosen as the threshold). There is no ROC curve, precision-recall analysis, or threshold sweep. The 100% success rate in Table 2 is consistent with a well-calibrated threshold but equally consistent with a threshold that is unnecessarily conservative for these 16 tasks (potentially triggering many retries that are invisible in the aggregate 100% success metric) or unnecessarily permissive (succeeding on these tasks but failing on others that weren't tested).

Mitigation status. None. The readiness checking mechanism is analytically justified (Appendix A.3) but empirically opaque. The paper would need at minimum: (1) the threshold value and how it was determined, (2) a characterization of the null distribution, (3) per-task retry and failure statistics to show how often the threshold gates execution, and (4) a sensitivity analysis showing that performance is stable within a reasonable range of threshold values. Without these, the readiness checker — which is marketed as a key differentiator from VLM agents — is a black-box parameter whose behavior in deployment cannot be predicted from the paper's evidence.

6.5 The MCP/CLI Integration Vision Is Unevaluated — We Don't Know Whether GPA Actually Helps VLM Agents

The assumption or constraint. A significant portion of the paper's positioning — in the abstract ("It can also be used as an MCP/CLI tool by other agents with coding capabilities so that the agent only reasons and orchestrates while GPA handles the GUI execution"), the introduction, and the conclusion — rests on the architectural claim that GPA can serve as a deterministic execution substrate for VLM agents, decoupling reasoning (handled by the VLM) from execution (handled by GPA). This claim is central to the paper's narrative that GPA is not competing with VLM agents but complementing them.

The consequence. This claim is entirely unevaluated. There is no experiment anywhere in the paper — not even a qualitative demonstration — where a VLM agent uses GPA as a tool. Specific unknowns:

How would a VLM agent invoke GPA? The paper mentions MCP/CLI integration but does not specify the interface: would the agent pass a workflow ID and variables? Select from a library of pre-recorded workflows? Generate new workflows on the fly by demonstrating them to GPA?
How would the agent handle GPA failures? When GPA's readiness checker refuses to execute a step (confidence below threshold), the agent needs to decide: retry? Switch to generative execution for this step? Ask the user for help? The paper provides no mechanism or protocol for this handoff.
What is the end-to-end reliability gain? The paper argues that VLM agents suffer from compounding per-step error probabilities, and that replacing those steps with deterministic GPA execution would improve reliability. This is a reasonable hypothesis, but without an integrated evaluation, we don't know: whether the VLM agent can correctly select the right GPA workflow for a given goal, whether the agent's orchestration decisions introduce new failure modes that offset GPA's execution reliability, or what happens at the boundary between agent reasoning and GPA execution (e.g., the agent decides to click a button that wasn't in the recorded workflow).
What is the latency/cost benefit? If a VLM agent uses GPA for execution, the per-action latency drops dramatically (sub-second local matching vs. multi-second cloud VLM inference). But the agent still needs to decide which workflow to run and when to invoke GPA, which presumably requires VLM calls. The paper provides no measurement of how many VLM calls are needed for orchestration vs. execution in a hybrid system, and therefore the claimed latency/privacy benefits are speculative.

What evidence exists in the paper. None. The MCP/CLI integration is mentioned as architectural vision in Sections 1 and 5, but is not implemented, tested, or even scoped with a protocol specification. It is a forward-looking claim, not an evaluated contribution.

Mitigation status. The paper frames this as future work (Section 5: "GPA can be extended to support fully automated, no human-in-the-loop operation: LLM agents could record workflows and perform self-healing"). This is appropriate framing for a proof-of-concept paper, but the prominence of the MCP/CLI claim in the abstract and introduction risks misleading readers into expecting an evaluation that does not exist. A minimal demonstration — a VLM agent given a high-level goal, selecting a GPA workflow from a small library, executing it, and handling one failure case — would substantially strengthen this claim even as a qualitative case study.

6.6 The Privacy Claim Is Architectural, Not Empirical — No Threat Model or Leakage Analysis

The assumption or constraint. Privacy is one of GPA's three core advertised benefits (abstract: "Privacy through fast, fully local execution") and is contrasted against VLM agents in Table 1 ("Screenshots sent to external providers" vs. "Fully local"). The paper asserts that "sensitive visual data never leaves the local machine" (Section 1). This is true by construction — GPA runs locally, uses only local models (finetuned OmniParser, IconCLIP, OCR), and does not transmit screenshots over the network.

The consequence. However, "runs locally" is not equivalent to "preserves privacy" in any formal sense. Specific unaddressed concerns:

Intermediate representations. GPA extracts OCR text, icon embeddings, and bounding boxes from every screenshot. These intermediate representations — particularly OCR text — contain the sensitive data from the screen (financial figures, personal names, customer records, confidential document contents). These representations are stored in memory and written to disk as part of the workflow template (Appendix D shows step data stored in JSON files with bounding boxes, text, and embeddings). If these files are not encrypted or access-controlled, they represent a data exfiltration vector that is arguably more structured (and therefore more easily searched and exploited) than raw screenshots.
Model provenance and supply chain. The paper uses a finetuned OmniParser for icon detection and IconCLIP for embeddings. These models were presumably trained on external data and may have been downloaded from public repositories (the detector is on HuggingFace). The paper does not discuss whether these models could contain backdoors, whether they have been audited for security, or whether the training data might include sensitive information that could be extracted via model inversion or membership inference attacks. This is not specific to GPA — it applies to any system using downloaded models — but it complicates the "fully local = private" claim.
No threat model. The paper does not define against whom privacy is being preserved. Is the threat model a malicious cloud provider (which GPA addresses by running locally)? A compromised local machine (which GPA does not address — if the machine is compromised, the OCR output and stored workflow data are accessible)? A network eavesdropper (addressed by not transmitting data, but intermediate files may be synced to cloud storage or backup services)? Different threat models require different protections, and "runs locally" only addresses the cloud-provider threat.
No leakage measurement. The paper does not measure whether the stored workflow data (step subgraphs with OCR text and icon embeddings) could be used to reconstruct sensitive screen content. If an attacker obtained the steps_data.json file, could they reconstruct the UI that was recorded? Likely yes — the file contains bounding boxes, extracted text, and element types, which together provide a structured description of what was on screen. This is less detailed than a screenshot but more structured and easily searchable.

What evidence exists in the paper. None. The privacy claim is asserted based on the architectural property of local execution but is never empirically evaluated or formally analyzed.

Mitigation status. None. The paper does not discuss encryption of stored workflow data, access control for intermediate representations, model supply-chain security, or differential privacy for embeddings. These are not unreasonable expectations for a proof-of-concept paper, but the prominence of privacy as a core benefit in the abstract and Table 1 deserves more careful qualification — particularly given that GPA's stored workflow data (OCR text, bounding boxes, element types) may actually be more privacy-sensitive in some respects than the raw screenshots that VLM agents transmit, because the extracted text is directly machine-readable and searchable.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a diagnostic reframing rather than a paradigm shift. It does not propose a new model architecture, training objective, or grounding algorithm that advances the state of the art on standard benchmarks. Instead, it challenges a foundational assumption that has guided much of the recent GUI agent literature: that more capable generative models, deployed with more sophisticated reasoning, represent the natural trajectory for GUI automation. The paper's core argument — that for replay tasks, generative action selection is not merely unnecessary but actively harmful — is a system-design insight with sharp practical consequences, even though it is currently supported only by a pilot study.

The reframing operates at two levels:

First, it separates the GUI automation problem into two distinct sub-problems that have been conflated in prior work. The VLM agent literature (CogAgent, OS-ATLAS, Aguvis, UI-TARS, and the productized systems from Anthropic, OpenAI, and Google) treats GUI interaction as a unified perception-planning-execution loop: the model perceives the screen, reasons about what to do, and generates the action. GPA argues that for a large and economically important class of workflows — those that are repetitive, structured, and demonstrable — the planning step has already been solved by the human demonstrator. What remains is a robust visual localization problem: given a recorded interaction, find the same UI elements on a possibly-transformed screen and replay the recorded actions. This is a fundamentally different technical problem than open-ended task completion, and it admits fundamentally different solutions — deterministic geometric matching rather than probabilistic next-token prediction. The consequence is not that VLM agents are obsolete, but that they are over-engineered for replay tasks, carrying unnecessary failure modes and latency costs that a purpose-built system can eliminate.

Second, it elevates determinism from a performance metric to a system property. The VLM agent literature reports success rates (89%, 93%, 95%) as statistical aggregates over task distributions, implicitly accepting that some fraction of executions will fail unpredictably. GPA's readiness-gated architecture makes a different promise: the system either acts with calibrated confidence or refuses to act, and it never guesses. The 100% success rate on the 16 pilot tasks (Table 2) is not presented as a claim about statistical superiority — with 16 tasks, that would be meaningless — but as a demonstration that deterministic replay with confidence gating can achieve perfect reliability on tasks within its scope. Whether this generalizes to larger task sets is an open empirical question, but the conceptual contribution is the shift from probabilistic reliability ("this works 93% of the time") to structural reliability ("this works when the evidence is sufficient, and fails explicitly when it isn't"). This is a qualitatively different guarantee, and one that matters enormously for enterprise adoption in regulated industries where "93% reliable" means "requires 7% manual intervention and audit overhead."

The paper also resolves a latent tension in the demonstration-based GUI automation literature. Prior work on learning GUI tasks from demonstrations (HILC, LearnAct, Instruction Agent) uses demonstrations to inform a generative agent — the demonstration provides instructions, examples, or constraints, but the agent still reasons about what to do at runtime. This hybrid approach inherits both the flexibility of generative agents and their non-determinism. GPA takes the opposite position: if you have a demonstration, use it directly as the execution template, with no generative reasoning in the loop. The paper does not empirically compare against LearnAct or Instruction Agent, so we cannot conclude which approach is superior in practice. But the conceptual contribution is clarifying that there are two distinct design points on the demonstration-based spectrum — demonstration-as-guidance (generative agents constrained by demos) and demonstration-as-template (deterministic replay of recorded interactions) — and that the latter has been underexplored relative to its practical importance for structured enterprise workflows.

Research directions that become more attractive:

Deterministic execution substrates for AI agents. The MCP/CLI integration vision, while unevaluated in this paper, is a natural research direction: can VLM agents achieve higher reliability by delegating known interaction patterns to deterministic tools like GPA, reserving generative reasoning for novel situations, exception handling, and workflow orchestration? This reframes the agent design problem from "build a better end-to-end model" to "build a better composition of deterministic and generative components," which is a more tractable engineering challenge with clearer reliability properties.
Probabilistic matching with deterministic gating. The paper's architecture — SMC for localization, multiplicative confidence gating for execution decisions — is a template for how to combine probabilistic inference with deterministic guarantees. This pattern could apply beyond GUI automation: any system that needs to operate in visually variable environments while maintaining reliability guarantees (robotic manipulation with camera-based perception, document processing with variable layouts, industrial inspection) could adopt the same "infer with uncertainty, act only with confidence" architecture.
UI-specific grounding evaluation. The paper's SMC formulation is designed for a specific failure mode — visually ambiguous elements disambiguated by spatial context — that is poorly captured by existing grounding benchmarks (ScreenSpot, SeeClick), which primarily test single-element retrieval. A new benchmark focused on context-dependent grounding (identical icons in different rows, buttons distinguished only by adjacent labels, form fields in repetitive layouts) would enable proper evaluation of methods like GPA's SMC.

Research directions that become less attractive:

Scaling VLM agents for replay-heavy enterprise workloads. If GPA's 10× speedup and 100% reliability on pilot tasks generalize (a big if), then the marginal benefit of using ever-larger VLMs to execute known workflows is negative — the VLM adds latency, cost, and failure modes without adding value over deterministic replay. Research effort is better spent on improving the deterministic substrate (better detectors, more robust geometric matching, automatic workflow maintenance) and on the orchestration layer that decides which workflow to run, rather than on improving end-to-end VLM execution for replay tasks.
Template matching and pixel-level visual comparison for GUI automation. The paper's argument — that spatial graph matching is more robust than low-level visual correspondence — is analytically well-motivated (neighbors are stable even when pixels change) and consistent with the 100% success rate (though not empirically ablated). Researchers working on visual GUI automation should consider whether template-matching baselines are still worth pursuing, or whether the field should standardize on graph-based representations for robustness against visual variation.

Follow-Up Research This Work Enables

1. Controlled grounding evaluation isolating the SMC context mechanism. The paper's central technical claim — that neighbor context via SMC resolves ambiguity that appearance-only matching cannot — is analytically justified but empirically unevaluated. A strong follow-up would construct a grounding benchmark where half the test cases involve visually ambiguous targets (identical icons, identical buttons, identical checkboxes appearing at multiple screen locations) and half involve visually distinctive targets. On this benchmark, compare four variants: (a) GPA's full SMC with neighbor context, (b) GPA's fast-path direct match (target appearance only, with entropy gating), (c) a VLM-based grounding method (e.g., SeeClick or OS-ATLAS's grounding module), and (d) a simpler geometry-based method without SMC (e.g., rigid neighbor-voting without particle filtering). The key measurements are: per-variant accuracy on ambiguous vs. distinctive targets, the fraction of cases where SMC resolves ambiguity that fast-path direct match fails on, and the latency overhead of SMC relative to the simpler methods. The pilot data (Table 2) suggests SMC is fast enough (<0.2 sec), but we need to know whether it actually works on controlled ambiguity, and at what ambiguity level it begins to fail. ScreenSpot-Pro (Li et al., 2025) provides a suitable evaluation platform with high-resolution professional screenshots containing repetitive UI elements; GPA's detector + SMC retriever could be evaluated on ScreenSpot-Pro's grounding subtask with controlled rescaling.

2. Quantifying the workflow staleness tolerance curve. The paper identifies workflow staleness as a limitation (Section 5, 6.2) but provides no measurement of how much visual change GPA tolerates. A systematic stress-test would: (a) record GPA demonstrations on a set of standard enterprise applications (email client, spreadsheet, CRM, ERP form), (b) apply controlled visual perturbations — font size changes (±2 pt, ±4 pt), application zoom levels (90%, 100%, 110%, 125%), window rescaling (from 1280×720 to 1920×1080 and back), icon set updates (replace application icons with updated versions), and deliberate layout shifts (reorder toolbar buttons, add/remove sidebar panels), and (c) measure the success rate and confidence-score distribution as a function of perturbation magnitude. This would produce a staleness tolerance curve showing, for each perturbation type, the threshold at which GPA's confidence drops below the execution gate. The practical value is immediate: it tells practitioners how much application drift is acceptable before workflows need re-recording, and it identifies which types of UI changes pose the greatest risk (layout shifts vs. appearance changes vs. rescaling). The scale prior and adaptive geometric tolerance (Appendix B.3) predict that rescaling should be well-handled, font changes should be partially handled (OCR degrades but icon embeddings remain stable), and layout reorganization should cause failures — but none of these predictions are tested.

3. The MCP/CLI integration: does GPA actually improve VLM agent reliability? The paper's most forward-looking claim is unevaluated. A minimal integration experiment would: (a) build a small library of GPA workflows covering common enterprise tasks (5–10 workflows for email, calendar, form-filling, data entry), (b) implement a VLM agent (e.g., Gemini, GPT-4V, or an open-source model) that receives a high-level goal, decides which GPA workflow(s) to invoke (and with what variable substitutions), invokes GPA via CLI, and handles the result (success → proceed; failure → retry, switch to generative execution, or escalate to user), and (c) measure end-to-end success rate, latency, and VLM API calls on a set of 20–30 tasks that mix "within-workflow" steps (handled by GPA) and "between-workflow orchestration" steps (handled by the VLM). The comparison conditions are: VLM-only (the agent does everything generatively), GPA-only (works only if the task exactly matches a pre-recorded workflow), and the hybrid. The hypothesis: the hybrid achieves higher reliability than VLM-only (because GPA eliminates per-step generative errors on known interaction patterns) while handling more task variety than GPA-only (because the VLM can handle novel situations that fall outside recorded workflows). The 10× latency advantage of GPA per step (Table 2) suggests the hybrid should also be substantially faster than VLM-only on workflow-heavy tasks. This experiment would transform the MCP/CLI integration from architectural speculation to an evaluated system contribution — and it addresses the paper's own limitation about GPA's inability to handle reasoning and adaptation.

4. Readiness threshold calibration and sensitivity analysis with a null-distribution characterization. The readiness checker (Appendix A.3) is the mechanism that converts SMC's probabilistic output into GPA's deterministic reliability guarantee, but it is entirely opaque. A rigorous calibration study would: (a) construct a labeled dataset of correct and incorrect SMC matches across diverse UIs (by deliberately perturbing element positions, introducing distractor elements, or using screen pairs where the target has genuinely moved), (b) compute the likelihood confidence $\tilde{p}(Z \mid \theta)$ and spatial confidence $C_{\text{spatial}}$ for each match, (c) plot the joint distribution of these two components for correct vs. incorrect matches, showing whether the multiplicative product C = \tilde{p}(Z \mid \theta) \times C_{\text{spatial}}$ provides good separation, (d) produce an ROC curve and precision-recall curve for the readiness gate as a function of the threshold, and (e) report the threshold value that achieves a target false-positive rate (e.g., 0.1%, 1%). A sensitivity analysis would then sweep the threshold around the calibrated value on the 16 pilot tasks and report how success rate, average retries per step, and failure rate change. If success rate is flat across a wide threshold range, the system is robust to calibration errors; if it drops sharply, the threshold is a critical hyperparameter that practitioners need to tune per-application. The paper's claim that the threshold is based on "a pre-computed null distribution" (abstract) is meaningless without describing what distribution, computed from what data, and at what percentile the threshold was set.

5. Scaling the evaluation to public benchmarks with systematic difficulty characterization and multiple runs. The pilot study of 16 tasks, run once each, cannot support generalization claims. A scaled evaluation would: (a) select a public benchmark with known difficulty characteristics — OSWorld (Xie et al., 2024) provides 369 real computer tasks across diverse applications with documented success rates for multiple VLM agents, and ScreenSpot-Pro (Li et al., 2025) provides grounding annotations on professional high-resolution screenshots — (b) for OSWorld, record a single demonstration of each task (where feasible — tasks requiring runtime reasoning would need to be filtered or handled by the hybrid system), execute GPA on each task 5–10 times with different variable values and timing conditions, and report mean success rate with confidence intervals, (c) categorize tasks by characteristics that should affect GPA's performance: presence of visually repetitive elements, need for scrolling, presence of dynamic content, degree of visual similarity between demonstration and execution screens, and (d) report per-category performance to characterize where GPA works and where it breaks. This would produce a much richer picture than the current Table 2: instead of "GPA achieves 100% on our 16 tasks," we would learn "GPA achieves X% on OSWorld's form-filling tasks, Y% on its calendar tasks, Z% on its email tasks, and fails systematically on tasks requiring date selection or conditional branching." This is the kind of characterization that practitioners need to decide whether GPA is suitable for their specific workflows.

6. Learning-based workflow maintenance: can we detect and repair stale workflows automatically? The paper identifies workflow staleness as a key limitation and mentions "self-healing when a workflow becomes stale due to UI updates" as future work (Section 5). This is a concrete research problem that GPA's architecture makes newly tractable: because GPA stores structured step subgraphs (target element + neighbors with positions, text, and embeddings), it can detect staleness by comparing the runtime UI graph against the stored subgraph and identifying which elements have changed. Detection is straightforward — low confidence scores on specific steps signal staleness. Repair is harder but well-scoped: for a step where the target element has moved or changed appearance, use the surviving neighbor nodes (which GPA already tracks) to search the runtime graph for the element that best fits the old target's geometric relationship to its unchanged neighbors. This is essentially running the SMC procedure a second time, with the target treated as missing and the neighbors used to propose a new location, then updating the stored subgraph if a confident match is found. A repair evaluation would: record workflows, apply controlled UI changes (move a button, rename a label, replace an icon), measure the pre-repair failure rate, apply the repair procedure, and measure the post-repair success rate. This closes the loop on the maintenance burden that the paper identifies as RPA's core weakness: if GPA can self-repair minor UI drift, it achieves not just better initial robustness than RPA but dramatically lower total cost of ownership.

Practical Applications and Downstream Use Cases

Enterprise desktop automation for legacy applications without APIs. This is GPA's primary intended use case and the one the pilot tasks are drawn from. Many large organizations rely on legacy desktop applications — SAP GUI, Oracle Forms, custom internal tools built decades ago — that lack modern APIs for integration. Automating data entry, report generation, or cross-application workflows in these environments currently requires either: (a) traditional RPA, which demands developer effort to define selectors and breaks when the application updates, or (b) manual labor. GPA offers a middle path: a business user (not a developer) demonstrates the task once, and GPA replays it deterministically at 10× the speed of a human (Table 2: 17.84 seconds for a 10.8-step email workflow would take a human 60–120 seconds). The 100% success rate on the pilot tasks (with the caveats about scale) suggests that for structured, linear workflows within these applications, GPA can achieve reliability comparable to traditional RPA without the implementation burden. The privacy property (fully local execution) is particularly relevant here — legacy enterprise applications often display sensitive financial, HR, or customer data that cannot legally be transmitted to cloud APIs. GPA's local-only architecture eliminates the compliance barrier that prevents adoption of VLM-based agents for these use cases.

High-volume batch processing with privacy constraints. Consider a healthcare organization that needs to process thousands of insurance claims daily by copying data from scanned PDFs (or received emails) into a claims management system. The workflow is identical for each claim — open the claim form, transcribe patient ID, diagnosis codes, and procedure details, attach supporting documents, submit — with only the variable values changing. A single demonstration of this workflow, with the LLM-based variable extraction identifying which text fields are parameterizable, enables GPA to replay the workflow for each claim automatically. The 10× speedup over a VLM agent (Table 2: ~34 seconds vs. ~329 seconds average) translates to ~105 claims per hour for GPA vs. ~11 for Gemini on a ~22-step workflow. At thousands of claims per day, this is the difference between one local machine running GPA and a cluster of cloud instances running VLM inference — with the added benefit that patient data never leaves the organization's network. The deterministic execution also means the organization can certify the workflow (each claim was processed by the same sequence of actions) in a way that is impossible with non-deterministic VLM agents. The key deployment question — which the paper does not answer — is how much visual variation across claims (different PDF renderings, different window sizes, different system loads) GPA can tolerate before the readiness checker rejects a match. The stress-test experiment proposed in Follow-Up Direction 2 would provide this boundary directly.

VLM agent execution substrate for complex multi-application orchestrations. Even without the full MCP/CLI integration being evaluated, the architectural pattern is immediately useful: a VLM agent that handles high-level workflow selection and exception handling, delegating routine GUI interactions to GPA. Consider a customer service automation: a VLM agent receives a customer email, classifies the request type (refund, order change, technical support), selects the appropriate GPA workflow from a library (or asks the user to demonstrate a new one), invokes GPA to execute the workflow within the CRM system, and only engages its generative reasoning when something unexpected happens (a popup that wasn't in the demonstration, a field that requires judgment). The VLM agent's role shrinks from "generate every action" to "orchestrate and handle exceptions," which is both more reliable (fewer generative steps → fewer opportunities for error) and faster (GPA's per-step latency of <1 second vs. Gemini's ~14 seconds). This hybrid pattern mirrors how robotic process automation is deployed today — an orchestrator manages workflow dispatch, error handling, and queue management, while individual bots execute deterministic steps — but replaces the brittle RPA bots with visually robust GPA executors. The paper's 100% reliability on pilot tasks (Table 2) and 10× speedup provide the quantitative motivation for this integration; the proposed MCP/CLI integration experiment (Follow-Up Direction 3) would provide the first validation.

When to Prefer This Method

The paper explicitly scopes GPA's applicability through its design constraints and acknowledged limitations, providing a clear decision framework for practitioners:

Prefer GPA when the workflow can be demonstrated once and replayed without runtime reasoning. Tasks that are linear sequences of known actions (click this button, type in this field, press this hotkey), where all decisions were made by the human demonstrator, fall within GPA's capability boundary. The 16 pilot tasks in Table 2 fall into this category.
Prefer GPA when reliability, privacy, and latency are hard requirements, not aspirational goals. If a 10% failure rate means unacceptable manual intervention (regulatory filings, financial transactions, healthcare data entry), GPA's deterministic gating provides structural reliability that VLM agents' statistical success rates cannot match. If screenshots contain regulated data (HIPAA, GDPR, internal financials), GPA's local-only execution eliminates the cloud-transmission privacy risk that all current VLM agents incur. If the workflow runs at high volume and latency directly determines throughput, GPA's measured 10× speedup (33.74 s vs. 329.31 s average; Table 2) compounds dramatically at scale.
Prefer a VLM agent when the task requires runtime judgment that cannot be pre-recorded. Calendar date selection (where the current month state determines the click sequence), conditional branching based on UI state (if a dialog appears, handle it; if a field is pre-filled, skip it), and workflows that span applications with unpredictable intermediate states all fall outside GPA's scope. The paper explicitly names date pickers as a failure case (Section 5). A VLM agent's generative reasoning is necessary — and appropriate — for these tasks.
Consider the hybrid when the workflow is mostly structured but contains occasional judgment points. The MCP/CLI integration architecture (unevaluated but architecturally sound) suggests a division of labor: GPA handles the 90% of steps that are deterministic replay, and the VLM agent handles the 10% that require reasoning. This captures GPA's reliability/speed/privacy benefits for the bulk of the workflow while retaining the VLM's flexibility for the hard cases. The key unknown — which the proposed integration experiment in Follow-Up Direction 3 would address — is whether the handoff between VLM agent and GPA introduces new failure modes that offset GPA's per-step reliability advantage. Until this is measured, the hybrid pattern is a promising architectural direction rather than a deployable recommendation.
Prefer traditional RPA when the application exposes stable selectors (HTML IDs, accessibility metadata) and the workflow has already been automated with RPA scripts that work reliably. GPA's primary advantage over RPA is robustness against visual/selector drift, but if the application's selectors are stable (e.g., an internal web application whose DOM structure is version-controlled and changes infrequently), the maintenance burden that GPA addresses may not exist. RPA also offers mature orchestration, logging, and exception-handling infrastructure that GPA (as a research proof-of-concept) lacks. GPA becomes preferable when RPA scripts are breaking frequently due to UI updates, or when the application lacks accessible selectors entirely (legacy desktop applications, Citrix-virtualized environments, applications rendered as images). The paper provides no head-to-head comparison with RPA, so this recommendation is based on the paper's motivation (Section 1) rather than empirical evidence.