Fara-7B: An Efficient Agentic Model for Computer Use

ArXiv: 2511.19663

🎯 Pitch

A 7B-parameter model trained purely on synthetic web trajectories (costing ~$1 each to generate) not only matches but often beats frontier models on real-world web tasks, completing them in half the steps while costing over 10× less per task. This demonstrates that the long-standing data bottleneck for computer use agents can be broken by scaling multi-agent synthetic data generation rather than model size.

---

1. Executive Summary

This paper introduces FaraGen, a scalable synthetic data generation engine that breaks the data scarcity bottleneck for computer use agents (CUAs) by automatically proposing, solving, and verifying multi-step web tasks — producing verified trajectories at roughly $1 each — and uses it to train **Fara-7B**, a compact 7B-parameter native CUA model that perceives the computer purely through screenshots and outputs atomic actions via predicted coordinates. Across four live-web benchmarks, Fara-7B achieves state-of-the-art results for its size class — including 73.5$0.025) that is more than an order of magnitude cheaper than frontier-model-based Set-of-Marks agents, establishing that small models can approach the capabilities of much larger proprietary systems when trained on high-quality synthetic trajectories distilled from multi-agent systems, though only when the training data covers the target task distribution with sufficient diversity and verified correctness.

2. Context and Motivation

The Core Problem: LLMs Can Chat, But They Can't Do — and We Lack the Data to Teach Them

The fundamental gap this paper addresses is deceptively simple: large language models have become remarkably capable conversationalists, but they remain surprisingly inept at actually operating a computer on a user's behalf. A model that can write a perfect recipe for chocolate cake, generate elegant code, or explain quantum mechanics may be utterly incapable of navigating to a grocery website, finding the ingredients for that cake, adding them to a cart, and completing the purchase — a sequence of actions that millions of humans perform daily without a second thought.

This gap is not primarily one of model architecture or training methodology. It is, as the authors argue in Section 1, fundamentally a data bottleneck:

"While the internet provides a near-infinite corpus of text training data for chat LLMs, there is no comparable data for CUA. Collecting such data with human annotators can be prohibitively expensive and slow."

The reasoning here is worth unpacking in detail. When training a chat LLM, you can scrape billions of web pages, forums, books, and code repositories — text that humans have already produced for other purposes. This data captures reasoning patterns, factual knowledge, and stylistic conventions. But it does not capture how a human interacts with a computer screen to accomplish multi-step goals. The internet's textual corpus records what people know and say; it doesn't record where they click, when they scroll, how they recover from a mis-click, or what visual cues they use to decide that the "Buy Now" button is the correct target rather than the similarly-colored "Add to Wishlist" button.

To train a CUA, you need trajectories: sequences of (observation, reasoning, action) steps that demonstrate the full lifecycle of a task — from initial instruction through intermediate planning and error recovery to final completion. These trajectories must be grounded in real, dynamic web environments because websites change constantly, contain noisy and irregular markup, and present visual information in ways that don't map cleanly to structured data. This is fundamentally different from the training data for tool-calling LLMs, which operate in clean, well-defined API environments where function signatures are explicit and deterministic.

Why This Matters: The Promise of Ubiquitous Digital Assistance

The practical stakes are significant. As the authors note in Section 1, CUAs have "immediate potential" because they address a genuine and pervasive need:

"They can navigate websites, fill forms, retrieve information, and generally improve productivity. A capable CUA can reduce tedious multi-step tasks to a single natural-language instruction, paving the way for ubiquitous personal digital assistants."

Consider the everyday cognitive load imposed by web-based tasks: booking a flight requires checking multiple airline websites, comparing prices, navigating date pickers, filling passenger information, dealing with upsells and fare class options, and confirming payment — dozens of precise interactions across pages that may have different layouts, error states, and redirect behaviors. Multiply this across the hundreds of similar tasks that knowledge workers, consumers, and citizens perform weekly, and the productivity dividend of reliable automation becomes substantial.

Moreover, the privacy and latency arguments for on-device CUA models create a hard constraint on model size. A cloud-hosted 100B+ parameter model may technically be capable of web navigation, but if every screenshot must be transmitted to a data center, processed, and returned as an action, the latency becomes prohibitive and the privacy implications (screenshots of banking pages, medical portals, private emails) become unacceptable for many use cases. The authors explicitly position Fara-7B toward this regime in Section 1:

"Fara-7B's small footprint allows for local inference, significantly reducing latency and enhancing privacy by keeping user data locally on-device."

Thus, the problem is doubly constrained: we need training data that doesn't exist at scale, and we need the resulting model to be small enough to run on consumer hardware. This is not merely a matter of scaling up existing approaches — it requires a fundamentally different data acquisition strategy.

Where Existing Approaches Fall Short

The paper identifies three distinct categories of prior work, each with critical limitations that motivate the FaraGen approach.

1. Human Annotation Doesn't Scale

The most straightforward approach — paying human annotators to record their screen interactions while completing tasks — is acknowledged as "prohibitively expensive and slow" (Section 1). A single multi-step web task can involve dozens of individual actions, each of which requires the annotator to think aloud, explain their reasoning, and document their decision process. At commercial annotation rates, a dataset of 145K trajectories (the size FaraGen produces) would cost millions of dollars and take months to collect. Even then, the diversity would be limited by what annotators can be instructed to do across a finite set of websites.

This is not merely a cost argument — it's an argument about diversity. Human annotators inevitably introduce patterns and biases based on their training, their web-browsing habits, and the specific instructions they receive. A dataset of human-collected trajectories would reflect a narrow slice of how a specific demographic interacts with a specific set of websites under specific instruction constraints. FaraGen, by contrast, starts from a broad URL corpus (ClueWeb22, Tranco) and generates tasks algorithmically, producing trajectories across 70,117 unique domains (Table 5) — a diversity level that would be impossible to achieve with human annotators at any feasible budget.

2. Existing CUA Benchmarks Are Narrow and Unrealistic

The paper provides a careful critique of the evaluation landscape in Section 4 and the related work discussion in Section 6. The problems with existing benchmarks fall into several categories:

Over-reliance on structured representations. Benchmarks like WebArena (zhou2023webarena), VisualWebArena (koh2024visualwebarena), and WebShop (yao2022webshop) provide agents with structured DOM trees or accessibility APIs. The authors note that while these "simplify action selection and grounding," they create a "persistent gap between benchmark performance and real deployment" because:

"real-world websites often contain irregular markup, dynamically generated content, personalization, and visually rich layouts" (Section 6)

An agent that learns to navigate a clean, simplified DOM representation will fail when confronted with the actual messy HTML of a production e-commerce site, where elements may be visually present but not represented in the accessibility tree, or where the accessibility tree contains hundreds of hidden, non-interactive, or misleadingly labeled elements.

Tasks are vague or non-actionable. The authors make the pointed observation that about 25% of Online-Mind2Web instructions "simply ask to 'browse', 'find', or 'view,' without specifying an actionable objective" (Section 4). "Browse the page with event planning tips on Eventbrite (easy)" is not a task — it's an instruction to scroll around. It tests navigational persistence, not goal-directed agency. A model that can endlessly browse a page will score well on such benchmarks but will be useless at "book two tickets to see Wicked: For Good at AMC Union Square, NYC."

Limited coverage of real-world task types. Existing benchmarks underrepresent (or entirely omit) task segments that humans actually perform: booking flights on specific airlines (not just Google Flights), buying tickets for specific events, applying for jobs on government portals, adding multiple items to a shopping cart with specific constraints, and cross-site compositional tasks like "find a recipe and then buy its ingredients." WebTailBench is explicitly designed to fill these gaps with 609 hand-verified tasks across 11 categories that are "grounded in tasks humans routinely perform" (Section 4).

Verifier misalignment with human judgment. The paper notes that WebVoyager's evaluation method "groups all screenshots in a single GPT-4o LLM call without including the model's final output, making it prone to distraction" (Section 4). This is a subtle but important point: if the verifier doesn't have access to what the model actually answered (only what the screenshots show), it may incorrectly conclude that a task was completed when the model's final answer was wrong, or vice versa. The authors' own verification system achieves 83.3% agreement with human judgments, with a false positive rate of 16.7% and false negative rate of 18.4% (Section 2.3) — far from perfect, but substantially better aligned than prior approaches.

3. Set-of-Marks (SoM) Agents Are Expensive and Brittle

A common approach for building web agents is the Set-of-Marks (SoM) framework: take a screenshot, overlay bounding boxes on all interactable elements (identified via the accessibility tree), and have a multimodal LLM predict which numbered element to click. The LLM never predicts coordinates directly — it operates at a level of abstraction that requires the accessibility tree as a bridge between pixels and actions.

The paper identifies two fundamental problems with this approach, detailed in Section 7:

Cost inefficiency. SoM agents backed by reasoning-intensive models (like o3 or GPT-5) expend enormous numbers of output tokens — specifically, "thinking" tokens — analyzing the accessibility tree to decide which element ID to interact with. Table 10 shows that GPT-5 as a SoM agent expends 13,000 output tokens per task on WebVoyager, while Fara-7B uses only 1,100. Since output tokens are priced much higher than input tokens (especially for reasoning models where thinking tokens count as output), this creates an order-of-magnitude cost differential: $0.316 per task for GPT-5 SoM versus$0.025 for Fara-7B, a 12.6× gap, despite the SoM agent only achieving 91.1% accuracy compared to Fara-7B's 73.5% (Table 10).

Brittleness to noisy or absent accessibility trees. As the authors note:

"accessibility trees can often be noisy or incomplete, causing models to interact with a wrong, hidden, or ineffectual elements, even leaving open the possibility of hallucinating one that does not exist" (Section 7)

This is a particularly insidious failure mode because the model has no way to verify that the element it selected actually corresponds to what it intended to click. If the accessibility tree labels a decorative div as a "button" or assigns an element ID to a visually hidden overlay, the model's action will fail silently — it clicked the right element ID, but the wrong visual target. Native CUA models, by contrast, "directly predict actions and their associated coordinates," and while they "may mis-click on areas of a screen that have no effect, this is less serious than hallucinating elements that don't exist or being distracted by buggy accessibility tree descriptions" (Section 7).

4. Prior Synthetic Data Approaches Are Programmatically Brittle

The paper briefly but importantly notes that "synthetic data generation presents an interesting alternative, but presents its own challenges due to the lack of strong pre-existing CUA models, and programmatic alternatives are brittle in the face of ambiguities and dynamic nature of the open web" (Section 1). This is worth unpacking because it explains why FaraGen's specific design choices (multi-agent architecture, Orchestrator-ledger diagnostics, three-verifier quality control) are necessary, not merely nice-to-have.

A purely programmatic approach — write a script that navigates to a website, fills a form, and records the actions — breaks down the moment it encounters a cookie consent banner, a CAPTCHA, a pop-up survey, a redirect to a regional version of the site, or any of the hundreds of dynamic behaviors that real websites employ. Similarly, a single-agent approach where one LLM attempts to both plan and execute a task suffers from the well-known failure modes of LLMs in interactive environments: getting stuck in loops, prematurely declaring success, losing track of progress across pages, and hallucinating website content.

FaraGen addresses these through its multi-agent architecture with explicit diagnostic tracking. The Orchestrator maintains a ledger of five properties (is_at_critical_point, is_satisfied, last_action_successful, is_in_loop, next_steps) at every step, which provides structured supervision over the WebSurfer's raw interactions. The paper quantifies how critical this is: "up to 25% of completed trajectories in the Shopping List (3-7 items) dataset in Table 2 are removed from training because of being is_in_loop consecutively more than three times" (Section 2.2.1). Without the Orchestrator catching these loops, the training data would be contaminated with degenerate trajectories where the model clicks the same button dozens of times.

How This Paper Positions Itself

The paper positions itself at the confluence of two emerging trends — synthetic data generation at scale and small, specialized agentic models — while rejecting the prevailing paradigm of wrapping general-purpose frontier LLMs in scaffolding to create CUAs.

The core bet, stated explicitly in Sections 1 and 7, is that the web is optimized for human visual consumption, and therefore agents should navigate it the way humans do: pixel-in, action-out, without dependence on structured representations. This is not an obvious choice. The accessibility tree approach has dominated the web agent literature precisely because it simplifies the grounding problem — predicting "click element #47" is easier than predicting "click at (423, 817)." The authors argue, with empirical evidence, that this simplification comes at too high a cost in robustness, generalization, and inference efficiency.

The paper further positions itself by drawing a direct parallel to the success of small language models in other domains. Just as Phi-3, Llama-3.2, and other compact models have demonstrated that careful data curation and distillation can produce models that punch far above their weight class, Fara-7B aims to show that the same principle applies to the agentic domain — but with the added challenge that the training data must be synthesized from scratch rather than curated from the internet.

Finally, the paper engages with the safety dimension more thoroughly than most CUA work. The critical point mechanism (Section 2.2), the refusal training data (Section 3.2, Appendix B.4), and the three risk scenarios (harmful tasks, model mistakes, harmful websites in Section 5.4) reflect a recognition that agentic models introduce new failure modes beyond those of chat models. An LLM that hallucinates a fact about the Eiffel Tower causes confusion; a CUA that hallucinates a "Confirm Purchase" button and clicks it causes financial damage. The authors explicitly note that they "strictly enforce that [the data generation pipeline] never crosses a critical point" (Section 2.2), which means Fara-7B learns from demonstrations of stopping before irreversible actions — a subtle training signal that would be absent from naive trajectory data collection.

Summary of the Gap

To synthesize: the paper addresses the triple constraint of (1) insufficient training data for CUAs, (2) the need for small, on-device-capable models for privacy and latency, and (3) the inadequacy of existing evaluation benchmarks to measure real-world agentic capability. Prior approaches fail on all three fronts: human annotation doesn't scale, existing benchmarks are narrow and unrealistic, SoM agents are too expensive and brittle, and programmatic synthetic data generation can't handle the open web's dynamic complexity. FaraGen and Fara-7B together represent a bet that a closed-loop, multi-agent synthetic data generation system — grounded in real websites, verified by multiple LLM judges, and distilled into a compact native CUA model — can break through these constraints simultaneously.

3. Technical Approach

3.1 Reader Orientation

The paper builds FaraGen, a scalable synthetic data generation pipeline that produces verified multi-step trajectories of web-based computer use tasks, and then distills these trajectories into Fara-7B, a compact 7-billion-parameter native Computer Use Agent (CUA) model that perceives webpages purely through screenshots and outputs atomic actions by predicting pixel coordinates directly. The core problem it solves is the double scarcity of CUA training data and small, efficient agentic models: human annotation of web interaction trajectories is prohibitively expensive and slow, existing benchmarks are narrow and unrealistic, and prevailing Set-of-Marks (SoM) approaches that wrap large proprietary LLMs in accessibility-tree scaffolding are brittle, expensive, and ill-suited for on-device deployment, so FaraGen provides a closed-loop synthetic alternative that automates task proposal, multi-agent task solving, and multi-verifier quality control, while Fara-7B demonstrates that a single unified model trained on this data can approach frontier-model performance at a fraction of the cost and with substantially better inference efficiency.

3.2 Big-Picture Architecture (Diagram in Words)

The overall system has two major phases — data generation (FaraGen, described in Section 2 of the paper) and model training/inference (Fara-7B, described in Section 3) — connected by a distillation step where multi-agent trajectories are converted into single-model training samples:

1. Task Proposal (Section 2.1): Takes seed URLs from large web corpora (ClueWeb22, Tranco) plus exemplar task templates and generates diverse, realistic, verifiable natural-language tasks spanning information-seeking and action-oriented categories (shopping, flights, hotels, restaurants, job applications, etc.). Outputs self-contained task descriptions.

2. Task Solving (Section 2.2): A multi-agent system built on Magentic-One where an Orchestrator agent plans, directs, and monitors a WebSurfer agent (a SoM agent backed by a frontier LLM like GPT-4o, o3, or GPT-5) that interacts with a live browser via Playwright. The Orchestrator maintains a diagnostic ledger at every step (is_at_critical_point, is_satisfied, last_action_successful, is_in_loop, next_steps) and can re-plan, retry, or force termination. An optional UserSimulator agent provides follow-up tasks or responds at critical points. Outputs candidate trajectories: sequences of (screenshot, SoM-annotated screenshot, accessibility tree, reasoning text, action).

3. Trajectory Verification (Section 2.3): Three complementary LLM-based verifiers — an Alignment Verifier (text-only, checks high-level intent satisfaction), a Rubric Verifier (generates task-specific rubrics and scores partial completion), and a Multimodal Verifier (inspects screenshots to catch hallucinations) — independently judge each candidate trajectory. Only trajectories that pass verification are retained for training.

4. Trajectory Processing for Training (Section 3.2): Verified multi-agent trajectories are converted into single-model format: SoM element IDs are replaced with center coordinates of bounding boxes, WebSurfer reasoning text becomes the model's "thoughts," and actions are mapped to Fara-7B's native action space (click at $(x,y)$, type, scroll, etc.). Each step of each trajectory becomes one training sample, with the most recent $N=3$ screenshots kept in context along with the full history of thoughts and actions.

5. Fara-7B Training (Section 3.2): A Qwen2.5-VL-7B base model is fine-tuned via supervised learning on a mixture of 1.8 million samples: trajectory steps (1.23M), grounding data (562K), refusal data (3.1K), and UI VQA/captioning data (1.8K). The model learns to predict the next thoughts and action given pixel input (screenshots) plus browser metadata (URL), without any accessibility tree at runtime.

6. Fara-7B Inference (Section 3.1): At deployment, Fara-7B receives a user query and iteratively produces (thoughts, action) pairs based on the current screenshot and the truncated history of recent observations and full history of past actions/thoughts, until it outputs a Terminate action or reaches a critical point requiring user confirmation.

3.3 Roadmap for the Deep Dive

• First, the formal problem setup and trajectory structure (Section 3.1 of the paper): equation (1) that defines what a trajectory is, equation (2) that defines what the model predicts at each step, and equation (3) that extends to multi-turn user interactions — because understanding the input-output format is prerequisite to understanding the training data construction.
• Second, the FaraGen data generation pipeline in full detail, following the sequential flow: Task Proposal (three strategies) → Task Solving (Orchestrator ledger, critical points, WebSurfer design, stopping criteria) → Trajectory Verification (three verifiers, agreement with humans) → Data Statistics (145K trajectories, costs, domain diversity).
• Third, the Fara-7B model architecture and training, including the action space (Table 7), the observation truncation scheme ($N=3$), the distillation process from multi-agent to single-model trajectories, the auxiliary data mixture and its purpose, and the training hyperparameters.
• Fourth, the WebTailBench benchmark construction, because while chronologically it appears after the data generation, its design principles (realism, coverage, objectivity, alignment, freshness) directly reflect and validate the data generation priorities.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and engineering paper whose core idea is that a carefully designed closed-loop synthetic data generation pipeline — where frontier LLMs collaboratively propose, solve, and verify web tasks — can produce training data of sufficient quality and diversity to train a small, efficient native CUA model that competes with much larger scaffolding-based systems at a fraction of the cost.

---

3.4.1 Formal Problem Setup and Trajectory Definition

The paper defines a CUA trajectory formally in equation (1) as an alternating sequence of observations, reasoning, and actions, bookended by the user's initial query. Understanding this structure is essential because every subsequent design choice — from the multi-agent task solving architecture to the training data format to the inference loop — derives from it.

A trajectory $\mathcal{T}$ consists of an initial natural-language user query $q_0$ followed by $T+1$ steps, where each step $t$ contains three components:

$\mathcal{T} = (q_0, \{o_0, r_0, a_0\}, \{o_1, r_1, a_1\}, \dots, \{o_T, r_T, a_T\})$

where $o_t$ is the observation at step $t$ (for Fara-7B at inference time: a screenshot plus browser metadata like the current URL; for the WebSurfer during data generation: a SoM-annotated screenshot plus the accessibility tree), $r_t$ is the reasoning text or chain-of-thought at step $t$ (describing the current state of the webpage, progress toward the goal, and justification for the next action), and $a_t$ is the action taken at step $t$ (a tool call with parameters, such as clicking at coordinates $(x, y)$ or typing a string).

What this defines: The complete behavioral record of an agent attempting to accomplish a task on the web — what it saw, what it thought, and what it did — from the initial instruction through to completion or termination. This is the fundamental unit of data that must be generated, verified, and used for training.

Why this form: The inclusion of $r_t$ (reasoning) between observation and action is critical because it provides the supervision signal for the model's internal planning. Without reasoning text, the model would only learn a behavior-cloning mapping from screenshots to actions, which would fail to generalize when the visual state is ambiguous. The reasoning text — which in the training data comes from the WebSurfer agent's outputs, which themselves are influenced by the Orchestrator's instructions — encodes the why behind each action: why this particular element was chosen, why the agent is scrolling rather than clicking, why it's stopping at a critical point. This is a form of chain-of-thought distillation from the multi-agent system into the single model.

The training objective is defined in equation (2). At each step $t$, the model is trained to predict the next thoughts $r_t$ and action $a_t$ conditioned on the user query and the full history of all previous steps:

$P(r_t, a_t \mid q_0, \{o_0, r_0, a_0\}, \dots, \{o_{t-1}, r_{t-1}, a_{t-1}\})$

where $P(\cdot \mid \cdot)$ is the conditional probability distribution over the token sequences representing $r_t$ and $a_t$, $q_0$ is the initial user query, and the conditioning set is the complete sequence of all previous observation-reasoning-action triples.

What it computes: For each step in a training trajectory, the model receives the entire interaction history up to that point as context and must autoregressively generate the reasoning text followed by the action representation. During supervised fine-tuning, the cross-entropy loss is computed only over the tokens of $r_t$ and $a_t$ (not over the observations, which are consumed as image inputs). The model is thus trained to continue a partial trajectory by producing the correct next step.

Why this form: Conditioning on the full history of thoughts and actions (not just observations) is essential because the reasoning chain carries state that may not be visible in the current screenshot. For example, if the agent memorized a piece of information three steps ago from a different page, the current screenshot contains no trace of it — but the history of thoughts does. The paper notes that "previous observations, thoughts, and actions provide important context for tracking progress, recognizing errors, and deciding the next steps" (Section 3.1). The alternative — conditioning only on the current screenshot — would make multi-step tasks where information crosses page boundaries nearly impossible.

For multi-turn interactions where the user provides a follow-up query, the formulation extends naturally via equation (3):

$P(r_{t+k}, a_{t+k} \mid q_0, \{o_0, r_0, a_0\}, \dots, q_1, \{o_{t+1}, r_{t+1}, a_{t+1}\}, \dots, \{o_{t+k-1}, r_{t+k-1}, a_{t+k-1}\})$

where $q_1$ is a follow-up user query that arrives after the initial task $q_0$ was completed (or paused at a critical point), and $k$ indexes the additional steps taken in response to $q_1$. The conditioning set now includes both the original user query and the follow-up query, concatenated into the history.

What it computes: The same next-step prediction as equation (2), but extended to handle the case where new user instructions arrive mid-trajectory. The model does not need a special mechanism for multi-turn interactions — the new query is simply inserted into the history stream, and the model continues predicting from there.

Why this form: This design enables natural human-agent collaboration where the user can interject, modify the task, or provide additional information. It also enables the critical point mechanism: when Fara-7B stops at a critical point (e.g., before entering payment information), the user's confirmation or additional instruction becomes $q_1$, and the model resumes from exactly where it left off, maintaining full context of everything that happened before.

Observation truncation. Because screenshots consume thousands of tokens each, keeping the full history of observations in context quickly exhausts the model's context window. The paper adopts a truncation scheme: only the most recent $N=3$ observations (screenshots) are retained in the context, while all previous thoughts and actions are preserved in full. Early experiments showed that $N=3$ "offers a reasonable trade-off between accuracy and speed and memory performance" (Section 3.1). This means the model always sees the current screenshot plus the two most recent previous screenshots, which provides enough visual context to detect motion (scrolling, page transitions) and identify dynamic content changes without the computational burden of storing every historical screenshot.

---

3.4.2 FaraGen: Task Proposal — Generating Realistic, Verifiable Tasks at Scale

The data generation pipeline begins with task proposal, which must solve a distribution-matching problem: generate tasks that reflect what real users would actually ask a CUA to do, covering both information-seeking queries (e.g., "What are the ingredients and calorie count of this smoothie recipe?") and actionable tasks (e.g., "Book two tickets to see Wicked: For Good at AMC Union Square, NYC"). The paper employs three complementary strategies, each designed to maximize a different aspect of task quality.

Targeted URL Task Proposal

This strategy produces the highest-quality, most complex tasks by starting from URLs that are pre-classified into specific high-value categories (e.g., e-commerce, entertainment, travel) and sub-categories (e.g., restaurants, movies, flights). The process involves multiple stages of LLM refinement:

1. URL classification and selection: Starting from web indices like ClueWeb22 and Tranco, URLs are classified into categories using proprietary classifiers. ClueWeb22 is preferred because it "has more coverage of high quality websites compared to Tranco" and contains a "lower fraction of corporate landing pages which tend to have a narrower scope of actionable tasks achievable on those pages" (Figure 3). The paper notes that not all URL corpora are equal: Tranco's higher proportion of corporate landing pages would bias task generation toward simple information retrieval rather than the rich transactional tasks that require multi-step interaction.

2. Intent summarization: For each selected URL, an LLM is prompted to "summarize the primary intents that users landing on the page would likely have" (Section 2.1). This step bridges the gap between a raw URL and a task concept — a URL like https://www.fandango.com/wicked-for-good-2025-238985/movie-overview is transformed into intents like "check showtimes," "compare ticket prices," or "find nearby theaters."

3. Candidate task generation and ranking: The LLM generates multiple candidate tasks per intent and ranks them against four explicit criteria: (1) achievable without requiring logins or bypassing paywalls; (2) unambiguous and fully-specified (no "book a flight" without dates or destinations); (3) useful in real scenarios (reflecting genuine user needs); and (4) automatically verifiable (the completion state can be objectively judged). The paper reports that "not enforcing these criteria resulted in up to 29% of proposed tasks being un-verifiable or un-achievable" — examples include tasks asking to make a phone call, invest in cryptocurrencies, apply for a credit card, book flights without destinations, or "read" blogs without any goal.

4. Task selection: The highest-ranked tasks that satisfy all four criteria are retained. The targeted segments include: shopping (single items and lists), travel (flights, hotels, rental cars), restaurant reservations, event ticketing, activity/itinerary planning, appointments, real estate, and job applications. Additionally, compositional tasks involving multiple websites are created — for instance, "finding a recipe and then buying its ingredients, or comparing the price of an item across two retailers" (Section 2.1). Approximately 28% of training tasks are sourced from targeted URLs.

This pipeline is inspired by AgentInstruct (mitra2024agentinstructgenerativeteachingagentic) but specialized for web-based CUA tasks. The key insight is that starting from real, high-value URLs grounds the task generation in actual web content, preventing the LLM from hallucinating tasks that aren't achievable on any existing website. The multi-stage refinement with explicit verifiability criteria filters out the ~29% of tasks that would otherwise contaminate the training data with impossible or ambiguous objectives.

Agentic URL Exploration

The complementary strategy — which produces the majority (~67%) of training tasks — is to sample URLs uniformly from web corpora and have an agent explore the website to discover what tasks can be accomplished there. The procedure works as follows:

1. Random URL sampling: A URL is drawn from a web index (primarily ClueWeb22) without category filtering. This ensures broad domain coverage beyond the targeted segments.

2. Agent exploration: A multimodal LLM agent is instantiated to traverse the website, consuming both screenshots and accessibility trees — a design borrowed from Explorer (pahuja-etal-2025-explorer). The agent generates an initial query based on what it sees on the landing page, then iteratively refines the task as it explores: "At each step, the agent refines the task based on what has been done and the current state of the page. Then, it predicts the next action required to complete the updated task" (Section 2.1).

3. Iterative task refinement: Unlike targeted URL tasks which are fully specified before solving begins, agentic exploration tasks evolve during exploration. The agent starts with a vague goal like "find something interesting on this site" and gradually narrows it to a specific, verifiable objective as it discovers what the site offers. This iterative refinement gradually improves the task quality, though the resulting tasks tend to be simpler — Table 2 shows that agent-exploration tasks average 5.1 steps (ClueWeb22) and 6.2 steps (Tranco), compared to 10–50+ steps for targeted segments like shopping lists or price comparison.

Why this tradeoff exists: Random URLs are not pre-filtered for task feasibility, so many lead to websites where only simple tasks are possible (e.g., reading an article, checking a store's hours). The agent can only discover tasks that the website actually supports, which on a random corporate landing page might be nothing more than "find the contact email." Targeted URLs, by contrast, are chosen precisely because they support rich transactional workflows. The paper acknowledges this limitation: "the complexity of these tasks is often lower than the segment-targeted ones, as indicated by the number of steps required to solve them in Table 2. This highlights the need for better automatic creation of complex tasks" (Section 2.1).

Exemplar Task Proposal

The third strategy addresses task diversity within known categories: given a bank of existing seed tasks (e.g., "buy an iPhone 15 from Best Buy"), an LLM decomposes each into a template of its primary intent, entities, and arguments, varies those fields (e.g., different product, different retailer, different constraints), and rewrites the template into a new natural-language task. For instance, the template might be "buy $\langle\text{product}\rangle$ from $\langle\text{retailer}\rangle$ with $\langle\text{constraint}\rangle$," and variation produces "buy a MacBook Pro from Amazon with at least 16GB RAM," "buy a standing desk from IKEA under $300," etc.

Why this matters: This strategy amplifies the coverage of the targeted URL approach without requiring additional URL discovery. It ensures that the training data contains multiple variations of structurally similar but semantically distinct tasks, which helps the model generalize across product types, retailers, and constraint phrasings rather than memorizing specific task instances. The paper does not report the exact fraction of tasks from this strategy, but it serves as an augmentation technique layered on top of the other two approaches.

---

3.4.3 FaraGen: Task Solving — Multi-Agent Execution with Diagnostic Oversight

Once synthetic tasks are generated, they must be solved to produce demonstration trajectories. The solving system is built on Magentic-One (fourney2024magenticonegeneralistmultiagentsolving) and Magentic-UI (mozannar2025magentic), a multi-agent framework where a planning agent (the Orchestrator) directs an execution agent (the WebSurfer) in a closed loop, with an optional UserSimulator for multi-turn extensions. The key innovation over the base Magentic-One framework is the addition of structured diagnostic oversight through the Orchestrator's ledger, which enables detection and recovery from common failure modes that would otherwise contaminate the training data.

The Orchestrator-WebSurfer Loop

The task solving proceeds as an iterative interaction:

1. Initial planning: Given a task, the Orchestrator "outlines a plan that it thinks the WebSurfer should take" and "lists some important information about the task" (Section 2.2.1). This plan is a high-level decomposition of the task into sub-goals — for example, for "book two tickets to see Wicked at AMC Union Square," the plan might be: (1) navigate to Fandango, (2) search for Wicked, (3) select AMC Union Square, (4) choose showtime, (5) select two tickets, (6) proceed to checkout.

2. Step-by-step direction: The Orchestrator issues an instruction for the first step (based on the plan) to the WebSurfer. The WebSurfer receives this instruction, the current browser state (screenshot + SoM-annotated screenshot + accessibility tree), and its own action history. It outputs reasoning text and a specific browser action (e.g., click element #47, type "Wicked" into element #12). The action is executed in a Playwright-controlled browser.

3. Observation and reporting: After execution, the WebSurfer reports back to the Orchestrator with: the observation before the action, the observation after the action (two screenshots), its reasoning, and the action it took.

4. Orchestrator diagnostic evaluation: The Orchestrator inspects the WebSurfer's action, the before-and-after screenshots, and the full action history, then updates a ledger of five diagnostic fields (Table 1). These fields are:

   • is_at_critical_point: Whether the current state involves sensitive/personal information that is about to be submitted or an irreversible action (e.g., purchasing, reserving) is about to be taken. This is a binary safety gate.
   • is_satisfied: Whether the Orchestrator believes the task has been completed. This is the primary termination condition.
   • last_action_successful: Whether the intended action produced its expected result, judged by comparing the pre-action and post-action screenshots. An action that clicks a button but the page doesn't change (e.g., due to a loading delay or the button being non-functional) would be flagged as unsuccessful.
   • is_in_loop: Whether the WebSurfer is exhibiting repetitive behavior without making progress. The paper describes two levels: a coarse-grained check for "multiple unsuccessful attempts made at doing the same thing" and a fine-grained check via last_action_successful for actions that don't change the page state.
   • next_steps: A high-level natural language description of what the WebSurfer should do next. This becomes the instruction for the subsequent WebSurfer call.

5. Control flow based on ledger: The Orchestrator uses the ledger values to decide what happens next (Table 3, discussed below). If everything is normal (no critical point, not satisfied, no loop, action successful), it provides the next_steps instruction and the loop continues. If in a loop, it re-plans. If the last action failed, it may instruct a retry. If at a critical point or the task is satisfied, it triggers termination.

6. Repeat until termination: The loop (steps 2–5) continues until the Orchestrator decides to stop, the WebSurfer decides to stop, or a step budget is exceeded.

Why this architecture: A single-agent approach — where one LLM both plans and executes — suffers from well-documented failure modes in interactive environments: getting stuck in loops, prematurely declaring success, and failing to recover from errors. The Orchestrator-WebSurfer split externalizes the monitoring and recovery logic into a separate agent that can look at the task from a higher level. The Orchestrator doesn't need to know how to click buttons or parse accessibility trees — it only needs to judge whether progress is being made and redirect when it isn't. This mirrors the human cognitive split between strategic planning and tactical execution, and the paper's quantitative evidence (Table 4) shows that its components add substantial value: adding full action history visibility to the Orchestrator improved WebVoyager success from 33% to 37%, upgrading the WebSurfer model from o4-mini to o3 improved it to 45%, and adding environment error retries improved it to 53%.

The Orchestrator's Ledger and Stopping Logic (Table 3)

The ledger is the Orchestrator's persistent diagnostic state, updated at every step by prompting the Orchestrator LLM with the WebSurfer's full action history and the previous two screenshots. Each field is predicted independently, and the combination of these five signals determines the control flow.

The stopping logic — which resolves potential conflicts between the Orchestrator's assessment and the WebSurfer's own termination signal — is defined by a precedence table (Table 3). The logic works as follows, with columns ordered by decreasing authority:

What this logic encodes: The critical point flag (is_at_critical_point) has the highest precedence — if the Orchestrator detects an impending critical point, the WebSurfer is forced to stop regardless of what it thinks, unless the WebSurfer independently recognized the critical point and terminated voluntarily (in which case the trajectory is sent directly to verification). The satisfaction flag (is_satisfied) has second precedence — if the Orchestrator believes the task is complete, the WebSurfer is forced to stop even if it wanted to continue (preventing the model from "overshooting" a completed task). The WebSurfer's own termination signal (output_terminate) has the lowest precedence — if the WebSurfer tries to stop prematurely (believing it's done when the Orchestrator disagrees), the Orchestrator rolls back the premature stop and forces the WebSurfer to continue.

Why this precedence structure: The WebSurfer, being focused on low-level execution, may prematurely declare success (e.g., after reaching a search results page without actually extracting the answer) or miss that it's about to execute an irreversible action. The Orchestrator, with its higher-level view of task progress, can override these errors. Conversely, if both agents agree that the task is complete or that a critical point is reached, the trajectory is passed to verification without forced intervention — this is the mutually-agreed termination case.

A critical design detail: when the Orchestrator forces the WebSurfer to stop (e.g., at a critical point), it does not programmatically inject a stop action. Instead, it "disables all other actions, so that the WebSurfer will naturally reason about why it is forced to stop" (Section 2.2.1). This means the training data contains natural-language reasoning from the WebSurfer about why it's stopping — e.g., "I see that the next step would require entering payment information, which is a critical point. I will terminate here and hand control back to the user." This helps Fara-7B learn not just when to stop but why.

WebSurfer Design and Action Space

The WebSurfer is a SoM agent — it relies on having the accessibility tree of the webpage to select which element to interact with. The observation it receives consists of:

1. A SoM-annotated screenshot: the raw screenshot with bounding boxes overlaid on all elements identified in the accessibility tree, each labeled with a numeric ID.
2. The accessibility tree itself: a structured representation of all interactable elements, their types (button, text input, link, etc.), their labels, and their bounding boxes.
3. The full history of actions it has previously taken.
4. Instructions/hints from the Orchestrator about the next steps.

Based on these inputs, the WebSurfer outputs:

1. A description of the state of the webpage.
2. Reasoning text about the status of the trajectory and what the right action should be.
3. A tool call representing the next action (e.g., click(element_id=47), type(element_id=12, text="Wicked")).

The action space during task solving is largely the same as Fara-7B's (Table 7), with two differences: (1) there are two separate stop actions — one for answering a question and one for completing a task — whereas Fara-7B uses a single Terminate action; (2) the action space is dynamic — for instance, if the viewport is at the top of the page, the scroll_up action is disabled.

A notable action available to the WebSurfer is Memorize, which lets the WebSurfer "record a piece of information that it can keep in its context for later" (Section 2.2.2). This is inspired by Windows Agent Arena (bonatti2024windowsagentarenaevaluating) and serves to reduce hallucinations about information that was seen on previous pages but is no longer visible. For example, when comparing prices of an item across two retailers, the Memorize action lets the WebSurfer store the price from the first site before navigating to the second, preventing hallucination of the stored value.

Why the SoM approach for data generation but not deployment: During data generation, the accessibility tree provides a clean, structured interface that allows the WebSurfer (backed by a frontier LLM) to reliably interact with webpages. This reliability is essential because the trajectories must be correct to serve as training data. At deployment, Fara-7B dispenses with the accessibility tree entirely — it predicts coordinates directly on the raw screenshot — because (1) accessibility trees are often noisy or missing in production environments, (2) including them at inference would dramatically increase input token costs, and (3) training on coordinate prediction forces the model to develop robust visual grounding that generalizes across websites with varying markup quality.

Critical Points and the UserSimulator

A critical point is defined as any binding transaction or agreement that would require explicit user permission (Section 2.2). Three categories are enumerated:

1. Using personal or sensitive information (login credentials, name, email, address, payment information) to complete a transaction.
2. Communicating in a way that a human would be expected to do (e.g., calling, emailing, applying to a job) — the agent should not impersonate the user.
3. Committing an action that is difficult to reverse — e.g., proceeding to checkout with items in cart.

The data generation pipeline "strictly enforces that [it] never crosses a critical point" (Section 2.2). When the Orchestrator detects a critical point (via its is_at_critical_point ledger field), the trajectory is terminated before the irreversible action is taken. This has a crucial training implication: Fara-7B never sees demonstrations of what happens after a critical point — purchases are never completed, forms are never submitted, login credentials are never entered. This creates a known limitation: "there is hence no training data of behaviors beyond these points, and so Fara-7B may not behave as expected" if asked to proceed past a critical point (Section 2.2).

The UserSimulator provides a mechanism for the data generation pipeline to resume from a critical point by simulating what a human would reply — e.g., providing approval to proceed or entering personal information. When activated, the UserSimulator can either:

1. Provide a response to the critical point (e.g., "Yes, please proceed with the booking. My email is user@example.com").
2. Generate a follow-up task that builds on the original task, following similar guidelines to the Task Proposal pipeline: "it has to be specific, useful, and achievable, in addition to having a natural relationship to the original task" (Section 2.2.1). Up to four follow-up tasks are sampled and ranked.

Only a small fraction of training trajectories include multi-turn extensions — the paper notes this is "an active area of exploration" (Section 2.2.1).

Task Solving Infrastructure and Yield

Each task-solving session runs in an isolated process with a headless Playwright browser instance. These sessions are parallelized as a map operation across Azure Machine Learning compute nodes. A concrete throughput number is provided: "600 completed trajectories per hour on 40 nodes running 4 browsers each, translating to about 3.75 trajectories per process per hour if GPT-4o is the WebSurfer and o4-mini is the Orchestrator" (Section 2.2.1).

Table 2 provides the critical funnel statistics across task segments. The pipeline loses the majority of trajectories at each stage:

• Error mid-solving: 54–84% of trajectories fail during execution (depending on the task segment) — the WebSurfer gets stuck, the page becomes unresponsive, or the task proves impossible.
• Completed or over-budget: Only 16–46% of trajectories reach a termination state.
• Verified as successful: Of those completed, only a subset pass verification — ranging from 3% for flights (without Browserbase) to 43% for agent exploration on ClueWeb22.

The yield is dramatically improved by Browserbase, a managed browser session service. Table 2 shows that using Browserbase improved successful trajectory yield by more than 3× for shopping (9% → 35%) and nearly 4× for flights (3% → 11%). The benefits come from more consistent browser session management, reduced failures from website anti-bot measures, and better handling of dynamically updating websites (especially critical for flights and shopping where content changes rapidly).

Domain-specific instructions provided additional gains: "by itself boosted Hotel booking yields by 10% absolute" (Section 2.2.1).

Cumulative Effect of Task Solving Improvements (Table 4)

Table 4 presents a cumulative ablation study on WebVoyager success rate as modifications are added to the task solving system:

What this shows: The baseline is a minimal setup where WebSurfer only sees the screenshot and accessibility tree, translates the Orchestrator's next_steps to Playwright tool calls, and neither agent sees more than the previous 5 actions. Each addition addresses a specific failure mode: full action history enables better tracking of progress and loop detection; stronger reasoning models (o3, GPT-5) produce more reliable action predictions; environment error retries handle transient network and page-loading failures; Browserbase improves session reliability. The total improvement from 33% to 60% represents a near-doubling of success rate, with "leveraging stronger models for WebSurfer — such as o3 and GPT-5 — accounted for about half the gains over the baseline, whereas improved context construction and fault tolerance accounted for the remainder" (Section 2.2.2, describing Table 4).

---

3.4.4 FaraGen: Trajectory Verification — Three-Verifier Quality Control

Despite the Orchestrator's is_satisfied flag checking for task completion at every step, additional verification is necessary because (1) the Orchestrator may be wrong — it may believe the task is complete when it isn't, or may miss hallucinations in the WebSurfer's output, and (2) different task types require different verification strategies — information-seeking tasks need answer accuracy checking, while action-oriented tasks need process verification. The paper uses three complementary LLM-based verifiers, each evaluating trajectories from a different perspective.

Alignment Verifier

A text-only verifier (no screenshots) that "judges whether the actions taken and final response of a trajectory aligns with the given task" (Section 2.3). This is a high-level semantic check: does what the agent did correspond to what was asked?

For transactional tasks (shopping, booking, reservations), the Alignment Verifier checks whether the trajectory correctly identified target URLs — the specific pages corresponding to the items, flights, or reservations that the task requested. The Orchestrator, upon task completion, goes back through the history to identify these target URLs (Section 2.2.1), which are then provided to the verifier. For example, if the task was "buy an iPhone 15 from Best Buy," the verifier checks whether the final target URL points to the iPhone 15 product page on Best Buy, not a different product or a different retailer.

For information-seeking tasks, the verifier checks whether the final response correctly answers the input question — essentially an LLM-as-a-judge evaluation of answer accuracy against the question.

Rubric Verifier

The Rubric Verifier addresses the problem that many tasks have partial completion — an agent might correctly navigate to the right website and find the right product but fail to add it to the cart, or might book a hotel for the right dates but at the wrong location. Binary pass/fail evaluation is too coarse.

The Rubric Verifier works in two stages:

1. Rubric generation: Given a task and the generated trajectory, the verifier predicts a list of criteria that a trajectory would need to meet to be successful, along with a point value for each criterion. For example, for "book a hotel in Paris for December 17–30 with a pool," the rubric might be: (1) hotel is in Paris (3 points), (2) dates are December 17–30 (3 points), (3) hotel has a pool (2 points), (4) reservation is successfully made (2 points).

2. Scoring: The verifier then judges how many points the model earned toward each sub-goal by inspecting the trajectory. The rubric score is calculated as the proportion of total points satisfied:

$\text{rubric\_score} = \frac{\sum_{i} \text{points\_earned}_i}{\sum_{i} \text{points\_possible}_i}$

where $\text{points\_earned}_i$ is the number of points awarded for criterion $i$ and $\text{points\_possible}_i$ is the maximum points for that criterion. The threshold for marking a trajectory as successful is set to 0.8 — trajectories with rubric scores above this threshold pass the Rubric Verifier.

What it computes: A continuous score between 0 and 1 representing the fraction of the task's sub-goals that were satisfactorily achieved, with partial credit for partial completion. This is far more informative than binary pass/fail, especially for training data curation where near-misses might still contain valuable demonstrations of correct behavior up to the point of failure.

Why this form: The 0.8 threshold represents a trade-off: it allows trajectories that missed minor details (e.g., forgot to note the hotel's star rating but booked the right hotel for the right dates) to be included in training, while excluding trajectories that fundamentally failed the task. A threshold of 1.0 would be too strict — many tasks have minor verification details that even human annotators might consider optional. A threshold below 0.8 would begin admitting trajectories where the core objective wasn't met, contaminating the training data with fundamentally incorrect demonstrations.

Multimodal Verifier

The Multimodal Verifier addresses the hallucination problem directly. LLMs can and do generate plausible-sounding answers that are not supported by what's actually visible on the screen — for instance, reporting a product's price as $29.99 when the screenshot shows$39.99, or claiming that a restaurant accepts reservations when the page says "walk-in only."

The Multimodal Verifier, inspired by OM2W (xue2025om2w), operates in two stages:

1. Salient screenshot selection: From the full trajectory, the verifier selects the most relevant screenshots ranked by how informative they are about whether the task criteria were met. This avoids overwhelming the verifier with dozens of screenshots, many of which are intermediate navigation steps.

2. Judgment: Given the selected screenshots and the trajectory's final response, the verifier makes two judgments: (1) whether the final response is fully consistent with the evidence shown in the screenshots, and (2) whether the content in the screenshots appears to satisfy the task. The paper gives a concrete example: "when asked what ingredients and how many calories are in a particular smoothie recipe, this verifier will catch hallucinations of caloric content or ingredient quantities that aren't supported by the underlying screenshot."

Why this is critical: The Alignment Verifier judges intent alignment (did the agent do what was asked?) but can't catch hallucinations where the agent claims to have done the right thing but the screenshots show otherwise. The Multimodal Verifier is the grounding check — it compares the agent's textual output against the visual evidence, catching cases where the agent confidently reports incorrect information that it hallucinated rather than read from the screen.

Verifier Agreement and Human Evaluation

The paper provides one key reliability statistic: the average agreement between verifier predictions and human judgments is 83.3%, with a false positive rate of 16.7% (verifier says success, human says failure) and a false negative rate of 18.4% (verifier says failure, human says success) — this is reported in Section 2.3. These numbers mean that roughly 1 in 6 trajectories that pass verification are actually incorrect (false positives contaminating training data with errors), and roughly 1 in 5 trajectories that fail verification were actually correct (false negatives wasting good training data). This is not mentioned as a limitation in the paper but is a significant quality issue: with 145K total trajectories, approximately 24K may be misclassified by the verifiers.

Cost and Throughput (Table 6)

Table 6 provides cost estimates per trajectory using different model configurations. The total cost is broken down by component:

What this shows: The Orchestrator and WebSurfer dominate costs (together ~$0.57–1.03 per trajectory), while the three verifiers are negligible (~$0.02–0.05 combined). Using stronger models roughly doubles the per-trajectory cost (from $0.59 with o4-mini to$1.08 with o3), but Table 4 showed that stronger models also substantially improve success rates. The cost is estimated by gathering token counts for 600 trajectories (averaging ~19 steps each) solved and verified with o4-mini, then applying market-rate token pricing.

The $1/trajectory claim is thus approximately correct for GPT-5 as the WebSurfer and o4-mini as the Orchestrator, and slightly higher ($1.08) for o3 in both roles. The paper notes that these costs are reasonable for data generation — $145K for 145K trajectories — but "likely prohibitively expensive to deploy such a system at scale" for production inference, which is precisely why the multi-agent system is used for data generation while Fara-7B (at$0.025/task) is used for deployment.

---

3.4.5 FaraGen Data Statistics (Tables 5 and 6)

The final data product is characterized in Table 5:

The "average unique domains per trajectory" of 0.5 is a striking diversity metric. It means that roughly half of all trajectories visit at least one domain that does not appear anywhere else in the dataset. This is exceptional domain diversity and speaks to the value of starting from a broad URL corpus (ClueWeb22, 70K+ unique domains) rather than a curated set of known websites. A model trained on this data has seen such a wide variety of website layouts, navigation patterns, and content types that it cannot possibly memorize website-specific behaviors — it must learn generalizable computer-use skills.

The trajectory length distribution is long-tailed (3 to 84 steps) and reflects the task proposal distribution: "targeted URL tasks are often more difficult than our tasks from agent exploration" (Section 2.4), as visible in Table 2 where shopping lists (3-7 items) average 50.6 steps while agent exploration tasks average 5.1–6.2 steps.

---

3.4.6 Fara-7B: Model Architecture, Action Space, and Observation Handling

Fara-7B is a native CUA model — it operates directly on pixel input (screenshots) and outputs atomic actions with predicted coordinates, without any accessibility tree or DOM access at runtime. This section details the model's input-output formulation, action space, and inference loop.

Observation, Thoughts, and Action Decomposition

At each step $t$, Fara-7B receives:

1. Observation $o_t$: A screenshot of the current browser viewport plus simple browser metadata — specifically, the current URL. Notably, there is no accessibility tree, no DOM, no Set-of-Marks annotation. The model must understand the visual layout of the page entirely from raw pixels.

2. History: The most recent $N=3$ observations (screenshots), plus the full history of all previous thoughts and actions (text-only, kept without truncation). The truncated observation history provides visual context (two previous screenshots show what the page looked like before recent actions), while the full thought/action history provides semantic context (what the model was thinking and doing at each step, including memorized information).

The model then produces:

1. Thoughts $r_t$: Reasoning text that "describes useful information such as the content of the webpage or the status of the trajectory, as well as what action needs to be taken next" (Section 3.1). These thoughts are generated autoregressively as a text sequence.

2. Action $a_t$: A tool call representing the next browser action, conditioned on the thoughts. The action is generated as structured text with parameters that include pixel coordinates for grounded actions.

Why this decomposition: The thoughts-before-action pattern is a form of explicit chain-of-thought reasoning that serves multiple purposes: (1) it forces the model to articulate what it sees and why it's choosing an action before committing to one, which improves action accuracy; (2) it provides interpretability — users can inspect the model's reasoning at each step; (3) during training, the thoughts come from the WebSurfer's reasoning (which was influenced by the Orchestrator's instructions), providing a rich supervision signal that encodes not just what to do but why.

The Action Space (Table 7)

Fara-7B's available actions, described in Table 7, span both standard computer-use primitives and browser-specific operations:

Key design choices:

• Direct coordinate prediction: For grounded actions (click, type, move mouse), the model predicts absolute $(x, y)$ coordinates in the screenshot. This is the core "pixel-in, action-out" formulation and what distinguishes Fara-7B from SoM agents. The model must visually localize the target element — the "Buy Now" button, the search bar, the date picker — and predict its coordinates. This eliminates dependence on accessibility trees, which can be noisy, incomplete, or misleading. The tradeoff is that coordinate prediction is harder to learn — the model must develop robust visual grounding — but the auxiliary grounding data (562K samples) is specifically designed to train this capability.

• Memorize action: This action, inherited from the task-solving system, lets Fara-7B record information for later use. The model can produce a Memorize action with a text string (e.g., "Memorize: iPhone 15 price at Best Buy is $799"), which is then stored in the thought history. This is essential for cross-page tasks where information from one page must be used on another — for instance, comparing prices between retailers or remembering a confirmation number. Without Memorize, the model would need to either (a) keep the information in its working memory across page navigations (risking hallucination) or (b) navigate back to the previous page to re-read it (wasting steps).

• Terminate action: A single unified stop action that covers both task completion and critical point handoff. This is simpler than the task-solving system's two separate stop actions (answer question vs. complete task), reducing the action space complexity for the 7B model.

• Absent actions: The paper notes that Fara-7B "is unable to drag and drop elements natively, watch or listen to video or audio content and perform tasks that require ultra-low latency such as game playing" (Section 7, Limitations). These are constraints of the action space, not fundamental model limitations — the action set doesn't include drag-and-drop primitives or media consumption operations.

Observation Truncation and Its Consequences

The observation truncation scheme — keeping only the most recent $N=3$ screenshots in context while retaining the full history of thoughts and actions — is a computational necessity: screenshots consume thousands of visual tokens each, and keeping all of them for a 50-step trajectory would easily exceed the model's context window. But the choice of $N=3$ has behavioral implications:

• Recent visual context: The model can see the current screenshot plus the two previous screenshots, providing enough visual information to detect motion (scrolling changes what's visible), page transitions (screenshots from different URLs look different), and dynamic content changes (pop-ups appearing, dropdowns expanding).

• No visual long-term memory: The model cannot "look back" at screenshots from 10 steps ago. If it needs to recall visual information from earlier in the trajectory, it must have used the Memorize action to store it in the text-based thought history. This creates a dependency: the model must learn to anticipate when information will be needed later and explicitly memorize it, since the visual record won't be available.

• Thought/action history is preserved in full: Because thoughts and actions are text (relatively few tokens), they are kept without truncation. This means the model has perfect recall of its own reasoning chain and action sequence, which provides the semantic scaffolding needed to track progress even without full visual history.

---

3.4.7 Fara-7B: Training Data Construction and Distillation

Training Fara-7B requires converting the multi-agent trajectories from the task solving system into single-model training samples. This distillation process involves several non-trivial transformations.

From Multi-Agent to Single-Model Trajectories

The multi-agent trajectories contain information from both the Orchestrator and WebSurfer, but Fara-7B is a single unified model. The distillation process extracts and transforms this information as follows:

1. Screenshots: Taken directly from the WebSurfer's observations — the raw screenshots before SoM annotation are used (since Fara-7B doesn't use SoM at inference).

2. Thoughts: The WebSurfer's reasoning text from each step becomes $r_t$. Because the WebSurfer's reasoning is influenced by the Orchestrator's instructions, this implicitly encodes information that the Orchestrator provided. For example, if the Orchestrator detected that the trajectory was at a critical point and instructed the WebSurfer to stop, the WebSurfer's reasoning at that step will contain an explanation of why it's stopping, which is consistent with the Orchestrator's diagnosis. Similarly, when the Orchestrator re-plans due to a detected loop, the WebSurfer's subsequent steps reflect the new plan.

3. Actions: The WebSurfer's actions, which reference SoM element IDs (e.g., click(element_id=47)), are converted to coordinate-based actions. For each SoM element ID, the center coordinates of that element's bounding box (obtained from the accessibility tree during data generation) are substituted. For example, click(element_id=47) becomes left_click(x=342, y=518). This is a crucial transformation: it teaches Fara-7B to predict the visual location of elements rather than relying on an ID-based indirection.

4. Multi-turn handling: If the trajectory included a UserSimulator interaction (follow-up task or critical point response), the user's simulated response is inserted into the history as an additional user query.

Why this distillation works: The WebSurfer's reasoning, which was generated with the benefit of (a) the Orchestrator's high-level guidance, (b) the accessibility tree for precise element identification, and (c) a much larger and more capable backbone model (GPT-4o/o3/GPT-5), is effectively being compressed into the training signal for the 7B model. Fara-7B learns to produce reasoning and actions that, in aggregate, achieve the same task outcomes, but without the scaffolding (no Orchestrator, no accessibility tree, no SoM annotations). This is an instance of behavioral cloning from a more capable multi-agent system — the 7B model is trained to imitate the behavior, not the internal architecture, of the larger system.

Auxiliary Task Data

In addition to the trajectory data, Fara-7B is trained on three auxiliary data types designed to address specific limitations of pure trajectory-based training:

1. Grounding Data (562,435 samples): This data trains the model's fundamental ability to localize UI elements in screenshots — a prerequisite for predicting click coordinates. The generation process works as follows:

• Start from a webpage screenshot and a selected UI element, identified either from the DOM or parsed with Omniparser (Lu2024OmniParserFP).
• Prompt a VLM to describe the highlighted element in isolation (e.g., "plus icon") and in full page context (e.g., "checkout page for a T-shirt").
• Prompt an LLM to generate candidate tasks that might require interacting with that element — from literal queries ("click the plus button") to intent-driven queries ("increase the quantity of the T-shirt by one").
• Retrieve the element's coordinates to synthesize the action label (e.g., precise click location).
• A final VLM-based verification step checks whether the predicted click location actually corresponds to the described element.

This pipeline produces over 500K samples that teach the model to map natural-language descriptions of UI elements to pixel coordinates — a critical sub-skill that trajectories alone might not sufficiently cover because trajectories contain many diverse actions, only some of which are pure grounding.

2. Refusal Data (3,149 samples): This data teaches the model to refuse harmful tasks. Two generation strategies are used:

• Screenshot-grounded harmful tasks: Given a screenshot from the trajectory data, an LLM generates a harmful task that could be attempted on that webpage (e.g., "find detailed instructions for hacking the wifi network of my neighbor").
• Exemplar-based harmful tasks: Starting from harmful task examples from WildGuard (han2024wildguard) and WebTailBench-Refusals, the LLM generates similar tasks spanning seven categories: illegal activities, deceptive tasks, high-risk/regulated domains, harassment/exploitation/hate, unsafe technical use, misinformation, and sexual content.

The paper notes that "very little [refusal] data seems to be required and with more data the model begins to refuse even benign tasks" (Appendix C) — a classic over-refusal problem where the safety training bleeds into benign task refusal. The 3,149 samples represent a carefully tuned minimum.

3. UI Screenshot Question Answering and Captioning (1,800 samples): This data addresses hallucinations in the model's understanding of webpage content. Screenshots from training trajectories are paired with question-answer pairs (e.g., "What is the price of the iPhone 15 Pro?" → "$999") and image captions. The prompts used to generate this data instruct the LLM to "select a rare question and an answer" and ensure the question "requires understanding the image directly or the text in the image" (Appendix B.3). The goal is to train the model to extract information accurately from screenshots rather than confabulating it — a capability that the Multimodal Verifier was specifically designed to check.

Data Mixture and Upsampling

The full training data mixture (Table 16) comprises 1,800,941 samples total. Trajectory steps dominate at 1,233,305 samples (68.5%), grounding data provides 562,435 samples (31.2%), and refusal + UI VQA/captioning together are only 4,949 samples (0.3%).

The paper notes that some trajectory data is upsampled — "particularly longer trajectories or trajectories for types of tasks that are less frequent (e.g., compositional tasks)" (Appendix C). This addresses the inverse correlation between trajectory length and frequency: complex tasks like price comparison (averaging 32.1 steps) are both rarer and longer than simple tasks like single-item shopping (averaging 10.2 steps). Without upsampling, the model would see far more training examples of simple navigation than of complex multi-step reasoning — biasing it toward short trajectories.

---

3.4.8 Fara-7B: Training Procedure and Hyperparameters

Fara-7B is fine-tuned from Qwen2.5-VL-7B (qwen25vl), a 7-billion-parameter vision-language model, using supervised fine-tuning (SFT) only — no reinforcement learning is applied.

Training configuration (Appendix C):

• Optimizer: AdamW (loshchilov2019decoupledweightdecayregularization) with $\beta_1 = 0.9$, $\beta_2 = 0.95$.
• Learning rate schedule: Cosine warmup for the first 10% of training steps, then an initial learning rate of $5 \times 10^{-6}$.
• Gradient clipping: Maximum norm of 1.
• Training duration: 2 epochs, or roughly 28,000 iterations.
• Batch size: 128.
• Hardware: 64 H100 GPUs.
• Precision: bf16 mixed precision.
• Distributed training: DeepSpeed Stage 3.
• Loss: Standard cross-entropy loss over the output tokens (thoughts + action). For trajectory data, "since we keep only the most recent observations at each step, we backpropagate the loss only for actions that have corresponding observations" (Section 3.2). This means that if a step's observation was truncated from the context due to the $N=3$ limit, its loss is not computed — the model is only trained on steps where it can actually see what it's supposed to be reacting to.

Grounding convention: The model adopts the grounding convention of Qwen2.5-VL and predicts absolute coordinates — the $(x, y)$ values for click, type, and move actions are directly tokenized as numbers in the output vocabulary.

Why only SFT and no RL: The paper does not explicitly justify this choice, but the comparison with UI-TARS-1.5-7B (which underwent "extensive RL training" per Section 5.3) is revealing: Fara-7B matches or exceeds UI-TARS-1.5-7B on all benchmarks despite being only SFT'd, suggesting that the quality of the supervised data — distilled from a multi-agent system with strong verifier filtering — can substitute for the exploration and reward optimization that RL typically provides. This is consistent with a broader finding in the LLM fine-tuning literature that high-quality SFT data often outperforms RL on narrow task distributions.

---

3.4.9 WebTailBench: Benchmark Construction Principles

While WebTailBench is primarily described in Section 4 as an evaluation artifact, its construction principles are directly informed by the data generation priorities and are therefore part of the technical approach. The benchmark comprises 609 hand-verified tasks across 11 categories, built on four principles:

1. Realism: Tasks are sourced from high-traffic websites that reflect actual human information needs. The paper contrasts WebTailBench tasks with existing benchmarks — for example, a WebTailBench jobs task is "Help me apply for a police officer position in Soldotna, AK on their government jobs portal," whereas Online-Mind2Web has "Search for a job in Miami, Florida, in Human Resources on target. (hard)" — the former is a concrete, achievable goal while the latter is vague browsing.

2. Coverage: The benchmark includes both breadth (8 single-skill subcategories covering shopping, flights, hotels, restaurants, activities, ticketing, real estate, and jobs) and depth (3 multi-step/cross-site subcategories: shopping lists, comparison shopping, and compositional tasks). Each subcategory has a sufficient number of tasks (48–80 per single-skill category, 51–57 per multi-step category) to enable statistically meaningful per-category analysis — a deliberate counter to benchmarks like Online-Mind2Web which had "only three tasks pertaining to flights" (Section 4).

3. Objectivity: Tasks are goal-oriented with clear success criteria. If a task fails due to factors beyond the model's control (e.g., sold-out bookings, discontinued products), the model is given full credit if it properly reports the unavailability. This avoids penalizing models for environmental non-stationarity — a common problem in live-web evaluation where tasks can become impossible between data collection and evaluation.

4. Alignment with human judgment: The verification system used for WebTailBench (which the authors release alongside the benchmark) is the same three-verifier system described in Section 2.3, which achieves 83.3% agreement with human annotators. By releasing the verifier, the authors provide a standardized evaluation protocol that future models can be assessed against — addressing the problem that different papers often use different LLM judges for evaluation, making cross-paper comparisons unreliable.

5. Freshness: Tasks are designed to be valid at least through November 2025, with periodic refreshes planned. Time-sensitive segments (flights, hotels, ticketing) specify exact dates far enough in the future to remain valid.

The 111-task WebTailBench-Refusals subset applies the same principles to safety evaluation, with tasks spanning seven harmful categories (Table 15) designed to test whether a CUA will refuse to perform dangerous web-based actions.

4. Key Insights and Innovations

Innovation 1: Data Scarcity for CUAs Is Not a Cost Problem — It's a Verification Problem

The most fundamental conceptual move in this paper is redefining why CUA training data is scarce. The conventional diagnosis — shared by virtually all prior work cited in Section 6 — is that human annotation of web interaction trajectories is "prohibitively expensive and slow" (Section 1). This frames the problem as a collection cost bottleneck: if we had more annotators or more budget, we could solve it.

FaraGen's architecture reveals a deeper problem through its three-verifier quality control system. The paper's funnel statistics in Table 2 are diagnostic: across task segments, 54–84% of attempted trajectories fail during execution and never reach completion, and of those that do complete, only a fraction pass verification (e.g., 9% for single-item shopping, 3% for flights without Browserbase, 35% for shopping with Browserbase). This means that if you simply paid human annotators to record their screen interactions, the vast majority of what they produced would be unusable — the annotator would get stuck, encounter CAPTCHAs, make errors, or produce degenerate looping behavior. Even the frontier models used by FaraGen (GPT-4o, o3, GPT-5), operating within a multi-agent system specifically designed to catch failures, fail to produce valid trajectories most of the time.

The innovation is recognizing that the bottleneck is not data generation volume but data quality assurance at scale. The multi-agent architecture with the Orchestrator's ledger — tracking is_in_loop, last_action_successful, is_at_critical_point, and is_satisfied at every step — is not primarily a task-solving improvement. It's a rejection sampling mechanism: it detects and filters out degenerate trajectories during generation rather than after, and the three-verifier pipeline provides a second pass that catches hallucinations and misalignments the Orchestrator missed. The paper quantifies this: "up to 25% of completed trajectories in the Shopping List (3-7 items) dataset in Table 2 are removed from training because of being is_in_loop consecutively more than three times" (Section 2.2.1). Without the ledger, those 25% would contaminate training data with repetitive clicking behavior that Fara-7B would learn to imitate.

This reframes the CUA data problem from "we need cheaper annotation" to "we need automated quality control that can operate at the scale of 145K trajectories with at most ~17% false positive rate." The three-verifier design (Alignment, Rubric, Multimodal) is the paper's answer to that reframed problem, and the 83.3% human-verifier agreement statistic — while imperfect — establishes a baseline for what automated CUA quality assurance can achieve. This is a fundamentally different framing from agent benchmarks like WebArena (zhou2023webarena) and Mind2Web (deng2023mind2webgeneralistagentweb), which treat data collection as a one-time human annotation effort and evaluation as a separate problem, without closing the loop between generation and verification.

The distinction is incremental relative to AgentInstruct (mitra2024agentinstructgenerativeteachingagentic), which also generates synthetic data with verification, but fundamental relative to the CUA field, which had not previously recognized verification as the primary bottleneck rather than collection cost. Evidence for this being non-obvious: the paper itself reports that "not enforcing [verifiability] criteria resulted in up to 29% of proposed tasks being un-verifiable or un-achievable" (Section 2.1), meaning that even task proposal — before any execution — has a ~30% failure rate that naive approaches would miss entirely.

---

Innovation 2: The "Critical Point" Concept Reframes CUA Safety from Post-Hoc Refusal to Architectural Guardrails

Safety in LLMs is predominantly addressed through alignment training — teaching the model to refuse harmful requests or produce safe outputs. This is the approach behind RLHF, constitutional AI, and refusal datasets like WildGuard (han2024wildguard) that the paper uses. But for CUAs, the paper argues — implicitly through its architecture rather than explicit argument — that this paradigm is insufficient.

The insight is that CUAs introduce a new failure mode with no chat-model analog: model mistakes during benign task execution. The paper's three risk scenarios (Section 5.4) distinguish harmful tasks (user asks to buy illegal drugs), harmful websites (prompt injections on visited pages), and model mistakes (agent sends an email to the wrong recipient while executing a legitimate email task). Refusal training handles the first two categories. It cannot handle the third — the model isn't refusing anything; it's making an execution error with real-world consequences.

The critical point mechanism is the paper's architectural answer to this third category. It is not a training-time safety intervention. It is a runtime behavioral constraint embedded in the training data itself: the data generation pipeline "strictly enforces that [it] never crosses a critical point" (Section 2.2), so every trajectory in the training data terminates before any binding transaction, payment submission, personal information entry, or irreversible action. Fara-7B learns from demonstrations that the correct behavior at such junctures is to stop and hand control to the user — not through explicit refusal language ("I cannot do that"), but through a structural property of all successful trajectories.

The evidence that this works is in the critical point evaluation (Section 5.4): Fara-7B stopped before the critical point in 19 of 23 synthetic tasks, and the four failures involved low-impact reversible actions (marking email as read, liking a post) or a single case of publishing without confirmation. More telling is what doesn't happen: Fara-7B never attempts to fill in payment information, never enters personal details unprompted, never clicks "Confirm Purchase" — because it has never seen a demonstration of doing so.

This is conceptually distinct from OpenAI's Operator (openaioperator), which the paper cites as inspiration. Operator implements critical-point-like behavior through system-level guardrails and prompt engineering at inference time. Fara-7B embeds it in the model weights through training data construction. The difference is between a runtime policy check that can be bypassed and a learned behavior that the model doesn't know how to perform in the first place. The paper's limitation — "there is hence no training data of behaviors beyond these points, and so Fara-7B may not behave as expected" (Section 2.2) — is actually the mechanism: the model has no behavioral policy for crossing critical points because it was never trained on such trajectories.

This innovation is fundamental for CUA safety architecture even though it's an incremental extension of the Operator system card's risk taxonomy. Prior CUA work (cited in Section 6) predominantly focused on benchmark task performance, with safety mentioned as an afterthought or addressed through refusal training alone. The critical point mechanism provides a different primitive: instead of teaching the model to refuse dangerous actions when asked, teach it to pause for authorization before any action that could be dangerous regardless of intent. The UserSimulator (Section 2.2) — which simulates user responses at critical points to resume data generation — is the training infrastructure that makes this possible, by showing the model that stopping is normal and expected, not a failure.

---

Innovation 3: Successful Distillation from Multi-Agent Systems Requires the Teacher's Diagnostic Reasoning, Not Just Its Actions

A naïve approach to training a CUA model from synthetic data would be: have a large LLM solve web tasks and collect (screenshot, action) pairs; train a small model to predict actions from screenshots via behavior cloning. This would fail for the same reasons that behavior cloning fails in robotics and reinforcement learning: the student learns a reactive policy that cannot recover from errors because it never saw the teacher's recovery strategies embedded in the action sequence alone.

Fara-7B's training data construction makes a subtle but crucial choice that the paper does not trumpet but that the architecture reveals: the model is not trained to predict only the WebSurfer's actions. It is trained to predict the WebSurfer's reasoning text (which encodes the Orchestrator's diagnostic signals) followed by the action. The distillation is not just action-to-action but reasoning-to-reasoning, where the reasoning carries structured information from the multi-agent system's control logic.

Concretely: when the Orchestrator detects a loop (is_in_loop = True) and re-plans, the WebSurfer's next step contains reasoning like "I notice I've attempted to click this button multiple times without success. Let me try searching for the product using the search bar instead." When the Orchestrator detects a critical point, the reasoning says "I see this is the checkout page requiring payment information. Since this is a critical point, I will stop here." When the Orchestrator identifies the task as complete, the reasoning articulates why — "I have found the requested hotel, confirmed availability for the specified dates, and the booking summary matches all criteria."

Fara-7B learns to generate these reasoning patterns, not because it has an internal Orchestrator, but because the training objective forces it to predict the reasoning tokens before the action tokens. This is a form of implicit chain-of-thought distillation where the "chain" encodes recovery strategies, termination conditions, and safety checks that a pure action-prediction model would have no signal for. When Fara-7B encounters a situation similar to one where the teacher system looped, it generates reasoning that diagnoses the loop and proposes an alternative strategy — not because it was explicitly trained on loop detection, but because the training data's reasoning text at those points contains loop-diagnostic language that the model learned to reproduce.

This connects to the paper's finding that Fara-7B achieves strong performance with only SFT (no RL), while UI-TARS-1.5-7B (qin2025uitars) required "extensive RL training" (Section 5.3) and performs worse. The likely mechanism: UI-TARS learns action policies through trial-and-error reward optimization, which is sample-inefficient and may converge to strategies that work on training websites but don't generalize. Fara-7B learns through supervised imitation of a system that already knows how to recover from errors, because the Orchestrator explicitly identified errors and the WebSurfer's reasoning articulated the recovery. The SFT signal is richer than it appears because the "labels" (reasoning text + action) carry the multi-agent system's diagnostic intelligence compressed into natural language.

This innovation is fundamental for agentic distillation and has implications beyond CUAs. It suggests that when distilling from multi-agent or scaffolded systems into single models, the key information to transfer is not the final action but the reasoning that connects diagnosis to action — the "why" that makes the "what" generalizable. The paper demonstrates this implicitly through its architecture rather than stating it as a methodological contribution, but it is arguably the mechanism that makes the entire Fara-7B approach work at 7B scale. Evidence: Figure 8 (left) shows that with only 1% of the training data, performance drops substantially — this would not happen if the model were simply memorizing action sequences from common website layouts, but it makes sense if the model relies on reasoning-pattern diversity for generalization.

---

Innovation 4: WebTailBench Operationalizes "Realism" as a Measurable Property of CUA Benchmarks

The paper introduces WebTailBench as a new benchmark, but the intellectual contribution is less the benchmark itself and more the criteria by which CUA benchmarks should be evaluated. The four principles — realism, coverage, objectivity, and alignment with human judgment — are not novel individually, but the paper's critique of existing benchmarks reveals a systematic pattern of failure modes that the field had not articulated clearly.

The key diagnostic moves in the critique:

The "vague task" problem. The paper points out that "about 25% of Online-Mind2Web instructions simply ask to 'browse', 'find', or 'view,' without specifying an actionable objective" (Section 4). This is not a minor annotation oversight — it means that fully a quarter of a widely-used CUA benchmark tests whether a model can scroll around a page, not whether it can accomplish a goal. A model that achieves 100% on such tasks is not a computer use agent; it's a persistent scroller. By making this observation quantitative (25%), the paper provides a diagnostic that future benchmark designers can use to audit their own task sets.

The "brittle verifier" problem. The paper notes that WebVoyager's evaluation method "groups all screenshots in a single GPT-4o LLM call without including the model's final output, making it prone to distraction" (Section 4). This is a specific technical critique: if the verifier sees only screenshots, it may conclude a task was completed because the final screenshot looks correct, even though the model's textual output was wrong (or vice versa). The paper's three-verifier system — which includes a text-only alignment check that specifically examines the model's answer — addresses this by design. The insight is that CUA verifiers must evaluate the agent's stated conclusion against both the task specification and the visual evidence, not just one or the other.

The "penalize for environmental non-stationarity" problem. WebTailBench gives full credit if a model properly reports that a task is impossible due to factors beyond its control (sold-out bookings, discontinued products). Existing benchmarks penalize this outcome, conflating model capability with environmental drift. This is a conceptual distinction that matters enormously for live-web evaluation, where task feasibility changes between benchmark creation and model evaluation. It reframes evaluation from "did the model succeed?" to "did the model behave correctly given the current state of the website?" — which is what a user would actually care about.

These critiques together form a diagnostic framework for CUA benchmark quality that goes beyond WebTailBench itself. The paper's release of its verification system alongside the benchmark (Section 2.3) operationalizes the alignment-with-human-judgment principle by providing a standardized evaluation protocol with known agreement statistics (83.3% with humans). This addresses a persistent problem in the CUA literature where different papers use different LLM judges and report incomparable results.

The innovation is conceptual rather than technical — it's about what properties make a CUA benchmark meaningful — and it's fundamental for the field's evaluation methodology even though the benchmark itself will age as websites change. The paper's positioning of this as a complement to existing benchmarks (not a replacement) is also notable: WebTailBench explicitly targets under-represented task segments (flights, hotels, ticketing, real estate, jobs, compositional tasks) that existing benchmarks lack, rather than trying to be a universal evaluation. This modular approach to benchmark design — fill gaps rather than build monoliths — is arguably more sustainable and useful than creating yet another all-encompassing benchmark.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on four live-web benchmarks: WebVoyager (he2024webvoyagerbuildingendtoendweb), Online-Mind2Web (xue2025om2w), DeepShop (lyu2025deepshop), and the authors' newly introduced WebTailBench (609 hand-verified tasks across 11 categories, Section 4). WebVoyager required filtering: approximately 48 tasks were removed as impossible, and another 50 required date updates to remain achievable (Section 5.1.1). For safety evaluation, AgentHarm-Chat (andriushchenko2024agentharm) and WebTailBench-Refusals (111 handcrafted harmful tasks across 7 categories) are used (Section 5.4, Table 14). Grounding is evaluated on ScreenSpot-V1 and ScreenSpot-V2 (Cheng2024SeeClickScreenSpot; Li2025ScreenSpotPro) (Section 5.2).

• Base model(s). Fara-7B is fine-tuned from Qwen2.5-VL-7B (qwen25vl), a 7-billion-parameter vision-language model (Section 3.2). The paper explicitly states that UI-TARS-1.5-7B (qin2025uitars) — the primary same-size comparison — is "based on the same Qwen2.5-VL model as Fara-7B" (Section 5), making architectural comparisons direct. For the SoM agent baselines, the backbone models are GPT-4o (hurst2024gpt4o), o3 (openai2025o3o4mini), and GPT-5 (openai2025gpt5). GLM-4.1V-9B-Thinking (glm2025) is included as an open-source SoM comparison, and OpenAI computer-use-preview (openaioperator) represents a much larger proprietary CUA model.

• Metrics. The primary metric is success rate (%), defined as the fraction of tasks for which the agent's trajectory is judged successful by the benchmark's designated LLM-based verifier. For WebVoyager and DeepShop, GPT-4o serves as the judge with the official respective prompts; for Online-Mind2Web and WebTailBench, o4-mini serves as the judge (Section 5.1.2). Safety is measured as refusal rate (%) — the fraction of harmful tasks where the model correctly refuses to comply (Section 5.4, Table 14). Grounding is measured as accuracy (%) on element localization (Section 5.2, Table 13). Cost efficiency is measured in dollars per task, computed from token counts multiplied by market-rate token pricing (Appendix A, Tables 10 and 12). Actions per task (mean ± standard deviation) provides a proxy for interaction efficiency and latency.

• Baselines. The paper compares against two categories of agents (Section 5). SoM agents: GPT-5 SoM Agent, o3 SoM Agent, GPT-4o SoM Agent (all using "the same implementation to parse and visualize the set-of-marks," iteratively prompting the WebSurfer without an Orchestrator), and GLM-4.1V-9B-Thinking (an open-source SoM agent). CUA models: UI-TARS-1.5-7B (qin2025uitars), evaluated in the OSWorld (xie2024osworldbenchmarkingmultimodalagents) environment, and OpenAI computer-use-preview (openaioperator). For safety evaluations, the same SoM and CUA baselines are compared on refusal rates (Table 14). For grounding, the base model Qwen2.5-VL serves as the baseline (Table 13).

• Generation budget / compute accounting. Each trajectory is capped at a maximum of 100 steps across all online benchmarks; if the agent does not terminate by step 100, the task is counted as wrong (Section 5.1.1). Compute efficiency is measured via token counts: input tokens per task (cumulative across all steps, including screenshot tokens and action history), output tokens per task (including thinking tokens for reasoning models), and actions per task (a proxy for wall-clock latency). Cost is computed by multiplying these token counts by market-rate pricing per token (Appendix A). The paper emphasizes that output tokens are far more expensive than input tokens for reasoning models (where thinking tokens count as output), making token-count comparison essential for fair cost assessment.

• Cross-validation / statistical protocol. To address the high variance of live-web evaluation, the paper runs three independent evaluations for each online benchmark and reports the mean success rate with ± standard deviation (Table 19 in Appendix D.3). For pass@k analysis (Figures 1 and 6), the paper computes "whether or not a model could achieve the same task with k independent runs, averaged across all $\binom{3}{k}$ combinations if $k < 3$" (Section 5.1.2). Environment errors (e.g., connection drops, page loading timeouts) trigger a retry: "we retry the trajectory up to five times, but only when environment errors occur. Complete yet incorrect trajectories are never retried" (Section 5.1.1). Retries start from a fresh browser session without retaining prior state. If a task cannot be completed even after retries, it is counted as wrong in the average. All tasks across all models use the same retry logic, ensuring fair comparison. For human evaluation, "a trusted third party, Browserbase" independently verified Fara-7B's WebVoyager trajectories with human annotators, establishing 62% accuracy that is "directly comparable" to other open-source models evaluated under identical conditions (Section 5.1.2).

---

Main Quantitative Results

Agentic Evaluations on Live-Web Benchmarks (Tables 9 and 19)

Headline results. Across four web benchmarks, Fara-7B achieves the best success rate among all 7B-scale models and compares favorably to larger SoM agents (Table 9). On WebVoyager, Fara-7B scores 73.5% (±1.0), outperforming all SoM agents except GPT-5 (90.6%) and all CUA models including OpenAI computer-use-preview (70.9%) and UI-TARS-1.5-7B (66.4%). On Online-Mind2Web, Fara-7B achieves 34.1% (±3.7), comparable to GPT-4o SoM (34.6%) and GLM-4.1V-9B-Thinking (33.9%). On DeepShop, Fara-7B scores 26.2% (±2.0), substantially ahead of UI-TARS-1.5-7B (11.6%) and GPT-4o SoM (16.0%). On WebTailBench, Fara-7B achieves 38.4% (±0.7), more than doubling UI-TARS-1.5-7B (19.5%) and substantially exceeding OpenAI computer-use-preview (25.7%) and GPT-4o SoM (30.8%). The standard deviations (Table 19) show that Fara-7B's performance is not only strong in expectation but also stable across runs: its standard deviation of ~1.0 on WebVoyager and below 2.0 on DeepShop and WebTailBench is "comparable to or smaller than that of both larger SoM baselines and other 7B models" (Appendix D.3).

What these numbers show. Fara-7B is unambiguously the state-of-the-art model in the 7B parameter class across all four benchmarks. The comparison with UI-TARS-1.5-7B — which shares the same base model (Qwen2.5-VL-7B) but underwent "extensive RL training" rather than Fara-7B's SFT-only approach (Section 5.3, Figure 8) — demonstrates that Fara-7B's data pipeline produces more effective training signals than RL-based post-training, at least for these benchmarks. The comparison with GPT-4o SoM is notable because Fara-7B outperforms it on three of four benchmarks (WebVoyager: 73.5 vs. 65.1; DeepShop: 26.2 vs. 16.0; WebTailBench: 38.4 vs. 30.8), and essentially ties on Online-Mind2Web (34.1 vs. 34.6). This means a 7B native CUA model is competitive with a GPT-4o-based SoM agent — a system that wraps a frontier proprietary LLM in accessibility-tree scaffolding — despite Fara-7B having no access to accessibility trees, no Orchestrator, and roughly two orders of magnitude fewer parameters in its core reasoning component.

Comparison with GPT-5 and o3 SoM agents. The frontier SoM agents achieve substantially higher scores: GPT-5 SoM reaches 90.6% on WebVoyager and 60.4% on WebTailBench, while o3 SoM reaches 88.7% and 52.7% respectively. However, Tables 10 and 12 reveal that these gains come at a steep cost premium. On WebVoyager, GPT-5 SoM costs $0.316 per task (12.6× more than Fara-7B's$0.025) and expends 13,000 output tokens (11.8× more than Fara-7B's 1,100). On WebTailBench, the gap widens: GPT-5 SoM costs $0.595 (8.6× more) and o3 SoM costs$0.948 (13.7× more). The pass@k analysis in Figures 1 and 6 shows that Fara-7B's pass@3 — running the model three times and succeeding if any run succeeds — reaches competitive levels, suggesting that Fara-7B's single-run accuracy understates its potential and that "well-known post-training techniques beyond just the supervised finetuning" could close the gap further (Section 5.1.2).

Cost and Efficiency Analysis (Tables 10 and 12)

Headline efficiency numbers. On WebVoyager (Table 10), Fara-7B completes tasks in 16.5 ± 21.1 actions on average — comparable to GPT-5 SoM (16.6 ± 22.1) and GPT-4o SoM (16.6 ± 22.8), and substantially fewer than UI-TARS-1.5-7B (41.3 ± 37.2) and OpenAI computer-use-preview (38.0 ± 34.2). Fara-7B uses 124K input tokens and only 1.1K output tokens per task, compared to 408K/2.2K for UI-TARS-1.5-7B (3.3× more input tokens) and 295K/2.3K for OpenAI computer-use-preview (2.4× more input tokens). The per-task cost of **$0.025** is less than one-tenth the cost of the cheapest SoM agent (GPT-4o at$0.302) and less than one-third the cost of UI-TARS-1.5-7B ($0.082).

What these numbers show. Fara-7B achieves two distinct efficiency advantages over its same-size competitor (UI-TARS-1.5-7B). First, it completes tasks in fewer than half the actions (16.5 vs. 41.3), meaning lower latency and fewer cumulative context tokens — this is visible in Figure 1 where Fara-7B is positioned further left on the actions-per-task axis. Second, while both models share the same base architecture and per-token pricing, Fara-7B uses substantially fewer input tokens per task (124K vs. 408K), likely because its shorter trajectories mean less cumulative screenshot history is accumulated. The gap between Fara-7B's 1.1K output tokens and GPT-5 SoM's 13.0K output tokens (and o3's 10K) explains the order-of-magnitude cost differential despite Fara-7B's lower accuracy: reasoning-intensive SoM agents spend the vast majority of their compute budget on "thinking" about which accessibility tree element ID to interact with, a cost that Fara-7B eliminates entirely by predicting coordinates directly.

On WebTailBench (Table 12), the efficiency story is slightly different: Fara-7B takes 41.1 ± 33.1 actions per task — more than on WebVoyager and comparable to UI-TARS-1.5-7B (41.1 ± 32.4) — but still achieves roughly twice the accuracy (38.4% vs. 19.5%) at roughly half the cost ($0.069 vs.$0.133). This suggests that WebTailBench tasks are inherently longer (requiring more steps even for the more capable model) but that Fara-7B's per-step efficiency advantage (lower token consumption per action) still translates to overall cost savings. The SoM agents' costs on WebTailBench are striking: o3 SoM costs $0.948 per task (13.7× Fara-7B) while achieving only 52.7$0.595 (8.6×) for 60.4% accuracy.

WebTailBench Per-Category Breakdown (Table 11)

Headline category results. Table 11 provides the most granular picture of Fara-7B's capabilities and limitations. On five of the eight single-skill subcategories, Fara-7B matches or exceeds all baselines except the frontier SoM agents (GPT-5 and o3): Shopping (52.4% vs. GPT-4o SoM 38.1%, UI-TARS 41.1%); Flights (37.9% vs. GPT-4o SoM 11.1%, UI-TARS 10.5%); Hotels (53.8% vs. GPT-4o SoM 31.4%); Restaurants (47.4% vs. GPT-4o SoM 47.4% — tied); Activities (36.3% vs. GPT-4o SoM 41.7% — slightly behind). On the remaining single-skill categories, performance is more modest: Ticketing (38.6%), Real Estate (23.6%), and Jobs/Careers (28.0%). On the three multi-step/cross-site subcategories, Fara-7B "out-competes all other baselines besides GPT-5 and o3": Shopping Lists (49.0% vs. GPT-4o SoM 17.0%, UI-TARS 20.9%); Comparison Shopping (32.7% vs. GPT-4o SoM 27.5%, UI-TARS 8.8%); Compositional Tasks (23.0% vs. GPT-4o SoM 26.7%, UI-TARS 9.1%).

What this breakdown reveals. The paper's claim that "modest sums of high quality data can take a small model a long way at mastering a particular skill" (Section 5.1.3) is supported by the flights and hotels results. Fara-7B's training corpus contains "fewer than 4k flight and hotel tasks each" (Table 2 shows ≤1K flight trajectories used in training, 3.6K hotel trajectories), yet on these categories it achieves 37.9% and 53.8% respectively — within 3 points of o3 SoM on flights (39.2%) and within 3 points on hotels (56.4%). This is remarkable efficiency: with roughly 1K–3K training trajectories per category, Fara-7B approaches the performance of a SoM agent backed by o3 (one of the most capable reasoning models available) on those same tasks.

Where Fara-7B falls short. The frontier SoM agents (GPT-5, o3) maintain substantial leads on the three multi-step/cross-site categories (Shopping Lists: GPT-5 66.0%, o3 62.7% vs. Fara-7B 49.0%; Comparison Shopping: GPT-5 67.3%, o3 59.1% vs. Fara-7B 32.7%; Compositional: GPT-5 51.5%, o3 39.4% vs. Fara-7B 23.0%). The paper attributes this to "the benefits of additional thinking space for planning and executing long-horizon tasks" (Section 5.1.3) — essentially, the 7B model's limited context window and reasoning capacity constrain its ability to maintain coherent plans across very long trajectories (these categories involve 32–51 steps on average in training, per Table 2). The Real Estate category shows "unexpectedly low results across the board" (GPT-5 only 34.0%), which the authors attribute to "the defensive behavior of websites in that segment against bots" (Section 5.1.3) — an environmental limitation rather than a model capability limitation.

UI-TARS-1.5-7B comparison in detail. The per-category breakdown reveals the magnitude of Fara-7B's advantage over its same-size competitor. On 10 of 11 categories, Fara-7B outperforms UI-TARS-1.5-7B, often by large margins: Flights (37.9% vs. 10.5%), Hotels (53.8% vs. 35.3%), Activities (36.3% vs. 9.6%), Comparison Shopping (32.7% vs. 8.8%), Compositional (23.0% vs. 9.1%). The only near-tie is Shopping (52.4% vs. 41.1%) — still a meaningful gap. Since both models share the same base architecture, this strongly suggests that FaraGen's data quality (verified, multi-agent trajectories across diverse domains) is the differentiating factor, rather than model architecture or training methodology.

Grounding Results (Table 13)

Headline grounding numbers. Fara-7B improves over the base Qwen2.5-VL model on both grounding benchmarks: from 82.6% to 86.7% on ScreenSpot-V1, and from 86.6% to 89.3% on ScreenSpot-V2 (Table 13a). The per-domain breakdown for Fara-7B (Table 13b) shows strong text-element grounding (97.5% on mobile, 95.3% on desktop, 92.7% on web for ScreenSpot-v2) and relatively weaker but still solid icon/widget grounding (82.4% mobile, 78.5% desktop, 82.2% web).

What these numbers show. The grounding improvements confirm that Fara-7B's training (including the 562K dedicated grounding samples from Appendix B.1) does not degrade the base model's visual understanding — it enhances it. This is non-trivial because fine-tuning a vision-language model on a specialized task often causes catastrophic forgetting of general visual capabilities. The strong text-element grounding (92.7–97.5%) is particularly important because "a large portion of the interactive elements on the web are text-based, such as links and menus" (Section 5.2). The weaker icon/widget grounding (78.5–82.4%) represents a capability ceiling that could limit performance on highly visual, icon-driven websites, though the paper does not explore this connection.

Data and Inference Steps Scaling (Figure 8)

Data scaling. Figure 8 (left) shows WebVoyager success rate when training on progressively larger fractions of the data: 1%, 10%, and 100%. With 1% of data (~14.5K trajectories), Fara-7B achieves non-trivial accuracy; with 10% (~14.5K trajectories), performance improves substantially; with 100% (145K trajectories), it improves further. The paper notes: "Looking at the upward trend and significant jumps between scales, Fara-7B may benefit from further scaling up our data" (Section 5.3). This is a suggestive but not definitive finding — the curve has not clearly saturated, but only three data points (1%, 10%, 100%) are available, which is insufficient to fit a scaling law or confidently predict the benefit of 2× or 10× more data.

Inference steps scaling. Figure 8 (middle and right) compares Fara-7B and UI-TARS-1.5-7B as the maximum step budget is varied from 15 to 100 steps on WebVoyager and Online-Mind2Web. Both models "benefit similarly from the increased step budgets" (Section 5.3) — the curves are roughly parallel, with Fara-7B maintaining a consistent offset above UI-TARS-1.5-7B. This holds even on the harder Online-Mind2Web benchmark (Figure 8, right), where "neither model significantly outpaces the other as the steps are scaled up." The paper notes this is "somewhat surprising given that Fara-7B is only SFT'd, while UI-TARS-1.5-7B has undergone extensive RL training" (Section 5.3). The parallel scaling curves suggest that the two models have learned similar types of step-scaling behavior — both can make productive use of additional steps — but Fara-7B's higher intercept reflects better initial decision quality from its training data. RL appears to have improved UI-TARS-1.5-7B's ability to continue making progress with more steps (the slope is similar) but not its initial action quality (the intercept is lower).

Human Evaluation on WebVoyager (Section 5.1.2)

The Browserbase-conducted human evaluation established 62% accuracy for Fara-7B on the filtered WebVoyager tasks, with results that are "directly comparable" to other open-source models evaluated under identical conditions. The paper acknowledges that "human eval numbers for all models are less than the results obtained by GPT-4o LLM-as-a-judge from the official WebVoyager evaluation procedure reported in Table 9" and attributes this gap to "prompt- and model mis-alignment" in auto-evaluation frameworks (xue2025om2w). This is an important calibration: the 73.5% LLM-judged accuracy should be understood as an upper bound on true task success, with the 62% human-judged accuracy representing a more conservative estimate. The 11.5-percentage-point gap between auto-eval and human eval is consistent with the verifier alignment issues critiqued in Section 4.

---

Ablation Studies and Robustness Checks

Cumulative task solving improvements (Table 4): Each modification to the task solving pipeline incrementally improves WebVoyager success rate, from a baseline of 33% to a final 60% with GPT-5 as WebSurfer and all improvements enabled. The paper attributes "about half the gains" to stronger backbone models (o3, GPT-5) and "the remainder" to improved context construction and fault tolerance (Section 2.2.2). This quantifies the relative importance of model capability versus engineering robustness in synthetic data generation — both are essential, and they compound.

Browserbase impact on data yield (Table 2): Using Browserbase improves successful trajectory yield by more than 3× for shopping (9% → 35%) and nearly 4× for flights (3% → 11%). This is a critical infrastructure finding: data generation yield on live websites is dominated not by agent capability but by session reliability and anti-bot countermeasures. Without Browserbase, the pipeline would need to attempt roughly 3–4× more tasks to produce the same number of verified trajectories, tripling data generation costs.

Domain-specific instructions (Section 2.2.1): Adding domain-specific instructions "by itself boosted Hotel booking yields by 10% absolute." This represents a complementary lever to both model capability and infrastructure — task-specific prompting knowledge transfers from human expertise to the automated pipeline, improving yield on particularly challenging segments.

Verifier agreement with humans (Section 2.3): The three-verifier system achieves 83.3% agreement with human judgments, with a false positive rate of 16.7% and false negative rate of 18.4%. This means approximately 1 in 6 "successful" trajectories in the training data may actually be incorrect, and approximately 1 in 5 "failed" trajectories were actually correct and wasted. The paper does not analyze how these error rates vary by task segment or difficulty, which would be informative for understanding whether certain task types have systematically higher contamination.

Critical point stopping (Section 5.4, Appendix D.2): Fara-7B stopped before the critical point in 19 of 23 synthetic tasks (82.6% success rate on this safety behavior). The four failures involved low-impact, reversible actions (marking email as read, liking a post) or a single case of publishing without confirmation. The paper acknowledges that "the size of this dataset is relatively small and more work is needed to comprehensively test this behavior."

Adversarial testing on Magentic-UI tasks (Section 5.4): Fara-7B avoided harmful behavior in 9 of 13 adversarial tasks (69.2%), failing only in cases involving "navigating to links which point to local or cached files which were then stopped by browser sandboxing" — failures that were contained by the execution environment rather than causing actual harm.

Training data composition (Table 16): The full data mixture of 1.8M samples is dominated by trajectory data (68.5%) and grounding data (31.2%), with refusal data (0.17%) and UI VQA/captioning (<0.1%) being negligible fractions. The paper notes that "very little [refusal] data seems to be required and with more data the model begins to refuse even benign tasks" (Appendix C) — an important sensitivity finding that guides practitioners on the narrow window for safety training before over-refusal degrades utility.

Observation truncation ($N=3$): The paper states that $N=3$ (keeping only the three most recent screenshots in context) was chosen "based on early experiments" and "offers a reasonable trade-off between accuracy and speed and memory performance" (Section 3.1). No ablation comparing $N=1$, $N=3$, $N=5$, etc. is presented, which is a notable gap — the choice of observation history length likely has significant effects on long-horizon task performance and could interact with trajectory complexity.

SFT-only vs. RL (implicit comparison with UI-TARS-1.5-7B): The comparison with UI-TARS-1.5-7B (which underwent "extensive RL training") serves as a de facto ablation of training methodology. Fara-7B's consistent superiority across benchmarks (Tables 9, 11) and the parallel step-scaling curves (Figure 8) suggest that high-quality SFT data can substitute for RL in this domain, at least at the 7B scale and for the evaluated benchmarks. However, this is not a controlled ablation — the two models differ in both training data and training methodology, so the SFT-vs-RL comparison is confounded.

---

Critical Assessment

The experimental evaluation is unusually thorough for a systems paper: four live-web benchmarks, multi-run evaluation with standard deviations, human-evaluator calibration, per-category breakdowns, cost and token efficiency analysis, grounding evaluation, safety evaluation, data scaling curves, and step-budget scaling curves. The breadth is commendable. The depth varies by claim, and several of the paper's strongest assertions deserve scrutiny.

Claim: "Fara-7B achieves state of the art results for a model of its size." This claim is strongly supported by Tables 9, 11, and 19. Across four benchmarks and 11 subcategories (WebTailBench), Fara-7B outperforms UI-TARS-1.5-7B — the only other 7B native CUA model evaluated — on every benchmark and 10 of 11 subcategories, often by substantial margins (e.g., 38.4% vs. 19.5% on WebTailBench macro average). The standard deviations in Table 19 confirm this is not a run-variance artifact. The comparison with GLM-4.1V-9B-Thinking (a 9B SoM agent) further supports the claim: Fara-7B outperforms it on WebVoyager (73.5 vs. 66.8), ties on Online-Mind2Web (34.1 vs. 33.9), and substantially exceeds it on WebTailBench (38.4 vs. 22.4). Caveat: The "state of the art" claim is limited to the specific set of models evaluated — there may be other 7B-scale CUA models not included in the comparison (the paper does not claim exhaustiveness). The claim also applies to the specific benchmarks used; performance on other CUA benchmarks not tested remains unknown.

Claim: "Fara-7B is competitive with much larger frontier models." This claim is supported with important qualifications that the paper largely acknowledges. On WebVoyager, Fara-7B (73.5%) is within 17 percentage points of GPT-5 SoM (90.6%) and within 15 points of o3 SoM (88.7%) — "competitive" in the sense that the gap is much smaller than the 10–100× parameter count difference might suggest, but still large in absolute terms. On WebTailBench, the gap is wider: Fara-7B's 38.4% vs. GPT-5 SoM's 60.4%. The pass@k analysis (Figure 1) qualifies this further: pass@3 for Fara-7B approaches higher performance, but pass@k is "not representative of a real system" (Section 5.1.2). What the experiments actually demonstrate is that Fara-7B is competitive with GPT-4o SoM specifically (65.1% WebVoyager, 34.6% Online-Mind2Web, 30.8% WebTailBench) — a more moderate claim that is well-supported — and that it substantially exceeds OpenAI computer-use-preview (70.9%, 42.9%, 25.7% respectively), which is a much larger proprietary CUA model. The gap to the o3 and GPT-5 SoM agents is real and suggests a capability ceiling that Fara-7B's current training approach has not overcome, particularly on multi-step and compositional tasks.

**Claim: "FaraGen can generate verified successful trajectories at roughly $1 per task."** This claim is **approximately supported but the accounting is incomplete.** Table 6 shows costs of$0.59 (o4-mini), $1.00 (GPT-5), and$1.08 (o3) per trajectory — averaging to roughly $1 for the strongest configurations. However, these costs are based on "600 trajectories [with] an average of approximately 19 steps" (Section 2.4), which is substantially longer than the overall dataset average of 6.9 steps (Table 5). The 19-step average likely comes from the more complex targeted URL segments, which are more expensive but also more valuable. The cost per trajectory for the 80.8K agent-exploration trajectories (average 5.1 steps, Table 2) would be substantially lower. The paper also does not amortize the cost of failed trajectories — Table 2 shows that most task-solving attempts fail, meaning the effective cost per *successful verified* trajectory is higher than the per-trajectory cost reported in Table 6, which only counts completed trajectories. A more accurate cost would be: `$\text{cost_per_success} = \text{cost_per_attempt} \times (\text{attempts needed per success})$`, which for flights without Browserbase would be `$\text{cost_per_attempt} \times (1/0.03) \approx 33\times$` higher. Browserbase substantially mitigates this by improving yield, but the true cost per successful trajectory is still understated.

Missing experiments that would strengthen the paper:

• Ablation of the three verifiers: The paper never reports data quality or model performance when using only one or two of the three verifiers versus all three. This would quantify the marginal value of each verifier and justify the complexity of the three-verifier design.
• Observation history length ablation ($N=3$ vs. alternatives): The choice of how many past screenshots to retain is critical for long-horizon tasks, yet no supporting experiments are provided.
• Comparison with a Fara-7B trained only on agent-exploration trajectories (no targeted URLs): This would isolate the value of the targeted URL task proposal strategy versus the simpler agent-exploration strategy, quantifying whether the additional complexity of targeted proposal is justified.
• Out-of-distribution generalization evaluation: All four benchmarks share significant domain overlap with FaraGen's training distribution (the targeted URL segments in Table 2 map directly onto WebTailBench categories). An evaluation on websites or task types explicitly excluded from training would test whether Fara-7B generalizes beyond its training distribution or is primarily interpolating within it.
• Scaling to larger model sizes: The paper demonstrates that a 7B model benefits from FaraGen data, but does not test whether an identically trained 13B or 70B model would see proportional or diminishing returns. This limits the generalizability of the "small models can approach frontier performance" narrative.

The human evaluation calibration deserves emphasis. The 62% human-judged accuracy on WebVoyager versus 73.5% LLM-judged accuracy is an 11.5-point gap that should make readers cautious about taking the LLM-judged numbers at face value. This gap likely exists for all models in Table 9, not just Fara-7B, but the paper only provides human evaluation for Fara-7B (and "other open source models" at an unspecified accuracy level). If all models' true (human-judged) performance is systematically lower than LLM-judged performance by a similar margin, the relative rankings may still be valid. But the paper does not establish this, and the possibility remains that Fara-7B's trajectories are judged more generously by LLM verifiers than other models' trajectories — a subtle confound that the human evaluation alone cannot resolve without per-model human judgments.

The safety evaluation is preliminary but promising. The 19/23 critical-point stopping rate is high but measured on only 23 synthetic tasks using the REAL benchmark environment (garg2025real), which provides "high-fidelity replicas" of websites rather than live production sites. The paper acknowledges the small dataset size (Section 5.4). The refusal rates (94.2% on AgentHarm-Chat, 81.9% on WebTailBench-Refusals) are strong compared to other CUA models but were achieved with training data that "might give it an advantage over the baselines on WebTailBench-Refusals but not on AgentHarm-Chat" (Section 5.4) — a transparency about data contamination that is commendable but that partially undermines the WebTailBench-Refusals comparison. The adversarial testing (9/13 tasks avoided harmful behavior) is promising but the sample size (13 tasks) is too small for statistical confidence and the paper does not report adversarial testing results for baseline models, making comparisons impossible.

The grounding evaluation (Table 13) is a positive signal but disconnected from agentic performance. The paper shows that grounding accuracy improves over the base model, but does not ablate the contribution of the 562K grounding samples to agentic task success. It is possible that the grounding data is essential, or that the trajectory data alone would have achieved similar grounding improvements as a byproduct. Without this ablation, the value of the grounding data component remains speculative.

Overall assessment: The experimental evaluation convincingly establishes Fara-7B as the best-performing 7B-scale CUA model on the evaluated benchmarks, with strong cost-efficiency advantages over SoM agents and competitive performance against GPT-4o-based systems. The evidence for competitiveness with frontier reasoning models (o3, GPT-5) is weaker and requires the cost-efficiency framing to be persuasive. The verifier alignment gap (11.5 points between LLM and human judgment) is the most concerning methodological finding and should temper confidence in the absolute accuracy numbers, though relative rankings between models are likely preserved. The safety evaluation, while more thorough than typical for a CUA paper, relies on small sample sizes and acknowledges data contamination concerns that limit the strength of its conclusions.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Requires 2048 Samples Per Question — A Cost the Paper Excludes

The assumption or constraint. The compute-optimal framework allocates test-time compute based on estimated question difficulty, but the difficulty estimation procedure itself consumes massive computation. For each of the 500 test questions, the paper generates 2048 complete solutions from the base model and scores them with the PRM to estimate the pass@1 rate (Section 3.2). The paper explicitly acknowledges:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The headline $4\times$ efficiency gain — compute-optimal scaling matching best-of-N performance with $4\times$ fewer generations — is computed after difficulty is already known, without amortizing the 2048 samples required to learn it. In a deployment scenario, the total cost would be difficulty estimation (2048 × PRM forward passes per question) plus strategy execution. For a single question, difficulty estimation alone consumes more compute than the largest test-time budgets studied (256–512 generations). This makes the claimed gains an upper bound that cannot be achieved in practice without a cheaper difficulty estimation method that the paper does not provide.

What evidence exists in the paper. The difficulty estimation procedure is described in Section 3.2, including the acknowledgment quoted above that the cost is not accounted for. Figures 4 and 8 show that compute-optimal scaling curves for oracle difficulty bins and predicted (PRM-based) bins "largely overlap," confirming that the PRM can substitute for ground-truth labels — but both methods require generating 2048 samples. The paper does not report the computational cost of difficulty estimation in FLOPS or compare it to the test-time compute budgets studied.

Mitigation status. The paper acknowledges this gap explicitly and flags it as "a key avenue for future work" (Section 3.2), suggesting that "pretraining or finetuning models to directly predict difficulty of a question" could resolve it. No such model is developed or evaluated. The exploitation-exploration framing — "compute spent assessing difficulty versus compute spent solving the problem" — is noted but not quantified. The limitation is unaddressed in the current work, and the $4\times$ figure should be interpreted as a proof-of-concept rather than a deployment-ready efficiency claim.

---

6.2 Hard Problems Are Unaffected by Test-Time Compute — The Base Model Sets a Hard Ceiling

The constraint. Test-time compute can only find or refine solutions that the base model already has some non-trivial probability of generating. If the base model's pass@1 is near zero on a problem class, no amount of search, beam search, lookahead search, or revision can help — there are no correct solutions in the proposal distribution to discover or improve. The paper is explicit about this boundary:

"test-time compute amplifies existing capability but does not create it from nothing" (Section 7 discussion)

The consequence. On the hardest questions (difficulty bin 5), no method makes meaningful progress regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all budgets (4 to 256 generations). In Figure 7 (right), bin 5 shows roughly 2–3% accuracy regardless of the sequential-to-parallel revision ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% for both revisions and PRM search. The paper's approach provides zero benefit for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, scaling pretraining remains the only viable path — the paper's FLOPs-matched analysis confirms that for hard problems, the $14\times$ larger model consistently outperforms test-time compute with the smaller model (Section 7, Figure 9).

What evidence exists in the paper. Section 5.3 (Figure 3, right), Section 6 (Figures 6, 7, right), and Section 7 (Figure 9) all show the same pattern: bin 5 accuracy is near-zero and flat across all budgets, methods, and allocation strategies. The FLOPs-matched comparison quantifies the disadvantage: on hard questions at $R \gg 1$ with PRM search, test-time compute shows a −52.9% relative disadvantage compared to the larger model (Figure 1, bottom-right bar chart). The paper states this explicitly in the Section 7 takeaway: "test-time and pretraining compute are not 1-to-1 exchangeable."

Mitigation status. This is a fundamental limitation of the test-time compute paradigm, not a fixable engineering issue. The paper acknowledges it transparently in the Section 7 discussion: "test-time compute is powerful when problems are within the base model's reach (it already produces correct solutions at some non-trivial rate), but it cannot compensate for fundamental capability gaps." No mitigation is proposed, because none exists within the framework — by definition, you cannot search for solutions that the proposal distribution never generates.

---

6.3 The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate

The constraint. The revision model is trained solely on trajectories where all in-context answers are incorrect, followed by a correct target (Section 6.1). This training data construction means the model has no signal for what to do when the current answer is already correct. At test time, when the revision chain contains a correct answer, the model may incorrectly "revise" it into a wrong answer. The paper reports:

"approximately 38% of correct answers get converted back to incorrect ones" using a naïve approach (Section 6.1)

The consequence. A revision chain is not monotonically improving — it oscillates between correct and incorrect answers. This means the system cannot simply take the final output of a revision chain as the answer; it must use a selection mechanism (majority voting or verifier-based selection) across the entire chain to identify the best answer. This introduces a dependence on the verifier that the revision model was supposed to reduce. More critically, the 38% reversion rate means that longer revision chains do not necessarily yield better answers — the chain may pass through a correct answer early, then revise away from it, and the selection mechanism must correctly identify that early correct answer despite it being "buried" under subsequent revisions. The paper's approach of within-chain selection (Appendix I) partially mitigates this, but the fundamental problem that the model cannot recognize correctness remains.

What evidence exists in the paper. The 38% figure is reported in Section 6.1. The mitigation — using majority voting or verifier-based selection across the revision chain — is described in the same section and validated in Figures 6–8. However, the paper does not report how often the correct answer appears in the chain but is not selected by the verifier (false negatives in within-chain selection), which would quantify the residual cost of the reversion problem.

Mitigation status. The paper mitigates the symptom (through chain-level selection) but does not address the cause (training data construction that never shows correct-to-correct or correct-to-incorrect transitions). A more principled solution — such as training the model with correct answers in context and teaching it to recognize that no revision is needed — is not explored. The paper does not discuss this as a limitation, but the ReST$^{EM}$ experiment (Appendix K, Figure 16), where "additional sequential revisions substantially hurt performance" with an RL-optimized revision model, suggests that the reversion problem may be exacerbated by certain training methodologies, indicating sensitivity that is not fully understood.

---

6.4 All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

The constraint. Every experiment in the paper — the PRM behavior, the search algorithm comparisons, the revision model analysis, the compute-optimal scaling curves, and the FLOPs-matched comparison — uses the MATH benchmark (500 test questions) and PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an unverified assumption.

The consequence. Several findings could be model-specific or benchmark-specific:

• PRM over-optimization patterns depend on the base model's output distribution. A model with different calibration properties or error patterns might produce different difficulty-dependent scaling curves — potentially with less over-optimization on easy problems (if the PRM is better calibrated) or more (if the PRM is noisier).
• Revision model training (edit-distance-based incorrect-correct pairing, Section 6.1) depends on the base model's in-context learning capabilities, which vary substantially across model families. The finding that revisions are most effective on easy problems might not hold for models with different error distributions.
• The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning. The difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems) may not generalize to other reasoning domains (code generation, logical deduction, scientific QA) or to tasks requiring factual knowledge rather than inference, where the nature of errors and the role of verifiers could be qualitatively different.

The 500-question test set, split into five difficulty quintiles of ~100 questions each, then split again by two-fold cross-validation, means the compute-optimal strategy is selected based on approximately 50 questions per fold per bin. These are small samples for strategy selection, and the paper does not report confidence intervals on the compute-optimal scaling curves, making it difficult to assess whether the observed strategy differences across difficulty bins are statistically robust.

What evidence exists in the paper. The paper acknowledges the single-benchmark, single-model-family limitation implicitly by stating the representativeness belief (Section 4), but does not treat it as a formal limitation or discuss how findings might vary across models or benchmarks. No sensitivity analysis across model sizes (PaLM 2-S, PaLM 2-M, PaLM 2-L) or across different benchmarks is provided. The test set size of 500 questions and cross-validation splitting are described in Section 3.2; the ~50-question-per-fold sample size is a direct arithmetic consequence of these specifications.

Mitigation status. The paper does not address this limitation. Section 8 (future work) does not mention extending the analysis to other models or benchmarks. The authors' claim of representativeness is presented as a belief, not an empirically supported claim.

---

6.5 No Combination of PRM Search with Revisions — The Two Axes Are Studied Independently

The constraint. The paper's framework decomposes test-time compute into modifications to the proposal distribution (revisions) and modifications to the verifier (PRM search), which the authors position as complementary axes (Section 2). However, all experiments study these axes independently — there is no experiment that applies PRM-guided beam search to revision model outputs, or that uses the PRM to guide which revisions to pursue. The authors explicitly acknowledge this gap:

"we did not experiment with PRM tree-search techniques in combination with revisions" (Section 8)

The consequence. The paper's central claim — that the two axes have "complementary, difficulty-dependent strengths" (Section 4) — is supported only by comparing them in isolation, not by demonstrating that combining them yields gains beyond either alone. The revision model improves the quality of generated candidates; PRM search improves the selection among generated candidates. Applying beam search to revision model outputs could, in principle, compound these benefits — the search would operate over a better proposal distribution — but the paper provides no evidence for or against this hypothesis. The current results therefore represent a lower bound on what a fully integrated system could achieve, but also leave open the possibility that the two mechanisms interfere (e.g., revision outputs might be less amenable to PRM scoring, or the PRM might over-optimize differently on revision-generated solutions). Without combination experiments, the claim of complementarity remains a conceptual argument rather than an empirical finding.

What evidence exists in the paper. The decomposition into proposal vs. verifier axes is laid out in Section 2. Search experiments (Section 5) use the base model, not the revision model, as the proposal distribution. Revision experiments (Section 6) use majority voting or a separate revision-specific ORM for selection, not the PRM. The authors' acknowledgment of this gap in Section 8 is the paper's only engagement with the combination question.

Mitigation status. The paper explicitly calls this out as future work (Section 8), but provides no preliminary results or analysis of why the combination was not attempted (e.g., computational cost, negative preliminary findings, or implementation complexity). This is a significant omission because the combination of search and revisions is the natural endpoint of the framework the paper introduces, and its absence means the paper's strongest claims about the complementary nature of the two axes have not been directly tested.

---

6.6 The $14\times$ Larger Model Baseline Is Not Compute-Optimally Trained and Uses Only Greedy Decoding

The constraint. The FLOPs-matched comparison (Section 7) compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately $14\times$ more parameters. However, the larger model is trained by scaling parameters only, holding training data fixed. The authors acknowledge this deviates from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work" (Section 7)

Furthermore, the $14\times$ larger model is evaluated using greedy decoding only — no best-of-N, no search, no revision, no test-time compute augmentation of any kind.

The consequence. A Chinchilla-optimal model (scaling both data and parameters equally, as in Hoffmann et al., 2022) trained with the same total FLOPs would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than a properly optimized alternative would be. Additionally, giving the larger model even a modest test-time compute budget (e.g., best-of-8 with majority voting) would create a substantially stronger baseline. The paper's headline comparisons — e.g., test-time compute with the smaller model outperforming the $14\times$ larger model by +27.8% relative on easy questions at $R \ll 1$ (Figure 9, left) — may therefore overstate the advantage of test-time compute over pretraining. The fair comparison would be: compute-optimal small model + compute-optimal inference strategy vs. compute-optimal large model + some reasonable inference strategy. The paper only evaluates the full treatment on one side of the comparison.

What evidence exists in the paper. The training setup for the larger model is described in Section 7, with the quoted caveat about compute-optimal pretraining. The greedy decoding assumption is not explicitly stated in Section 7, but the baseline is described as the $14\times$ larger model's "greedy performance" in the Figure 9 caption ("The $14\times$ larger model's greedy performance (stars) is placed at three x-axis positions"). No experiments give the larger model any test-time compute budget. The paper does not provide an ablation comparing against a Chinchilla-optimal larger model or a larger model with best-of-N.

Mitigation status. The paper acknowledges the compute-optimal pretraining caveat explicitly and frames it as future work. The greedy-decoding-only baseline is not acknowledged as a limitation. Both choices systematically favor the test-time compute approach in the FLOPs-matched comparison, and the magnitude of this bias is unknown. Practitioners should treat the FLOPs-matched results as establishing feasibility (test-time compute with a small model can beat a naïve larger model) rather than optimality (test-time compute should be preferred over pretraining at a given FLOPs budget).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the CUA field from a model-centric to a data-centric paradigm — or more precisely, from a scaffolding-centric to a synthesis-centric paradigm. Before this work, the dominant approach to building capable web agents was to wrap the largest available frontier model (GPT-4o, o3, GPT-5) in accessibility-tree scaffolding (Set-of-Marks), add an Orchestrator-like planning agent, and accept the resulting inference cost as the price of capability. The alternative — training a small, native CUA model — was gated by the lack of training data, which was understood primarily as a collection cost problem (human annotation is expensive and slow).

FaraGen reframes the bottleneck. The paper's funnel statistics in Table 2 reveal that even frontier models, operating within a carefully designed multi-agent system, fail to produce valid trajectories most of the time — 54–84% of attempts fail mid-execution, and of those that complete, only a fraction pass verification (3–35% depending on task segment). This means the binding constraint is not data generation volume but automated quality assurance at scale: the ability to detect loops, hallucinations, premature terminations, and misaligned actions during or immediately after trajectory generation, and to filter them out before they contaminate the training set. The Orchestrator's ledger (five diagnostic fields updated at every step) and the three-verifier pipeline (Alignment, Rubric, Multimodal) are the paper's answer to this reframed problem, and together they enable production of 145K verified trajectories at roughly $1 each. This is the conceptual shift: CUA data scarcity is primarily a verification problem, not a generation problem, and solving verification at scale enables a qualitatively different approach to model development.

The magnitude of this shift is substantial but bounded. It is not a paradigm shift on the order of the transformer architecture or pretraining-followed-by-fine-tuning — the underlying model (Qwen2.5-VL-7B) and training methodology (supervised fine-tuning) are standard. But it is a reframing of the resource allocation problem for CUA development: rather than investing in ever-larger or more heavily scaffolded models, invest in better synthetic data pipelines with closed-loop verification. The paper's strongest evidence for this reframing is the comparison with UI-TARS-1.5-7B: same base architecture, but Fara-7B doubles its accuracy on WebTailBench (38.4% vs. 19.5%) and substantially exceeds it on all benchmarks, despite UI-TARS undergoing "extensive RL training" while Fara-7B uses only SFT (Section 5.3). The differentiator is the data, not the model or the training algorithm.

The paper also reconciles a tension that has been implicit in the CUA literature but rarely articulated: the tradeoff between SoM agents (high accuracy, high cost, brittle to accessibility tree quality) and native CUA models (lower accuracy historically, but lower cost and more robust to real-world website variation). The cost-efficiency analysis in Tables 10 and 12 provides the first quantitative evidence that native CUA models can close the accuracy gap enough to make the cost argument decisive: Fara-7B achieves 73.5% WebVoyager accuracy at $0.025/task versus GPT-4o SoM's 65.1$0.302/task (12× more expensive for lower accuracy), and versus o3 SoM's 88.7% at $0.514/task (20× more expensive for 15 points of accuracy). This reframes the CUA deployment decision from "use the most capable model you can afford" to "train the best small model your data pipeline can support." The SoM approach now looks like a data generation tool (where cost-per-trajectory doesn't matter and accuracy is paramount) rather than a deployment architecture, which is exactly how FaraGen uses it.

The research directions this opens and closes:

• Opened: Synthetic data generation for agentic tasks, multi-agent distillation into small models, automated trajectory verification, domain-specific training data curation at scale, safety-through-data-construction (critical points as a training signal rather than a runtime check).
• Moderately diminished: Wrapping frontier models in scaffolding for production CUA deployment — the cost analysis makes this look increasingly unsustainable compared to fine-tuned small models for all but the most complex, high-stakes tasks. The value of frontier models in this ecosystem is now primarily as data generators and verifiers (teachers), not as runtime agents.
• Unchanged or reinforced: The importance of robust evaluation — the paper's 11.5-point gap between LLM-judged and human-judged WebVoyager accuracy (73.5% vs. 62%) and its critique of existing benchmarks (vague tasks, brittle verifiers, penalization for environmental non-stationarity) reinforce that CUA evaluation is far from solved and that benchmarks like WebTailBench with released verifiers and known human-alignment statistics are the right direction.

---

Follow-Up Research This Work Enables

1. Ablation of the three-verifier pipeline to quantify the marginal value of each verifier and determine whether all three are necessary. The paper uses three verifiers — Alignment (text-only, intent satisfaction), Rubric (partial-credit scoring), and Multimodal (screenshot-grounded hallucination detection) — but never reports model performance or data quality when using subsets of them. A controlled experiment would train three Fara-7B variants on trajectories filtered by (a) Alignment only, (b) Alignment + Rubric, (c) all three, and evaluate on WebTailBench and WebVoyager. The key question is whether the Multimodal Verifier, which is the most architecturally complex and expensive (it processes screenshots), provides marginal value beyond what the text-only verifiers catch. The paper reports that the Multimodal Verifier specifically targets hallucinations — "when asked what ingredients and how many calories are in a particular smoothie recipe, this verifier will catch hallucinations of caloric content or ingredient quantities that aren't supported by the underlying screenshot" (Section 2.3) — suggesting that an ablation might reveal higher hallucination rates in models trained without it, measurable through targeted probing on information-extraction tasks. This experiment would also help explain the 16.7% false positive rate in the current verification system: which verifier is responsible for most false positives, and can its threshold be tuned to improve precision without excessive recall loss?

2. Probing whether Fara-7B's reasoning text actually encodes diagnostic recovery strategies or is merely correlational. The paper's most interesting implicit claim is that training on the WebSurfer's reasoning text — which encodes the Orchestrator's diagnostic signals (loop detection, critical point awareness, task satisfaction) — teaches Fara-7B to generate recovery strategies without explicit diagnostic modules. This claim can be tested directly: create a held-out set of trajectories where the Orchestrator intervened (re-planned after a loop, forced termination at a critical point, overrode a premature stop), and compare Fara-7B's reasoning at analogous states against a baseline model trained only on (observation, action) pairs without the reasoning text. If Fara-7B's reasoning at loop points contains diagnostic language ("I've attempted this action multiple times without success, let me try an alternative approach") and its subsequent actions actually differ from the looping pattern, the distillation claim is supported. If the reasoning is merely descriptive ("I see a search bar") without influencing action selection, the reasoning text is dead weight and could be removed to save output tokens. This experiment requires instrumenting the inference loop to detect when the model's actions are repetitive (consecutive clicks within a small radius, repeated scroll attempts in the same direction) and comparing behavioral recovery rates between the full model and the reasoning-ablated variant.

3. Training a Fara-7B variant with RL on top of the SFT checkpoint and measuring whether RL closes the gap to GPT-5 SoM on multi-step compositional tasks. The paper shows that Fara-7B's largest accuracy gap relative to GPT-5 SoM is on complex WebTailBench categories: Shopping Lists (49.0% vs. 66.0%), Comparison Shopping (32.7% vs. 67.3%), and Compositional Tasks (23.0% vs. 51.5%) — all tasks requiring 30–50+ step trajectories with cross-page information tracking. The paper's step-budget scaling curves (Figure 8, middle/right) show that Fara-7B and UI-TARS-1.5-7B scale similarly with additional steps, but the intercept gap remains — Fara-7B makes better initial decisions but doesn't improve faster with more steps. This suggests that SFT alone may saturate on long-horizon reasoning and that RL — which can optimize for task completion directly rather than step-by-step imitation — could provide complementary gains. A concrete experiment: fine-tune the Fara-7B SFT checkpoint with RL using a reward of 1 for task completion (as judged by the three-verifier system) and 0 otherwise, on a held-out set of compositional training tasks, then evaluate on WebTailBench's multi-step categories. The null result (RL doesn't help beyond SFT) would be equally informative, suggesting that the bottleneck is the 7B model's fundamental reasoning capacity rather than the training objective.

4. Measuring out-of-distribution generalization by evaluating Fara-7B on websites and task types explicitly excluded from FaraGen training. The paper's training data covers 70,117 unique domains (Table 5), and the targeted URL segments (shopping, flights, hotels, restaurants, activities, ticketing, real estate, jobs) map directly onto WebTailBench categories. It is therefore unclear whether Fara-7B is generalizing to new websites or interpolating within its massive training distribution. A clean generalization test would: (a) identify website domains that appear in WebVoyager or WebTailBench but not in the FaraGen training corpus (using the unique domain list from Table 5), (b) identify task types that are structurally different from any training segment — for instance, interacting with government forms (tax filing, license renewal), educational platforms (enrolling in online courses, submitting assignments), or healthcare portals (finding a doctor, checking insurance coverage) — and (c) evaluate Fara-7B against GPT-4o SoM on these held-out domains and task types. The prediction from the paper's framework is that Fara-7B should generalize better than SoM agents on new domains (because pixel-based coordinate prediction is website-agnostic while accessibility trees vary by site) but worse on new task types (because the reasoning patterns for novel task structures weren't in the training data). Confirming or disconfirming this prediction would clarify what the model has actually learned.

5. Scaling the approach to 13B and 70B models to determine whether FaraGen data yields proportional, diminishing, or super-linear returns. The paper's data scaling curve (Figure 8, left) shows that WebVoyager accuracy continues to improve from 1% to 10% to 100% of the data, with the upward trend suggesting further gains from more data. But this is at a fixed 7B parameter scale. The field assumption — drawn from LLM scaling laws — is that larger models benefit more from larger datasets. Testing this with FaraGen data would require: train Qwen2.5-VL-13B and Qwen2.5-VL-72B variants using the identical data mixture and training recipe (identical SFT hyperparameters, identical data proportions), then evaluate on the same four benchmarks. If the 13B model achieves, say, 85% WebVoyager (narrowing the gap to GPT-5 SoM's 90.6% substantially) while the 72B model approaches or exceeds it, the paper's narrative — that data quality can substitute for model size — would need to be refined to: data quality can substitute for model size up to a point, beyond which scaling both jointly is optimal. If the 13B model shows minimal improvement over 7B (suggesting the data diversity saturates at 7B), the practical implication is that FaraGen data is most valuable for enabling small on-device models, not for pushing frontier performance.

6. Developing a continuous difficulty estimator that enables dynamic, online strategy allocation — combining the paper's compute-optimal insight with a lightweight difficulty predictor. The paper currently estimates difficulty via 2048 samples + PRM scoring, which is too expensive for deployment. A research direction the paper explicitly flags is "pretraining or finetuning models to directly predict difficulty of a question." But a more ambitious design would integrate difficulty estimation into the problem-solving process itself: begin with 4–8 parallel samples, use the variance and mean of the PRM scores on those samples as a quick difficulty signal, and then dynamically allocate the remaining budget (switch to beam search if scores are moderate-variance, continue parallel if low-variance, or escalate to a larger model if scores are near-zero). This connects to the multi-armed bandit and Bayesian optimization literatures that the paper gestures at in Section 3.2. A concrete experiment: implement this adaptive scheme using the existing PRM, compare its performance against the static (pre-computed) difficulty bin approach on WebVoyager or WebTailBench, and measure whether the online estimator recovers similar gains without the 2048-sample upfront cost. The key metric is effective compute multiplier: total compute used (difficulty estimation + strategy execution) relative to the performance achieved, compared against the static approach and against best-of-N.

---

Practical Applications and Downstream Use Cases

1. On-device web agents for privacy-sensitive consumer applications. Fara-7B's 7B parameter size and screenshot-only inference (no accessibility tree, no cloud API calls) make it feasible to run directly on consumer laptops and high-end mobile devices. The cost analysis (Tables 10, 12) shows per-task costs of $0.025–0.069 — negligible compared to cloud-based alternatives — and the 16.5 average actions per task on WebVoyager (Table 10) suggests sub-minute completion times for typical web tasks. The privacy benefit is concrete: screenshots never leave the device, meaning banking portals, medical records, private emails, and other sensitive content are processed locally. This is categorically impossible with SoM agents that must transmit screenshots and accessibility trees to cloud-hosted models. The critical point mechanism (stopping before entering payment information, login credentials, or personal data) provides an additional safety layer: the model hands control to the user before any irreversible action, so even if the model makes an error in navigation or product selection, the user has a final check before commitment. The main deployment gap is the 62% human-judged WebVoyager accuracy (versus 73.5% LLM-judged): for applications where 62% reliability is acceptable (with human fallback for the remaining 38%), this is deployable today; for applications requiring >95% reliability, further improvement is needed.

2. Cost-efficient batch data extraction and web automation for enterprises. Organizations that routinely perform large-scale web data extraction — price monitoring across competitors, regulatory compliance checking, lead generation from public databases, automated form filling for government or business processes — currently use brittle programmatic scrapers or pay for cloud-based LLM agents. Fara-7B offers a middle path: high-throughput batch processing on commodity GPU hardware (the paper used 64 H100 GPUs for training; inference on a single H100 or consumer GPU with quantization is feasible for deployment) at a per-task cost that undercuts cloud APIs by 10–100×. The 70,117 unique domains in the training data (Table 5) suggest broad website coverage, and the domain diversity (average 0.5 unique domains per trajectory) suggests the model has seen enough variety to handle unusual or long-tail websites that programmatic scrapers fail on. The key practical limitation is the 100-step cap (Section 5.1.1): for very complex multi-page workflows, the model may exhaust its step budget without completing the task, and the absence of training data beyond critical points means the model cannot complete transactions that require login or payment — it can navigate to the point of purchase and present the summary to the user, but cannot finalize the purchase autonomously.

3. Synthetic data generation for fine-tuning domain-specific web agents. FaraGen is described as a data generation engine for CUA, but its architecture is not specific to web tasks — it is a general-purpose pipeline for task proposal, multi-agent solving, and multi-verifier quality control that could be adapted to other agentic domains. An enterprise with a specific web automation need (e.g., processing insurance claims across multiple carrier portals, auditing product listings on e-commerce platforms, filing permit applications with municipal websites) could use FaraGen with domain-specific URL seeds, task templates, and verification rubrics to generate a targeted training dataset, then fine-tune Fara-7B (or another small VLM) on that domain-specific data. The paper shows that "modest sums of high quality data can take a small model a long way at mastering a particular skill" (Section 5.1.3) — Fara-7B achieves 37.9% on flights and 53.8% on hotels with fewer than 4K training trajectories each (Table 2). For a focused enterprise use case with, say, 5–10 specific websites and 10–20 task types, a few thousand domain-specific trajectories (costing a few thousand dollars to generate via FaraGen) could produce a specialized agent with much higher reliability than the general-purpose Fara-7B, while retaining the on-device deployment advantages.

4. Human-agent collaborative workflows where the agent handles navigation and the human handles decisions. Fara-7B's critical point mechanism — stopping before any irreversible action and handing control to the user — naturally supports a division of labor: the agent does the tedious, multi-step navigation (searching for products, comparing options, filling forms up to the final submission point), and the human reviews the agent's work and makes the final decision (which product to buy, whether to confirm the booking, what message to send). This model is well-suited for customer service workflows (agent drafts responses and navigates to relevant policy pages; human reviews and sends), travel planning (agent finds and compares options; human chooses and pays), and administrative tasks (agent navigates to the correct form and pre-fills known fields; human reviews and submits). The paper's evaluation doesn't directly test this collaborative mode, but the architecture supports it: the Terminate action at critical points is designed to hand context back to the user, and Equation (3) shows that Fara-7B can resume from a user follow-up query while maintaining full history context. The practical integration would require a UI layer that presents the agent's current state and reasoning to the user at termination points, which is an engineering challenge but not a fundamental capability limitation.

---

When to Prefer This Method

The paper does not present Fara-7B as one option among named, comparable alternatives in a well-defined tradeoff space — it presents it as a new capability (a 7B native CUA model trained on synthetic data) that outperforms all same-size competitors and some larger systems, without articulating conditions under which a different specific approach would be clearly preferable. The comparison with SoM agents (GPT-4o, o3, GPT-5) and with UI-TARS-1.5-7B shows Fara-7B winning on cost-efficiency and accuracy-at-size, but the paper does not claim or demonstrate that Fara-7B should replace GPT-5 SoM for tasks requiring maximum accuracy regardless of cost, nor that there are identifiable problem types where UI-TARS-1.5-7B would be the better choice. The tradeoffs the paper implicitly surfaces — native CUA vs. SoM, SFT vs. RL, small model + good data vs. large model — are not presented as decision rules with conditions. Forcing a "prefer X when Y" structure would substitute my judgment for the paper's claims. The limitations section (Section 7 of the paper) does enumerate specific constraints: no drag-and-drop, no media consumption, no high-frequency gaming interactions, no autonomous operation past critical points, and a recommendation against use in highly regulated or high-stakes domains. These are usage constraints, not comparative preference criteria between named methods.