MolmoWeb: Open Visual Web Agent and Open Data for the Open Web

ArXiv: 2604.08516

🎯 Pitch

A fully open 8B vision-language model trained only on screenshots matches or beats web agents built on proprietary giants like GPT-4o, shattering the assumption that open-weight models are inherently inferior for complex browser tasks. Even more surprisingly, running the model four times in parallel and selecting the best result nearly closes the performance gap, pushing a 78% success rate to almost 95% without any model changes.

1. Executive Summary

This paper introduces MolmoWeb, a family of fully open multimodal web agents, and MolmoWebMix, the corresponding open training data mixture. MolmoWeb agents operate as instruction-conditioned visual-language action policies — given a task instruction and a webpage screenshot, they predict the next browser action without access to HTML or accessibility trees — and are trained via supervised fine-tuning on over 100K synthetic task trajectories, 30K+ human demonstrations, atomic web-skill trajectories, and GUI perception data (referring expression grounding and screenshot question answering). Available in 4B and 8B sizes, MolmoWeb agents achieve state-of-the-art results among open-weight models on browser-use benchmarks including WebVoyager, Online-Mind2Web, and DeepShop — MolmoWeb-8B reaches 78.2% on WebVoyager, outperforming set-of-marks agents built on much larger proprietary models like GPT-4o while using only screenshots as input. Test-time scaling via parallel rollouts with best-of-N selection yields substantial further gains (94.7% pass@4 on WebVoyager compared to 78.2% pass@1), establishing that parallel trajectory sampling with a VLM judge provides a more effective inference-compute allocation than simply increasing per-trajectory step budgets.

2. Context and Motivation

The Core Problem: Web Agents Are Increasingly Capable But Fundamentally Opaque

The central gap this paper addresses is not that web agents don't exist — they do, and some are quite effective — but rather that the most capable systems are proprietary black boxes with undisclosed training data, architectures, and recipes. This opacity creates a scientific dead end: researchers cannot understand why these systems work, cannot reproduce their results, and cannot systematically build upon them. The paper states this directly in the introduction:

"the most capable end-to-end systems are typically offered as hosted, proprietary services with limited disclosure of training data and full recipes. This exacerbates long-standing concerns that insufficient reporting and artifacts hinder reproducibility and scientific understanding of what actually drives performance."

This is not merely an academic complaint about reproducibility norms. The paper argues that opacity is especially concerning for autonomous agents operating on the open web, where deployment involves real financial transactions, personal data, and interaction with live services. If an agent books the wrong flight, purchases the wrong item, or fills a form incorrectly, users need to understand why. Proprietary systems that function as opaque services cannot provide this accountability. The paper connects this to broader AI safety and trustworthiness concerns, citing the NIST AI Risk Management Framework and work on language model risk taxonomies — autonomous web agents that act on behalf of users raise the stakes for transparency, auditability, and controllability beyond what is required for, say, a chatbot that merely answers questions.

A Specific Scientific Gap: No Open Recipe for Vision-Only Web Agents

Zooming in from the general problem of opacity, the paper identifies a more specific technical gap: there is no fully open, reproducible recipe for training a web agent that operates purely from screenshots. Existing open-weight models either rely on structured page representations (DOM, accessibility trees) that provide privileged information not available to human users, or they distill from proprietary vision-based agents, which means their training signals ultimately derive from systems whose internals remain hidden. The paper emphasizes this distinction:

"It is also worth noting that, unlike prior work such as Fara, we avoid distillation from proprietary vision-based web agents. Our data pipeline largely relies on human trajectories and synthetic trajectories generated from AxTree agents that do not see screenshots."

This is a crucial design commitment. Distillation from a proprietary teacher (e.g., using GPT-4o or a commercial computer-use model to generate training trajectories) can produce capable student models, but it inherits the teacher's biases, failure modes, and limitations without providing any understanding of why the teacher behaves as it does. The data generation pipeline is part of the scientific contribution, not just an implementation detail — by releasing the full data mixture and generation code, the paper enables the community to inspect, critique, and improve every component of the training process.

Why Vision-Only? The Brittleness of DOM-Based Approaches

The paper's commitment to screenshot-only operation is not arbitrary. It is motivated by a critique of the dominant alternative: agents that parse the Document Object Model (DOM) or accessibility tree (AxTree) to identify interactive elements and predict actions by referencing element IDs. Prior work such as Mind2Web and WebGUM established this as a viable paradigm, and many production systems (including the AxTree agents used to generate MolmoWebMix's synthetic trajectories) rely on it. However, the paper identifies three concrete shortcomings:

1. Distribution shift across websites. DOM structures vary enormously across websites, frameworks, and even minor page updates. An agent trained to locate a "search button" via its AxTree role and ID on one e-commerce site may fail on another site where the same functionality uses a different element hierarchy. Screenshots, by contrast, present a visual interface that is — by design — standardized for human consumption across the web.

2. Incomplete or misleading structured representations. Dynamically rendered content (JavaScript-generated elements, infinite scroll, lazy-loaded images) may not appear in the initial DOM or AxTree, leading the agent to miss critical interactive elements. The paper notes that "structured page representations... can be incomplete or misleading for dynamically rendered content," whereas a screenshot captures exactly what a human user sees at that moment.

3. Token consumption. Accessibility trees can easily consume tens of thousands of tokens per page — the paper explicitly states that "AxTree inputs can easily consume tens of thousands of tokens per page, whereas a single screenshot provides a compact, information-rich representation of the same content." In a multi-step agent loop where the model must process the page representation at every step, this token overhead compounds quickly, increasing latency and compute cost.

The vision-only design is therefore not just a principled commitment to human-like perception. It is also a practical engineering choice that sidesteps the maintenance burden, token cost, and brittleness of DOM parsing while aligning the agent's input distribution with the interface that websites are actually designed to present.

Where Prior Approaches Fall Short

The paper identifies limitations across several categories of existing work:

Proprietary computer-use models (OpenAI, Google). Systems like OpenAI's Computer Use API and Google's Gemini computer-use represent the current capability frontier, but they are offered as hosted services. Their training data, model architectures, and failure modes are undisclosed. As the paper notes, this makes it "impossible to advance the science of these multimodal agentic systems" — researchers can benchmark these systems but cannot study how they work, why they fail, or how to improve them beyond prompt engineering.

Open-weight but not fully open models (Fara, UI-TARS, Holo1). Several recent models have released weights but not training data or full pipelines. Fara-7B, for example, reports strong results on WebVoyager (73.5%) but its training data is not publicly available, and the paper notes that it relies on distillation from proprietary vision-based agents. UI-TARS-1.5-7B performs competitively (66.4% on WebVoyager) but similarly lacks open data. Holo1-7B released some data but not comprehensive training and evaluation infrastructure. These models represent progress toward open-weight availability but stop short of the full scientific openness — data, code, evaluation harness — that the paper argues is necessary for community-driven progress.

Set-of-Marks (SoM) agents on large proprietary models. The SoM approach, introduced by Yang et al. (2023), augments screenshots with visual markers (bounding boxes, numeric labels) overlaid on interactive elements, then prompts a large VLM like GPT-4o to reference these markers in its action predictions. This approach can leverage massive proprietary models and achieves strong results — the paper reports SoM Agent (GPT-4o) at 65.1% on WebVoyager and SoM Agent (o3) at 79.3%. However, SoM agents have two significant limitations: (1) they still require access to the DOM or AxTree to generate the visual markers, so they are not truly vision-only; and (2) they depend on proprietary models that cannot be inspected, fine-tuned, or deployed without API access.

LLM-driven agents with structured inputs (ReAct-style, WebGUM, Mind2Web). A substantial body of work has explored using LLMs to operate on language representations of web pages — typically the accessibility tree serialized as text — with the LLM predicting element IDs to interact with. These systems (which the paper itself uses as trajectory generators) can be effective but inherit all the brittleness, token overhead, and distribution-shift problems of DOM-based representations discussed above. Moreover, because these systems do not see screenshots, they cannot leverage visual cues (layout, color, icons, images) that humans use to navigate websites, potentially missing context that is visually obvious but absent from the structured representation.

GUI grounding and perception in isolation. Several works (SeeClick, Ferret-UI, ScreenSpot, UGround) have studied the problem of GUI element grounding — given a screenshot and a natural language description, predict the pixel coordinates of the corresponding element — as a standalone task. Others (ScreenAI, OmniParser) focus on parsing screenshots into structured representations. While these capabilities are clearly necessary components of a web agent, prior work largely treats them as separate problems rather than integrating them into an end-to-end agent training pipeline. MolmoWeb's approach of including grounding and screenshot QA as auxiliary training objectives within the same fine-tuning stage is a deliberate integration choice that teaches perceptual skills while simultaneously training task-completion behavior.

How This Paper Positions Itself

MolmoWeb positions itself as filling the intersection of three desiderata that no prior system simultaneously achieves: (1) fully open — releasing model weights, training data, data generation code, evaluation harness, and training recipes; (2) vision-only — operating purely from screenshots without requiring DOM, AxTree, or any structured page representation at inference time; and (3) competitive with proprietary systems — achieving state-of-the-art results among open-weight models and matching or exceeding much larger proprietary SoM agents.

The paper explicitly frames this as a response to the opacity problem in web agent research. The release is designed to be comprehensive: model checkpoints, training data, code, and a unified evaluation harness. This is not merely a model release but an attempt to establish an open research platform — a common starting point from which the community can systematically study web agent capabilities, failure modes, data scaling, and algorithmic improvements.

A subtle but important positioning choice is the paper's relationship to the teacher models that generate its training data. While MolmoWebMix's synthetic trajectories are generated by Gemini-3-Flash (an AxTree agent) and a multi-agent harness using GPT-4o and Gemini-2.5-Flash, the paper is careful to note that these teacher agents do not see screenshots — they operate on accessibility tree representations. This means MolmoWeb is not distilling from a proprietary vision-based agent (unlike Fara and similar models). Instead, it translates non-visual demonstrations into a visual policy by pairing AxTree agent trajectories with the corresponding screenshots captured during execution. The student (MolmoWeb) must learn to map visual inputs to the same actions that the teacher derived from structured page representations — a form of cross-modal policy learning that the paper argues is more scientifically interesting and transparent than direct distillation from a proprietary visual agent.

3. Technical Approach

3.1 Reader Orientation

MolmoWeb is a vision-language model fine-tuned to act as a browser action policy: it takes a screenshot of a webpage plus a task instruction and outputs the next low-level browser action (click at coordinates, type text, scroll, navigate to URL). The core problem it solves is translating visual web page perception directly into executable browser actions without any access to the underlying HTML, DOM, or accessibility tree — a task that requires simultaneously learning to see (identifying buttons, text fields, links from pixels), to read (OCR on rendered text), to reason (understanding what step advances the task), and to act (outputting precisely parameterized actions in the correct action space).

The "shape" of the solution is a single end-to-end supervised fine-tuning pipeline: the base Molmo2 vision-language model is trained on a carefully constructed mixture of web interaction demonstrations (both synthetic and human-collected) and GUI perception tasks (grounding, screenshot QA), all formatted in a unified observation-action-thought structure, so that the model learns perceptual grounding, task-completion behavior, and action formatting simultaneously from a single training stage.

3.2 Big-Picture Architecture (Diagram in Words)

The complete MolmoWeb system spans training data generation and agent execution. At a high level:

1. Data Generation Pipelines (MolmoWebMix construction): Multiple complementary pipelines produce web interaction trajectories — synthetic trajectories from AxTree agents (single-agent and multi-agent), human demonstrations collected via a Chrome extension, node-traversal deterministic trajectories, atomic skill segments, grounding pairs from AxTree element enumeration, and screenshot QA pairs from LLM question generation — all unified into a common format of (screenshot, instruction, action history) → (thought, action).

2. Base Vision-Language Model (Molmo2): A pretrained multimodal model consisting of a SigLIP2 vision encoder, a Qwen3 language model, and an adapter connecting them. This model already understands images and text but has not been exposed to web interaction data.

3. Unified Training Mixture (MolmoWebMix): The training dataset configured with specific mixing ratios — 80% task trajectories (synthetic + human + skills), 20% GUI perception data (grounding + screenshot QA) — designed to jointly teach perceptual understanding and task-execution behavior.

4. Supervised Fine-Tuning: A single-stage end-to-end fine-tuning on the mixture, training the vision encoder, language model, and adapter simultaneously using a standard next-token prediction objective on the concatenated (observation, thought, action) sequences.

5. Inference-Time Agent Loop: At deployment, the model runs in a loop — receive screenshot + instruction + action history → predict thought + action → execute action in browser → receive new screenshot → repeat until task completion or step limit.

3.3 Roadmap for the Deep Dive

• First, the MolmoWeb observation and action space — what the model sees and what it outputs — because this defines the interface that all training data must conform to and determines what capabilities the model must learn.
• Second, the architecture and training procedure — what model is fine-tuned, how, and with what hyperparameters — because this establishes the learning framework into which the diverse data sources are fed.
• Third, the synthetic trajectory generation pipelines (AxTree single-agent, multi-agent, node traversal) — because these produce the bulk of the training data and their design determines what behaviors the model learns.
• Fourth, the human trajectory collection pipeline — because human demonstrations provide a complementary data distribution with different action patterns, website coverage, and failure modes.
• Fifth, the atomic skill trajectories — because these isolated skill demonstrations teach compositional building blocks that underpin the longer task trajectories.
• Sixth, the GUI perception data (grounding and screenshot QA) — because these auxiliary tasks teach the model to see and read web pages, an essential prerequisite for task execution.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems and data paper whose core idea is that a relatively small vision-language model (4B–8B parameters) can be trained to operate as an effective web agent purely from screenshots, provided it is trained on a sufficiently diverse and carefully constructed mixture of task demonstrations, skill trajectories, and GUI perception data. The key intellectual contribution is the design of the data generation pipelines and the training mixture composition, not a novel architecture or learning algorithm.

Observation and Action Space

The MolmoWeb agent operates in a standard Markov decision process loop: at each step $t$, it receives an observation, produces an action, the action is executed in the browser, and a new observation is produced. The observation and action spaces are designed to mirror human web interaction as closely as possible while being machine-parseable and learnable.

Observation space. At each step $t$, the model receives three components:

1. Current screenshot of the browser viewport — a single RGB image showing exactly what a human user would see at that moment, including any scroll position, pop-ups, or dynamic content. This is the primary perceptual input and the only representation the model has of the page content.

2. Task instruction — a natural language string describing the overall goal (e.g., "Find the cheapest nonstop flights from Seattle to Tokyo"). This instruction is provided at every step (not just the first step), ensuring the model never loses track of the objective even in long trajectories.

3. Action history — the actions taken in the previous 10 steps, along with the URL and title of the current page. The 10-step history window provides temporal context: the model can see what it just did, whether those actions succeeded or failed, and use that information to decide the next action. The paper states this explicitly: "To provide temporal context, the history of actions taken in 10 prior steps is appended as context along with the URL and title of the current page."

The design choice of a 10-step history window (rather than full history or no history) reflects a trade-off. Full history would provide maximum context but increase sequence length and potentially confuse the model with distant, irrelevant past actions. No history would prevent the model from learning from its own mistakes or maintaining coherent multi-step strategies. Ten steps is an empirical choice — long enough to capture recent context (did I just try to click something and it failed? am I in the middle of filling a multi-field form?) while keeping the input manageable.

Action space. The model outputs actions as JSON objects containing two fields: a thought (natural language rationale) and an action (structured operation specification). The full action vocabulary, listed in Table 3 of the paper, is:

• goto(url) — navigate the browser to a specified URL
• mouse_click(x, y, ...) — click at viewport coordinates normalized to $[0, 100]$
• mouse_drag_and_drop(...) — drag from one coordinate to another
• scroll(delta_x, delta_y) — scroll the page by a pixel offset
• scroll_at(x, y, dx, dy) — scroll at a specific coordinate (e.g., within a scrollable sub-element)
• hover_at(x, y) — hover at a specific coordinate
• keyboard_type(text) — type a string of text (after focusing on an input field via click)
• keyboard_press(key) — press a key or key combination (e.g., "Enter", "Tab")
• go_back() — navigate to the previous page
• new_tab() — open a new browser tab
• tab_focus(index) — switch focus to a specified browser tab
• noop(wait_ms) — wait for a specified duration (e.g., for page load or CAPTCHA)
• send_msg_to_user(msg) — display a message to the user (used for final answers or completion signals)

Spatial coordinates for mouse actions (mouse_click, scroll_at, hover_at, mouse_drag_and_drop) are normalized to the range $[0, 100]$ with 2 decimal places of precision during training. At execution time, these normalized coordinates are denormalized to the actual viewport pixel dimensions. The paper explicitly notes: "Mouse actions are parameterized by spatial coordinates normalized to [0, 100], with 2 decimal points, which are denormalized to viewport pixel coordinates at execution time."

This normalization is a critical design choice. It decouples the model's coordinate predictions from the specific viewport dimensions, making the model robust to different screen sizes, browser window dimensions, and zoom levels. A click at (48.5, 50.2) should land at roughly the center of the viewport regardless of whether the viewport is 1920×1080 or 1366×768 pixels.

The thought field is not optional commentary — it is part of the structured output the model must produce before the action. The thought serves multiple purposes: (1) it acts as an explicit reasoning trace that makes the agent's behavior more interpretable, (2) it provides a form of working memory — since past thoughts are included in the action history, the model can store information it needs later (e.g., "the price was $42.99"), and (3) it structures the action prediction as a chain-of-thought process, which has been shown to improve reasoning accuracy in language models. The paper notes this memory capability: "An interesting capability by virtue of producing thoughts is that MolmoWeb can sometimes use thoughts as memory for storing information during the trajectory and reference it for producing the final answer."

The action space is deliberately chosen to mirror what a human can do with a browser — clicking, typing, scrolling, navigating — rather than exposing lower-level primitives (DOM manipulation, JavaScript execution) or higher-level abstractions (semantic actions like "add to cart"). This choice keeps the agent's capabilities aligned with the visual interface, ensuring that any task the agent can complete is also a task a human could complete through the same interface. The paper notes one tension: certain common action sequences (click to select field + type text + press Enter) require three separate actions, whereas a combined type_at(text, x, y, press_enter=True) action would be more efficient. The current action space prioritizes simplicity and alignment with human behavior over trajectory length optimization.

Architecture and Training Procedure

Base model. MolmoWeb is built on the Molmo2 architecture, which is itself a standard vision-language model design. The architecture consists of three components:

1. Vision encoder: SigLIP2 — a vision transformer that processes the input screenshot and produces a sequence of visual feature tokens. SigLIP2 is a successor to the original SigLIP (Sigmoid Loss for Language Image Pre-training), which uses a sigmoid-based contrastive loss rather than the softmax-based loss of CLIP, providing better scaling properties.

2. Language model: Qwen3 — a transformer-based autoregressive language model that processes the interleaved sequence of visual tokens (from the vision encoder) and text tokens (instruction, action history) and generates the output text tokens (thought + action JSON). The paper uses Qwen3 as the base LLM within Molmo2, which provides strong multilingual text understanding and generation capabilities.

3. Adapter: A connector module (typically a small MLP or attention-based projector) that maps the vision encoder's output tokens into the same embedding space as the language model's text tokens, enabling the language model to process visual and textual information jointly.

The model processes "interleaved sequences of images and text" — at each step, the input is the single screenshot image tokenized by SigLIP2, concatenated with text tokens for the instruction and action history, and the output is a sequence of text tokens for the thought and action.

Training objective. Training uses standard supervised fine-tuning (SFT) with a next-token prediction objective — the standard autoregressive language modeling loss where the model is trained to predict each token given all previous tokens. The loss is computed only on the output tokens (thought + action), not on the input tokens (screenshot, instruction, history). This is the standard approach for instruction fine-tuning of language models: the model learns to produce the correct output sequence conditioned on the input sequence, but is not penalized for "predicting" the input tokens.

The paper does not provide an explicit loss equation, but the standard autoregressive objective for a sequence of tokens $y_1, y_2, ..., y_T$ conditioned on input $x$ is:

$\mathcal{L} = -\sum_{t=1}^{T} \log P(y_t \mid y_{<t}, x)$

where $P(y_t \mid y_{<t}, x)$ is the model's predicted probability for token $y_t$ given all previous output tokens $y_{<t}$ and the input context $x$.

What it computes: for each token in the target sequence (thought + action), the model produces a probability distribution over its vocabulary. The loss is the negative log-probability of the correct token. Summing over all tokens yields the total loss for that training example.

Why this form: this is the maximum-likelihood objective for autoregressive sequence generation, which is the standard objective for training language models to produce coherent, well-structured output sequences. It directly optimizes the model to assign high probability to the correct thought and action given the observation and instruction.

Training configuration. The paper reports the following training hyperparameters:

• Hardware: 64 H100 GPUs with a global batch size of 128
• Training duration: Up to 50K steps (approximately 3.2 epochs on average over the dataset)
• Trainable parameters: "we finetune Molmo2... and tune the language model, the vision encoder, and the adapter starting from the single-image checkpoint"
• Starting checkpoint: The Molmo2 model pretrained on image captioning and fine-tuned on single-image QA — this ensures the model already has strong visual understanding and text generation capabilities before web-specific training begins

The paper explicitly states that all parameters (vision encoder, language model, adapter) are trained, not frozen. This is important: the vision encoder must adapt to web screenshots (which differ significantly from natural images in captioning datasets — more text, structured layouts, UI elements), and the language model must learn to produce the specific JSON-structured action format and web-navigation reasoning patterns.

Data mixing ratios. All data types — task trajectories, atomic skill trajectories, and GUI perception data — are mixed in a single training stage. The mixing ratios are treated as hyperparameters and selected based on performance across evaluation benchmarks. The final ratios, as reported in Table 2:

• MolmoWebMix-Traj (80% overall): 35% AxTree single-agent trajectories, 18% AxTree multi-agent trajectories, 2% AxTree atomic skills, 2% node-traversal trajectories, 18% human trajectories, 5% human skills trajectories
• MolmoWebMix-Perception (20% overall): 15% grounding data (PixMoPoints + SyntheticGround), 5% Screenshot QA

The total training mixture contains 278.5K task trajectories with 2.2M steps (average 13.2 steps per trajectory) plus 10.5M perception examples. The perception data, though it represents only 20% of the mixture by ratio, contains many more individual examples (10.5M vs. 2.2M steps) because grounding and QA examples are single-step — each is one (screenshot, question) → (answer) pair rather than a multi-step trajectory.

Why mix everything in one stage rather than two-stage training? A common alternative would be to first train on perception data only (to build visual grounding), then fine-tune on trajectories (to add task-execution behavior). The paper's choice of single-stage mixing with a carefully tuned ratio reflects the hypothesis that perceptual and task-execution capabilities reinforce each other: the model benefits from seeing grounding examples that teach it to identify interactive elements at the same time as it sees trajectories that require it to use those elements to complete tasks. Joint training allows the vision encoder to develop features that are simultaneously useful for element identification and task-level reasoning, rather than first specializing for grounding and then being partially overwritten during trajectory training.

Inference-time decoding. The paper compares three sampling strategies (Table 7): greedy decoding (temperature 0.0), top-k sampling (temperature 0.7, k=20), and top-p nucleus sampling (temperature 0.7, p=0.8). The default for all experiments is top-p sampling with p=0.8 and temperature=0.7, which achieved 68.5% on WebVoyager versus 61.4% for greedy and 67.4% for top-k. The paper attributes the improvement over greedy decoding to the model's tendency to get "stuck at specific states (e.g., trying to click at the same location or continuing to scroll even when past attempts at doing so have failed)" under deterministic decoding, while randomized strategies allow the model to escape these loops by occasionally sampling alternative actions.

The choice of sampling parameters is based on "Qwen3's (the base LLM used in Molmo2) recommended parameters on HuggingFace," not on an extensive sweep — this suggests the parameters are inherited from the base model's known good generation settings rather than optimized specifically for web agent behavior.

Synthetic Trajectory Generation: AxTree Single-Agent Pipeline

The single-agent AxTree pipeline is the largest source of training trajectories (70K trajectories, 793K steps, covering 1,300 websites). Its design is conceptually straightforward but requires careful engineering.

Teacher agent design. The teacher is an LLM (Gemini-3-Flash-Preview) that operates on the accessibility tree (AxTree) representation of the webpage rather than the screenshot. The AxTree is a structured, serialized representation of the page's interactive elements, where each node has a role (e.g., "button," "combobox," "link"), a name (the visible text or accessible label), and a browser ID (bid) — a unique numeric identifier. Figure 3 in the paper shows an example: the Google Search page serialized as a list like [1] RootWebArea 'Google Search', [2] banner ' ', [3] link 'Go to Google Home', etc.

At each step, the teacher agent receives:

• The serialized AxTree of the current page
• The task instruction
• The history of past actions

It predicts the next action by referencing the bid of the target element — for example, click(11) to click on element with bid 11 (the Google Search button). This sidesteps the visual grounding problem entirely: the teacher knows exactly which element to interact with based on its structured properties.

Trajectory capture. Although the teacher sees only the AxTree, a screenshot is captured at each step — this is the crucial detail that makes the trajectories usable for training a vision-based student. The action predicted in bid-space is programmatically mapped to pixel-space coordinates using the element's bounding box (available from the AxTree). This produces a trajectory where each step consists of:

1. A screenshot (showing the page as a human would see it)
2. An action with pixel-space coordinates (e.g., mouse_click(48.5, 50.2))
3. The teacher's rationale (the thought explaining why that action was chosen)

The mapping from bid to pixel coordinates uses random sampling within the element's bounding box — specifically, "the ground-truth click coordinate is sampled randomly within the element's bounding box using a clipped Gaussian prior centered at the element's center, encouraging the model to learn spatially robust clicking rather than always targeting exact centers." This data augmentation is important: if every click were always at the exact center of the target element, the model might learn to click at center-typical locations and fail when the element's visual appearance is offset from its bounding box center (e.g., an icon-with-text button where the text is left-aligned).

Task sources. Tasks are drawn from:

• Manually authored templates: task templates covering popular websites and benchmark evaluation sites, with placeholders filled from predefined constants
• LLM-generated instructions: tasks generated by prompting an LLM (cycling between GPT-4o, GPT-4.1, GPT-5-mini, and GPT-5 for diversity) to produce instructions similar to benchmark tasks, but not paraphrases — the generation includes cross-domain transfer where a task originally for a shopping website is adapted to another shopping website
• Taxonomy-based generation: tasks systematically sampled from a Cartesian product of four axes — intent (info-seeking, transactional, tool-use, messaging, navigation), domain (13 website categories), difficulty (D0 through D3 based on constraint count), and ambiguity (A0 through A3, from single correct answer to vague/underspecified goals)

Success filtering. After trajectory execution, each trajectory is evaluated for task success using the WebVoyager LLM judge (GPT-4o with the published WebVoyager evaluation prompt). The judge receives the task instruction, the complete trajectory history, and screenshots, and outputs a success/failure decision with a rationale. Only trajectories deemed successful are retained for training. This is a critical quality-control step: training on failed trajectories would teach the model to replicate failure behaviors, and filtering ensures the training data consists only of demonstrations that actually accomplish the task.

This success-filtering step creates a distributional property of the training data that matters: the model only sees trajectories where the task was completed correctly. It never sees what happens when an action fails, what recovery looks like, or how to recognize when a task is infeasible. This is a form of optimism bias in the training distribution — the model learns that every trajectory eventually succeeds, which may contribute to the observed failure mode where the agent "keeps predicting the same action... without being able to recover or course-correct" when stuck.

Generation statistics. The single-agent pipeline produces 70K trajectories covering 1,300 websites with an average of 11.4 steps per trajectory. The paper reports that the Gemini AxTree agent achieves 74.4% on WebVoyager (30-step budget) and 85.6% with 100 steps — establishing the teacher's capability ceiling that the visual student is trying to approximate.

Synthetic Trajectory Generation: Multi-Agent Pipeline

The multi-agent pipeline (Figure 4) is designed to produce higher-quality trajectories by decomposing the web navigation problem across three specialized roles, inspired by the observation that single-agent systems can lose track of the overall goal or pursue inefficient action sequences.

Agent roles. The system orchestrates three agents, each implemented as an LLM call with role-specific prompts:

1. Planner (Gemini-2.5-Flash): Generates the next subgoal given the high-level task goal and verification feedback about task progress. For example, for the task "find a birria taco recipe under 3 hours on foodnetwork.com," the Planner might produce subgoals like "go to foodnetwork.com," then "search for tacos birria," then "find recipe under 3 hours." Each subgoal is a concrete, verifiable intermediate objective.

2. Operator (Gemini AxTree agent): Executes one low-level browser action at each step to accomplish the current subgoal. The Operator receives the current subgoal, completed steps, planner/verifier reasoning, and the current screenshot or AxTree, and returns a concrete action (click, type, scroll, etc.).

3. Verifier (GPT-4o): Checks whether the current subgoal has been completed by analyzing the most recent 5 screenshots. The Verifier's binary judgment (subgoal complete vs. not complete) drives two decisions: if complete, the Planner is called to generate the next subgoal; if not, the system continues attempting the same subgoal.

Execution loop. At each step, the system:

1. Runs the Verifier to check if the current subgoal is complete (based on the most recent 5 screenshots — this window provides enough context to detect state changes without overwhelming context length).

2. If the subgoal is verified as complete or fails to complete within 5 steps (a timeout to prevent infinite loops), the Planner is called to generate the next subgoal.

3. The current subgoal, completed steps, and planner/verifier reasoning are injected into the text prompt, which is passed along with the current screenshot to the Operator.

4. The Operator returns a concrete browser action, which is executed.

5. All historical information (subgoals, actions, verifier judgments) is tracked and fed back into subsequent cycles.

The loop repeats until the overall goal is completed or a maximum step limit is reached.

Why three roles? The decomposition separates concerns that a single agent must handle simultaneously: high-level planning (breaking the task into subgoals), low-level execution (translating subgoals into precise browser actions), and progress monitoring (determining when to move on). The Planner can focus on strategy without being distracted by the mechanics of clicking and typing; the Operator can focus on precise action execution given a clear subgoal; and the Verifier provides explicit feedback that helps the system avoid both premature subgoal transitions (moving to the next step before the current one is done) and infinite loops (trying the same action repeatedly).

Empirical benefit. The paper reports that the multi-agent setup "achieves higher task completion success rates than using the Gemini AxTree agent alone, scoring 78.5 vs. 74.4 on WebVoyager." This 4.1 percentage point improvement suggests that the explicit decomposition and verification feedback meaningfully improves trajectory quality, though the gain is modest relative to the additional complexity and compute cost of running three separate LLMs per step.

Data contribution. The multi-agent pipeline produces 35K trajectories with 438K steps across 1,100 websites, averaging 12.5 steps per trajectory. These trajectories are presumably higher-quality (more efficient, fewer detours, better aligned with the stated goal) than single-agent trajectories, though the paper does not present a direct quality comparison beyond the WebVoyager success rate difference.

Synthetic Trajectory Generation: Node Traversal Pipeline

The node traversal pipeline (Figure 5) takes a fundamentally different approach: instead of using LLMs to generate trajectories, it uses deterministic graph traversal to produce navigation trajectories that are guaranteed to be executable and verifiable.

Graph construction. The pipeline first builds a directed graph over 500 popular websites via breadth-first exploration:

1. Starting from each website's homepage, extract the accessibility tree to identify all navigational links (links, buttons, menu items that lead to other pages).
2. Prompt an LLM to select a diverse set of the most informative links (e.g., category pages, search features, content sections — avoiding trivial links like "Privacy Policy" or "Terms of Service").
3. For each selected link, follow it to the target page and repeat the process, continuing to a depth of four.
4. Prune cycles to avoid loops (a page visited previously is not expanded again).

The result is a graph where nodes are URLs and edges represent navigational links between pages. Each root-to-leaf path through the graph represents a valid navigation sequence from the homepage to some deeper page.

Deterministic trajectory execution. To convert a chosen URL sequence into a trajectory with browser actions:

1. Start at the root URL.
2. For each pair of consecutive URLs in the path (current → next), locate the link to the next target page in the current page's accessibility tree.
3. If the link is not visible in the current viewport, scroll until it becomes visible.
4. Click on the link.
5. Verify that the browser navigated to the expected URL. If navigation succeeds, continue to the next step. If it fails (e.g., the link was removed, the target URL changed, or a pop-up blocked navigation), truncate the path to the last successfully visited page.
6. At the terminal page, use an LLM to generate a plausible task instruction that is consistent with the path taken.

Why deterministic? This approach has several advantages over LLM-generated trajectories:

1. Verified correctness: The trajectory is known to be executable because it was produced by actually executing the path. There is no LLM judge needed to verify success — URL matching provides ground-truth verification.
2. Computational efficiency: No LLM calls are needed during trajectory execution — only the final instruction generation requires an LLM. This makes the pipeline cheap to run at scale (16K trajectories, 151K steps).
3. Complementary data distribution: The trajectories cover different behavior patterns than LLM-generated ones — they are purely navigational (going from page A to page B through a specific route) without the form-filling, searching, or question-answering behaviors present in other pipelines. The paper states these trajectories "resemble a goal-directed browser demonstration" once the LLM-generated instruction is paired with them.

Limitations. Node traversal trajectories are structurally simple — they only involve clicking and scrolling, never typing, form-filling, drag-and-drop, or other more complex interactions. They are also biased toward navigation-heavy tasks and may not cover information-seeking behaviors that require reading and comparing content across pages. Their contribution to the mixture is small (2% by ratio), suggesting they are treated as a supplementary data source rather than a primary one.

Human Trajectory Collection

The human trajectory pipeline provides 36K trajectories (623K steps, averaging 20.8 steps each — notably longer than synthetic trajectories, which average 11.4–12.5 steps) across 1,100 websites, plus an additional 116K atomic skill segments extracted from these trajectories.

Collection infrastructure. Crowdworkers use a custom Chrome extension that captures browser interaction events (clicks, scrolls, keystrokes) and corresponding screenshots. The extension records a stream of timestamped DOM events and screenshots, which are post-processed into a clean sequence of (screenshot, action) pairs. The paper notes that this post-processing is non-trivial: "Collecting high-accuracy human trajectories... is challenging due to various quirks of the Chrome extension, particularly around capturing screenshots along with DOM events."

Workers are instructed to wait long enough after each action for the page to complete loading and for the extension to trigger a screenshot automatically, or to manually trigger a screenshot if automatic capture fails. Quality-control interventions include warning messages when workers act too quickly and annotation training to ensure consistent behavior.

Task structure. Each task instruction is decomposed into an ordered sequence of subtasks using the atomic skill taxonomy defined in Table 1. For example, a shopping task might be decomposed as:

go to: walgreens.com
search: coffee
apply filters: brand=Lavazza, availability=Pickup
find and open: most relevant product

Workers check off each subtask upon completion within the annotation tool. If a subtask cannot be completed due to missing content or unexpected page state, workers record their best attempt along with a descriptive note explaining the failure. At the end, workers submit a final text response (answer to a question or completion acknowledgment).

This decomposition serves multiple purposes: (1) it provides workers with clear step-by-step guidance, reducing ambiguity about what to do; (2) it enables automatic extraction of atomic skill segments (each subtask becomes its own training example); and (3) it provides structured metadata about task complexity and required skills.

Quality assurance. Each trajectory is reviewed by a human to verify task completion and ensure correct capture of screenshots and actions. Trajectories that fail review are revised or re-collected. The paper partnered with Snorkel AI to manage the annotation workforce — they "provided tasks and our annotation tool to Snorkel AI who then managed the annotation workers, verified the annotations for correctness, and ensured quality control."

Task sampling strategies. To ensure diversity, tasks are generated through four strategies:

1. Manually written templates: Authors wrote task templates for common use cases (shopping, news, real estate, travel, maps, food, jobs, health, cars). Each template specifies a sequence of atomic skills with placeholders populated from predefined constants or generated on the fly.

2. LLM-sampled tasks with steps: A persona from PersonaHub (a dataset of 1 billion diverse personas) is sampled, and an LLM is prompted to generate web tasks for a given website using the atomic skill taxonomy. The persona sampling amplifies task diversity by covering many different user profiles and intents.

3. LLM-sampled navigation and QA tasks: A more flexible format where the LLM generates a task as "navigate: [navigation instruction]\n question: [question about target page]" without prescribing the atomic steps. This produces more open-ended tasks that may require exploration.

4. LLM-sampled benchmark-like tasks: Given all tasks for a specific website from WebVoyager and Online-Mind2Web as in-context examples, an LLM generates similar but non-paraphrased tasks. This closes the distribution gap between training and evaluation tasks.

Instruction specificity levels. Each task is generated at multiple levels of specificity for training:

• Step-by-step: The most detailed, listing each atomic step explicitly
• Low-level: A colloquial paraphrase of the atomic steps into natural English
• Mid-level: Less verbose, omitting details that may be obvious from context
• High-level: Only the intent, without prescribing how to achieve it (e.g., "Find Lavazza coffee for pickup at walgreens")

During training, one of the four levels is randomly sampled for each trajectory, with slightly higher probability for the high-level instruction. This teaches the model to handle varying levels of instruction detail — important because real users will not always provide step-by-step guidance.

Human vs. synthetic trajectory differences. Human trajectories average 20.8 steps versus 11.4–12.5 for synthetic, suggesting that humans take longer, more exploratory paths to complete tasks. The paper hypothesizes that "humans tend to exhibit more exploratory behavior, particularly on unfamiliar websites, resulting in longer and noisier trajectories with detours that may hinder imitation learning." This is supported by the ablation in Table 6, where training on 2,700 human trajectories yields substantially lower benchmark performance than training on 2,700 synthetic trajectories for the same tasks (e.g., 35.4% vs. 53.0% on WebVoyager).

Additionally, human trajectories contain actions that are rare or absent in synthetic data — specifically scroll_at (scrolling within a sub-element) and mouse_drag_and_drop. The paper notes that when trained on the combined synthetic+human data, the model "almost always produces scroll which tries to scroll the page instead of the element," suggesting that the model struggles to learn the distinction between page-level and element-level scrolling when both are present in the training data with different distributions.

Atomic Skill Trajectories

Atomic skill trajectories isolate individual web interaction skills from the full task-completion context. The taxonomy in Table 1 defines 11 skills: go_to, search, find, find_and_open, find_and_click, fill_form, fill_form_and_submit, apply_filters, apply_filters_and_search, add_to_cart, and navigate. The paper argues that "providing targeted supervision for each skill ensures the model develops reliable competence in these building blocks."

Extracted from human trajectories. Because human task trajectories were annotated with ordered subtask decompositions, each subtask segment can be automatically extracted as a standalone skill trajectory. For example, if a full task trajectory includes the segments "search: coffee," "apply filters: brand=Lavazza," and "find and open: most relevant product," each of these becomes its own training example with the corresponding instruction and the segment's screenshots and actions.

A key property of these extracted segments is that each "begins from the browser state in which the previous subtask ended." This means the skill demonstrations are not from clean initial states (like a fresh homepage) but from intermediate states within longer tasks — the agent might learn to apply filters on a search results page that already shows search results, rather than always applying filters immediately after searching. The paper notes this is beneficial because it "resembles response to a follow-up query to the user's previous query" and may "enable the agent to learn multi-turn interaction with the user."

Generated by an AxTree agent. To supplement the extracted segments, additional skill trajectories are generated by prompting the AxTree agent with targeted skill instructions for two specific skills: fill_form and find_and_open. Instructions follow the format go to:[URL]\n fill form:[form details] or go to:[URL]\n find and open:[target]. This directly yields skill-level trajectories without requiring segmentation of longer task trajectories.

Data contribution. The skill trajectories contribute 5.5K synthetic (AxTree atomic skills, 68.7K steps, average 12.4 steps) and 116K human-extracted segments (781K steps, average 6.8 steps). The much larger count of human-extracted skills (116K vs. 5.5K synthetic) reflects the decomposition of 36K human trajectories into many subtasks each, whereas the synthetic skill generation is a targeted supplement for specific skills.

GUI Perception Data: Grounding

Grounding data teaches the model to map natural language descriptions of page elements to their pixel coordinates — the fundamental perceptual capability required for any click-based interaction.

Data generation from AxTree agent trajectories. For each screenshot captured during AxTree agent trajectory execution, the pipeline enumerates all clickable elements in the AxTree (buttons, links, input fields, checkboxes, etc.). For each clickable element, a natural language description is generated using its accessible name and role. Two generation methods are used:

1. Template-based: A rule-based system generates descriptions like "Click on the 'Sign In' button" or "Click the 'Search' combobox" from the element's AxTree properties. This produces 3.4M examples with consistent, predictable phrasing.

2. GPT-5-generated: GPT-5 is prompted to generate more natural, varied descriptions for the same elements — producing queries with different syntactic structures, vocabulary choices, and levels of specificity. This produces 3.8M examples with much higher linguistic diversity.

The ground-truth click coordinate for each element is sampled randomly within the element's bounding box using a clipped Gaussian prior centered at the element's center. The paper explains this choice: "encouraging the model to learn spatially robust clicking rather than always targeting exact centers." If the model always sees clicks at the exact center, it might learn to click at visually central locations regardless of the element's visual boundaries. Gaussian sampling around the center introduces variation while keeping clicks within the element, teaching the model that any point within the element's visual extent is acceptable.

Inclusion of PixMoPoints data. In addition to newly generated grounding data, the paper repurposes the PixmoPoints dataset from the original Molmo work — "formatting single-point QA pairs into click actions" — adding 1.1M examples. This is an efficient way to leverage existing high-quality grounding annotations without duplicating data collection effort.

Total grounding data: 8.3M examples (3.4M templated + 3.8M GPT-5-generated + 1.1M PixMoPoints), constituting 15% of the training mixture by ratio.

Why so much grounding data? Grounding is the perceptual bottleneck for a vision-only web agent. Before the model can decide which action to take, it must be able to identify where the relevant element is. The large volume of grounding data ensures the model develops robust element localization capabilities across diverse websites, element types, and description styles. The inclusion of both templated and LLM-generated descriptions ensures coverage of both consistent, learnable patterns and natural linguistic variation that the agent will encounter from real user instructions.

GUI Perception Data: Screenshot QA

Screenshot QA data teaches the model to read and reason about webpage content — answering questions that require OCR, visual understanding, and reading comprehension from a single screenshot.

Generation process. For each screenshot in a subset of the AxTree agent trajectories, the corresponding AxTree is provided to an LLM (the paper does not specify which LLM) along with a prompt to generate question-answer pairs. The AxTree provides ground-truth information about what text and elements are present on the page, enabling the LLM to generate accurate questions and answers without needing to visually parse the screenshot itself.

Questions cover three categories:

1. OCR queries (54% of the data): Questions about text and values present on the page — prices, counts, product names, dates, text content. Example: "What is the name of the shoe with the red stripe?" with answer "Adizero Boston 13." These require the model to locate and read specific text within the screenshot.

2. Affordance queries (26%): Questions about what actions are available on the page — "Where would I find financial news on this page?" or "How would I sort these results by price?" These require understanding the functional structure of the interface, not just reading text.

3. Summarization queries (20%): Questions about the overall content or purpose of a page element — "What is this page about?" or "Summarize the key features of this product." These require higher-level comprehension of page structure and content.

Quality filtering. To ensure questions and answers rely solely on visual content (since the student model only sees screenshots, not AxTrees), the pipeline removes samples containing references to AxTree-specific information — specifically, "element IDs (e.g., 'Click on Bid 32')." This prevents the model from learning to rely on non-visual cues that would not be available at inference time.

Data scale. The Screenshot QA dataset covers 395 websites and contains 2,237,252 QA pairs. This large volume provides broad coverage of visual question types and website layouts.

Why Screenshot QA? Many web tasks require the agent to answer questions based on page content — a shopping task might require finding the price of an item, a travel task might require reading flight times, a research task might require extracting specific facts. Screenshot QA directly trains this capability in isolation, without the confounding factors of action selection and multi-step planning present in full trajectories. By including both trajectories (which require reading content as part of completing tasks) and QA pairs (which isolate the reading capability), the model learns both the perceptual skill and its integration into task-execution behavior.

Summary of Design Choices and Their Justifications

• Screenshot-only input over AxTree/DOM: avoids brittleness to website-specific DOM changes, reduces token consumption (single image vs. tens of thousands of text tokens), and aligns with human perceptual interface.
• Single-stage mixed training over two-stage (perception then trajectories): allows perceptual and task-execution capabilities to co-develop, with the vision encoder learning features simultaneously useful for both.
• AxTree agents as teachers over proprietary vision-based teachers: enables transparency in data generation (the AxTree is a deterministic, inspectable representation), avoids distillation from opaque proprietary systems, and ensures the student learns a genuinely distinct capability (vision → action mapping) rather than mimicking a visual teacher.
• Multi-agent decomposition over single-agent: improves trajectory quality (78.5 vs. 74.4 on WebVoyager) by separating planning, execution, and verification into specialized roles with explicit feedback loops.
• Node traversal deterministic trajectories over purely LLM-generated: provides guaranteed-correct navigation demonstrations at low computational cost, covering a complementary data distribution.
• Human trajectories with subtask annotation over unstructured collection: enables extraction of atomic skill segments, provides structured quality assurance, and creates multi-turn interaction patterns that synthetic data lacks.
• Grounding with Gaussian coordinate sampling over exact-center clicking: teaches spatial robustness — the model learns that any point within an element's visual extent is valid, not just the geometric center.
• Screenshot QA from AxTree-derived questions over human-written QA: enables large-scale generation (2.2M pairs) with ground-truth answers verified against the structured page representation, while filtering removes AxTree-specific artifacts to maintain visual-only validity.
• Thoughts as required output over action-only: provides interpretability, creates a working memory mechanism (past thoughts in history can store information), and structures action prediction as chain-of-thought reasoning.
• 10-step action history over full history: balances context provision (enough to track recent actions and detect loops) with input length management (longer histories increase compute cost and may distract with irrelevant past actions).

4. Key Insights and Innovations

Innovation 1: Openness as a Scientific Contribution, Not Just a Release Policy

The paper's most distinctive intellectual move is treating comprehensive openness — data, code, evaluation harness, training recipes, and model weights — as a first-class scientific contribution rather than a supplementary "release" appended to a methods paper. This is not merely commendable open-science practice; it is a deliberate intervention in a field where, as the paper argues, "the most capable end-to-end systems are typically offered as hosted, proprietary services" and even open-weight alternatives like Fara-7B and Holo1-7B stop short of releasing training data or full pipelines.

To appreciate why this framing is novel, consider the norm it pushes against. In web agent research, the dominant mode of progress has been: a lab with access to proprietary infrastructure releases a model or API demonstrating state-of-the-art results, accompanied by a paper that describes the approach at a high level but omits the precise data mixtures, filtering decisions, hyperparameter sweeps, and failure modes that actually determined performance. Other labs then attempt to replicate or build upon these results with incomplete information, leading to what the paper's citations on reproducibility (Gundersen and Kjensmo, 2018; Pineau et al., 2021) identify as a systemic problem in AI research. The paper positions MolmoWeb as a direct response: not "here is a better model," but "here is a complete, inspectable research platform."

The specific openness commitments matter. Releasing model weights alone (as Fara and UI-TARS do) enables inference and limited fine-tuning but tells researchers nothing about why the model behaves as it does — what data shaped it, what distributional biases it inherited, what failure modes are baked into its training. Releasing training data without the generation code (pipelines, prompts, filtering logic) makes the data a static artifact rather than a reproducible process. Releasing code without the evaluation harness means results cannot be independently verified or compared under consistent conditions. MolmoWeb releases all four, and the paper treats this as the central contribution — the abstract leads with the data (MolmoWebMix), the model family is second, and the release commitments (checkpoints, data, code, evaluation harness) close the introduction.

The significance extends beyond norms. By releasing the full data generation pipeline — including the prompts used for LLM-based task generation, the multi-agent orchestration code, the human annotation infrastructure, and the post-processing logic — the paper makes it possible for researchers to interrogate the relationship between training data characteristics and agent behavior. Without this transparency, questions like "does training on synthetic trajectories from an AxTree teacher bias the model toward teacher-like strategies?" or "what happens when you remove human data from the mixture?" or "how does task instruction specificity affect robustness?" can only be studied by the originating lab. With full openness, they become community-level scientific questions that any lab can investigate through controlled data ablations.

This reframes MolmoWeb not as a model competing with GPT-4o on a leaderboard, but as an infrastructure contribution analogous to what ImageNet provided for computer vision or what the Pile provided for language model pretraining — a shared, inspectable foundation that accelerates collective progress by making systematic experimentation possible. The paper's experiments (ablation studies in Section 4.4, data scaling analyses in Table 5, grounding benchmarks in Table 8) are demonstrations of the kind of science the platform enables, not merely validation of a particular model checkpoint.

Innovation 2: Cross-Modal Policy Learning from Non-Visual Teachers

A subtle but structurally important conceptual move is MolmoWeb's approach to training a vision-based student from teachers that do not see screenshots. The synthetic trajectory generation pipelines — single-agent AxTree, multi-agent, and node traversal — all produce demonstrations using teachers that operate on accessibility tree representations, not pixel inputs. The student, MolmoWeb, never sees the AxTree at training or inference time; it must learn to map screenshots to the same actions the teacher produced from structured page representations.

This is fundamentally different from the dominant paradigm in vision-based web agent training, which typically involves one of two approaches: (1) direct distillation from a proprietary visual agent (e.g., using GPT-4o or a computer-use model to generate screenshot-conditioned trajectories, as Fara-7B does), or (2) training on trajectories where the teacher and student share the same observation modality (e.g., using human demonstrations where both the human and the model see the same screenshots).

The cross-modal approach has several non-obvious implications that distinguish it conceptually:

First, it decouples teacher capability from visual perception ability. The AxTree teacher can be highly capable at web navigation without being able to see — its strengths come from the structured, symbolic representation that explicitly labels every interactive element with its role, name, and unique ID. This means the teacher's competence ceiling is determined by planning and reasoning quality, not by visual grounding accuracy. The student's training task is then to learn the perceptual mapping that bridges the teacher's symbolic world to the visual world — a cleaner separation of concerns than if the teacher also made visual mistakes that the student would inherit.

Second, it avoids the circularity problem of distilling from proprietary visual agents. If a student is trained on trajectories from GPT-4o's computer-use mode, it learns to mimic whatever strategies, biases, and failure modes GPT-4o exhibits — but since GPT-4o's training is opaque, the student's behavior is ultimately unexplained. The AxTree teacher's behavior, by contrast, is mechanistically interpretable: at each step, it selects an element from a known, finite set (the AxTree nodes), and the mapping from selected element to pixel coordinates is deterministic and inspectable. Researchers can study why the teacher chose a particular element by examining the AxTree representation it received, which is not possible with a black-box visual teacher.

Third, it creates an implicit form of data augmentation. Because the teacher's actions are mapped to pixel coordinates via the element's bounding box (with Gaussian sampling around the center), each trajectory provides a slightly different pixel-space realization of the same symbolic action. If the teacher clicks "the Search button" on Google, that button's pixel coordinates vary with viewport size, scroll position, and element layout — the same AxTree bid maps to many different pixel locations across different executions. The student learns that the visual appearance of the button, not a specific coordinate, is what matters.

The paper provides evidence that this cross-modal learning is more effective than learning from visual demonstrations: Table 6 shows that training on 2,700 synthetic (AxTree-sourced) trajectories yields 53.0% on WebVoyager versus 35.4% for 2,700 human trajectories on the same tasks. The paper's hypothesis — that "the LLM agent operates on the accessibility tree, which encodes rich structural and semantic information about page elements that may not be immediately apparent from visual cues alone" — suggests that the structured representation provides a higher-quality teaching signal than human visual demonstrations, even though the student ultimately operates from pixels. This is a counterintuitive finding with practical implications for how training data for visual agents should be generated: the best teacher for a visual student may be one that doesn't see at all.

Innovation 3: Test-Time Scaling via Parallel Rollouts as a Diagnostic for Single-Rollout Brittleness

The paper's test-time scaling results (Section 4.3, Figure 6) are not merely an impressive performance number — the jump from 78.2% pass@1 to 94.7% pass@4 on WebVoyager is striking — but a diagnostic signal about the nature of the remaining single-rollout failure modes. The magnitude of the gain from parallel rollouts reveals something about the agent that the pass@1 number alone obscures: the model is capable of successfully completing most tasks, but its single-trajectory reliability is limited by compounding errors that parallel sampling can overcome.

This is a conceptually distinct use of test-time compute from the standard "best-of-N as a performance booster" narrative. In that standard framing, best-of-N is an inference trick: run the model multiple times, pick the best output, and report a higher number without changing the underlying model. The MolmoWeb results invite a different interpretation: the large pass@1-to-pass@k gap is evidence that the agent's failure modes are stochastic rather than systematic. If the model were fundamentally incapable of completing certain tasks — lacking the necessary perceptual or reasoning capabilities — parallel rollouts would provide no benefit, because all rollouts would fail in the same way. The fact that pass@4 nearly saturates on WebVoyager (94.7%) means that for most tasks where pass@1 fails, there exists some sampling path that succeeds.

This is a reframing with direct implications for what kind of algorithmic improvement is most promising. If failures were systematic — the model consistently cannot handle form-filling on insurance websites, or always misreads prices with dollar signs — the appropriate response would be targeted data collection or architectural changes. The stochasticity result instead suggests that the primary bottleneck is not capability but reliability: the model knows how to solve most tasks, but occasionally takes wrong turns (clicks the wrong search result, scrolls past the target, misreads a date) from which it cannot recover. This points toward solutions that improve trajectory-level robustness — better error detection and recovery mechanisms, verifier-guided backtracking, or RL fine-tuning that penalizes unrecoverable errors — rather than solutions that teach fundamentally new skills.

The comparison with increasing per-trajectory step budget reinforces this interpretation. The paper explicitly notes that "gains from scaling via parallel runs (each with 30 steps) and picking the best result using an LLM judge far outperform increasing the number of inference steps (e.g., 8B model achieves 86.2% via 3 parallel runs with max 30 steps resulting in a total of 90 steps compared to 78.2% via increasing the steps to 100 in one run)." If the failure mode were simply running out of time to complete complex tasks, more steps would close the gap. The fact that more steps provides only a modest gain (78.2% → ~78.2% from 30 to 100 steps, based on the numbers reported) while parallel rollouts provide ~16 percentage points suggests that when the model goes off-course, it tends to stay off-course — additional steps don't fix the error, they compound it. Parallel rollouts avoid this by restarting from a clean initial state, giving the model fresh opportunities to take the correct path.

The paper hints at the practical implication: "self-distillation from best-of-N rollouts and RL might be effective strategies for further improving single rollout performance." If the underlying capability exists but is unreliably accessed, distillation from successful parallel trajectories could teach the model to take the paths that lead to success more consistently — effectively converting the stochastic success signal from best-of-N into improved single-rollout behavior.

Innovation 4: The Data Mixture as a Deliberate Curriculum Design

Beyond the raw scale of MolmoWebMix, the paper's approach to constructing the training mixture as a deliberate combination of complementary data types with distinct pedagogical roles represents a conceptual advance over the more common "collect as much data as possible and train on all of it" approach in web agent training. The mixture is not an ad-hoc aggregation of available data sources; each component is chosen and weighted to teach a specific capability, and the interactions between components are empirically studied.

The paper's ablation in Table 5a is revealing: approximately 85–90% of performance is achieved with just 10% of the dataset. This is not a failure of data scaling — it is evidence that the composition of the mixture matters more than the raw volume. A well-constructed 10% subset that preserves the mixture proportions teaches most of what the model needs to learn; the remaining 90% provides incremental refinement and coverage of edge cases. This pattern is reminiscent of the finding in language model fine-tuning that a small number of high-quality, diverse instruction examples can capture most of the benefit of much larger datasets — but applied specifically to the multi-capability challenge of web agents.

The mixture components play distinct roles that can be understood through a capability decomposition lens:

• GUI perception data (grounding + screenshot QA) teaches the model to see — to identify interactive elements, read text, and understand page structure from pixels alone. This is the foundational perceptual layer without which no task-execution behavior is possible. Its inclusion at 20% of the mixture, despite being "auxiliary" to the task-completion objective, reflects the paper's recognition that perceptual capability is the bottleneck for vision-only agents.

• Synthetic trajectories (AxTree single-agent and multi-agent) teach the model to act — to translate task goals into sequences of browser operations. These trajectories are efficient, goal-directed, and cover diverse websites, but they reflect the teacher's structured-representation perspective and may miss visual cues that humans use.

• Human trajectories teach the model to explore and recover — human demonstrations are longer (20.8 average steps vs. 11.4–12.5 for synthetic), contain more exploratory behavior, and include rare actions (scroll_at, drag_and_drop) that synthetic trajectories lack. They also introduce variability in action style and pacing that may improve robustness.

• Atomic skill trajectories teach the model to compose — by isolating individual skills, they ensure the model develops reliable competence in each building block rather than only learning them implicitly through full task trajectories. This is a form of curriculum design: decompose complex behavior into components, teach each component explicitly, then train on the full composition.

• Node traversal trajectories teach the model to navigate deterministically — they provide guaranteed-correct demonstrations of link-following behavior at low computational cost, covering a complementary distribution focused on site structure exploration.

The paper's finding that "synthetic and human data represent distinctly different web task completion policies and the model struggles to learn generalizable behavior across both" (Table 5b, where synthetic+human performs similarly to synthetic-only) is a negative result with conceptual significance. It suggests that the standard intuition of "more diverse data is always better" has limits when data sources reflect fundamentally different policies — in this case, the efficient, AxTree-guided policy of the synthetic teacher versus the noisy, exploratory policy of human annotators. The model faces a multi-task learning problem where the optimal action distribution for a given state differs between data sources, and simply mixing them can create conflicting training signals. The fact that the paper reports this honestly, rather than hiding it behind cherry-picked hyperparameters, demonstrates the value of the open-science approach: negative results about data mixing strategies are as scientifically informative as positive results about benchmark performance.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on four live-website browser-use benchmarks: WebVoyager (He et al., 2024), Online-Mind2Web (Xue et al., 2025), DeepShop (Lyu et al., 2025), and WebTailBench (Awadallah et al., 2025). WebVoyager's task set uses the version from Fara (Awadallah et al., 2025) which removed infeasible tasks. Dates in outdated task instructions are updated to remain meaningful at evaluation time (e.g., "find a flight on August 5, 2025" is adjusted to a current date).

• Base model(s). MolmoWeb is built on Molmo2 (Clark et al., 2026), which uses a SigLIP2 vision encoder, a Qwen3 language model, and an adapter connecting them. Models are fine-tuned in two sizes: 4B and 8B parameters. All parameters — vision encoder, language model, and adapter — are fine-tuned. Training starts from the Molmo2 single-image checkpoint (pretrained on image captioning and fine-tuned on single-image QA).

• Metrics. The primary metric is task completion accuracy (%) — the fraction of evaluation tasks for which the agent successfully completes the assigned goal. Success is determined by an LLM judge (not programmatic verification). For WebVoyager and DeepShop, the judge is GPT-4o with the published benchmark prompt. For Online-Mind2Web, the judge is o4-mini with the official prompt. For WebTailBench, since the original judge is unspecified, the WebVoyager judge is used. Results are averaged across 3–5 evaluation runs per benchmark per model, with each run capped at a maximum number of inference steps (30 or 100 depending on the experiment). Tasks that do not complete within the step limit or within 10 environment retry attempts are marked as failures.

• Baselines. The paper compares against three categories of prior work:

  Proprietary API models: Set-of-Marks (SoM) agents built on GPT-4o, o3, and GPT-5 (numbers from Fara, Awadallah et al., 2025, marked with * in Table 4), which receive both AxTree and SoM-annotated screenshots as input. Also included are OpenAI computer-use-preview, Gemini computer-use-preview, and Yutori Navigator (a proprietary system with only Online-Mind2Web results reported at 64.7%).

  Open-weight models: Fara-7B (Awadallah et al., 2025), UI-TARS-1.5-7B (ByteDance Seed, 2025), GLM-4.1V-9B-Thinking (Hong et al., 2025), and Holo1-7B (Andreux et al., 2025). Fara-7B reports 73.5% on WebVoyager, 34.1% on Online-Mind2Web, and 26.2% on DeepShop. Holo1-7B reports only 55.4% on WebVoyager with a 30-step budget.

  Non-visual teacher agents: The AxTree agent backbones used for trajectory generation — Gemini-3-flash and GPT-5 — evaluated with the same AxTree-based interface they used during data generation, providing an upper bound on what the visual student might approximate.

• Generation budget / compute accounting. The paper does not use a unified generation budget for fair comparison across methods because different baselines use fundamentally different interfaces (vision-only screenshots vs. AxTree vs. SoM-annotated screenshots). Instead, comparisons are made at a fixed maximum inference step budget (30 or 100 steps). For test-time scaling experiments (Section 4.3), the compute budget is measured in number of parallel rollouts k, with pass@k estimated using Equation 1 from m=5 total rollouts.

• Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported for the main benchmark comparisons. The paper acknowledges this implicitly by averaging across 3–5 evaluation runs to reduce variance from stochastic environment behavior and model sampling. For data ablation studies (Section 4.4), a single training run is used per ablation configuration with no confidence intervals reported. The test-time scaling pass@k estimator (Equation 1) provides an unbiased estimate from m > k rollouts, but no variance estimates (confidence intervals, standard errors) are computed.

Main Quantitative Results

Comparison to Prior Work (Table 4)

Headline result: MolmoWeb-8B achieves 78.2% on WebVoyager, 35.3% on Online-Mind2Web, 42.3% on DeepShop, and 49.5% on WebTailBench, establishing new state-of-the-art among fully open models and outperforming larger proprietary SoM agents on several benchmarks.

Table 4 presents a comprehensive comparison organized by model category. Among open-weight models, MolmoWeb-8B leads on all four benchmarks. The specific comparisons:

Versus open-weight models at comparable scale. MolmoWeb-8B outperforms Fara-7B (the previous strongest open-weight model) by 4.7 percentage points on WebVoyager (78.2% vs. 73.5%), 1.2 points on Online-Mind2Web (35.3% vs. 34.1%), 16.1 points on DeepShop (42.3% vs. 26.2%), and 11.1 points on WebTailBench (49.5% vs. 38.4%). Against UI-TARS-1.5-7B, the gains are 11.8 points on WebVoyager (78.2% vs. 66.4%) and 30.7 points on DeepShop (42.3% vs. 11.6%). MolmoWeb-4B achieves 75.2% on WebVoyager, already outperforming all open-weight models except Fara-7B on that benchmark, and leads all open-weight models on DeepShop at 35.6%.

Versus much larger closed SoM agents. MolmoWeb-8B at 78.2% substantially exceeds SoM Agent (GPT-4o) at 65.1% on WebVoyager — a 13.1-point advantage despite using only screenshots while the SoM baseline receives both AxTree and visually annotated screenshots, and despite GPT-4o being substantially larger (exact parameter count undisclosed). On DeepShop, MolmoWeb-8B's 42.3% beats SoM Agent (GPT-4o)'s 16.0% by 26.3 points. Against stronger proprietary models, MolmoWeb-8B trails SoM Agent (o3) on WebVoyager (78.2% vs. 79.3%) by 1.1 points — essentially matching within likely evaluation noise — and on Online-Mind2Web, MolmoWeb-8B's 35.3% trails o3's 55.4% by 20.1 points, a substantial gap. On DeepShop, MolmoWeb-8B's 42.3% is 7.4 points behind o3's 49.7% and 6.8 points behind GPT-5's 49.1%.

Versus non-visual teacher (Gemini AxTree agent). With a 100-step budget, the Gemini-3-flash AxTree agent — which generates MolmoWeb's training trajectories — achieves 85.6% on WebVoyager, 44.8% on Online-Mind2Web, 55.3% on DeepShop, and 63.5% on WebTailBench. MolmoWeb-8B trails by 7.4, 9.5, 13.0, and 14.0 points respectively. The paper attributes this gap to three factors: (i) the teacher likely has many more parameters than the 8B student, (ii) the teacher uses AxTree element IDs for precise targeting rather than pixel-space pointing, and (iii) the teacher reads text from the AxTree rather than requiring OCR from screenshots.

MolmoWeb-4B vs. MolmoWeb-8B scaling. The 8B model improves over the 4B model by 3.0 points on WebVoyager, 4.0 on Online-Mind2Web, 6.7 on DeepShop, and 5.7 on WebTailBench — consistent but moderate gains suggesting that data quality and diversity may contribute more to performance than parameter count within this range.

Test-Time Scaling (Figure 6)

Headline result: Parallel rollouts with best-of-N selection provide massive gains — pass@4 with MolmoWeb-8B reaches 94.7% on WebVoyager and 60.5% on Online-Mind2Web — and parallel sampling far outperforms increasing per-trajectory step budget.

Figure 6 plots pass@k as k increases from 1 to 4 for both MolmoWeb-4B and MolmoWeb-8B on WebVoyager and Online-Mind2Web, with two step budgets (30 and 100 steps) shown as separate traces.

WebVoyager results. For MolmoWeb-8B with 100 steps, the trajectory is: pass@1 = 78.2%, pass@2 = 89.0%, pass@3 = 92.8%, pass@4 = 94.7%. The gain from pass@1 to pass@2 is 10.8 percentage points; pass@2 to pass@3 adds 3.8 points; pass@3 to pass@4 adds 1.9 points — diminishing returns consistent with the unbiased estimator saturating as the success probability approaches 1. For MolmoWeb-4B with 100 steps, the trajectory is: pass@1 ≈ 75.2% (reading from Figure 6, exact value not quoted), pass@2 ≈ 86%, pass@3 ≈ 90%, pass@4 ≈ 92%. The 8B advantage over 4B is maintained at all k values, with the gap narrowing slightly at higher k.

Online-Mind2Web results. Gains are even more dramatic proportionally: MolmoWeb-8B with 100 steps goes from pass@1 = 35.3% to pass@2 = 48.5% (+13.2 points), pass@3 = 55.8% (+7.3 points), pass@4 = 60.5% (+4.7 points). The total gain of 25.2 percentage points from pass@1 to pass@4 represents a 71% relative improvement. MolmoWeb-4B with 100 steps reaches pass@4 approximately 56% (reading from Figure 6).

Step budget comparison. The paper explicitly contrasts two ways of spending inference compute: increasing the per-trajectory step budget vs. running parallel trajectories. For MolmoWeb-8B on WebVoyager: 100-step single rollout achieves 78.2%; 3 parallel 30-step rollouts (total 90 steps) with best-of-3 selection reaches 86.2%; 100 steps with best-of-3 reaches 92.8%. The parallel strategy achieves substantially more per unit of total inference compute, a finding the paper highlights: "gains from scaling via parallel runs (each with 30 steps) and picking the best result using an LLM judge far outperform increasing the number of inference steps."

Interpretation. The large pass@1-to-pass@k gap on Online-Mind2Web (35.3% → 60.5%) indicates that many tasks the model can complete in some rollout fail in most individual rollouts — the capability exists but is unreliably accessed. The near-saturation on WebVoyager (78.2% → 94.7%) suggests that most remaining pass@1 failures are stochastic rather than systematic: the model can solve nearly all WebVoyager tasks, just not consistently on the first try.

Training Data Ablations (Tables 5–6)

Headline result: Performance improves with data scale but plateaus early (~85–90% of final performance from 10% of data); synthetic trajectories are substantially more effective training signals than human demonstrations for the same tasks.

Table 5a examines the effect of training data volume (using an earlier, smaller version of MolmoWebMix). Training on 1% of the data yields 44.5% on WebVoyager and 11.7% on Online-Mind2Web. Scaling to 10% yields 63.2% and 20.4% — capturing 92% and 93% of the full-dataset performance (68.5% and 21.9% at 100%, respectively — note these are from the earlier dataset, not the final MolmoWebMix, so absolute numbers are lower than the final model). The final 90% of data provides diminishing returns, adding only 5.3 and 1.5 percentage points.

Table 5b ablates the contribution of human vs. synthetic data: human-only (28K trajectories) achieves 27.8% on WebVoyager and 13.2% on Online-Mind2Web; synthetic-only (106K trajectories) achieves 67.8% and 22.0%; synthetic+human (134K trajectories) achieves 68.5% and 21.4% — essentially no improvement from adding human data to the synthetic mixture, and a slight degradation on Online-Mind2Web. The paper interprets this as evidence that "synthetic and human data represent distinctly different web task completion policies and the model struggles to learn generalizable behavior across both." A specific qualitative failure: human data contains scroll_at (scrolling within a sub-element), but models trained on the combined mixture almost exclusively produce scroll (page-level), suggesting the model defaults to the majority behavior from the larger synthetic data source and fails to learn the rare action.

Table 6 performs a controlled comparison: 2,700 trajectories collected by both humans and the AxTree agent for the exact same set of tasks. Training on synthetic trajectories yields 53.0% on WebVoyager and 16.8% on Online-Mind2Web; training on human trajectories for the same tasks yields 35.4% and 9.0% — a gap of 17.6 and 7.8 points in favor of synthetic data. The paper hypothesizes two reasons: (1) human trajectories are longer and noisier (20.8 vs. 11.4–12.5 average steps), with exploratory detours that confuse imitation learning, and (2) the AxTree representation encodes structural and semantic information that enables the teacher to produce more direct, consistent trajectories.

Sampling Strategy Comparison (Table 7)

Headline result: Stochastic sampling (top-p with p=0.8, temperature=0.7) outperforms greedy decoding by 7.1 percentage points (68.5% vs. 61.4% on WebVoyager).

Table 7 compares three decoding strategies on WebVoyager using the earlier dataset version: greedy decoding (temperature 0.0) achieves 61.4%; top-k sampling (temperature 0.7, k=20) achieves 67.4%; top-p nucleus sampling (temperature 0.7, p=0.8) achieves 68.5%. The gap between greedy and stochastic strategies is substantial — over 5 percentage points, or roughly 10% relative improvement. This is notably larger than typical decoding strategy effects in language modeling, suggesting a specific failure mode: the paper reports that greedy models "could get stuck at specific states (e.g., trying to click at the same location or continuing to scroll even when past attempts at doing so have failed)," while stochastic strategies can break out of these loops by sampling alternative actions.

Grounding Evaluation (Table 8)

Headline result: MolmoWeb-Ground-8B (a grounding specialist trained only on grounding data) achieves 88.7% on ScreenSpot and 91.8% on ScreenSpot v2, outperforming several much larger proprietary models including Claude 3.7 and OpenAI CUA.

Table 8 presents grounding accuracy on two benchmarks. ScreenSpot (Cheng et al., 2024) and ScreenSpot v2 (Wu et al., 2024) evaluate the ability to ground natural language element descriptions to pixel coordinates. MolmoWeb-Ground-8B achieves 88.7% on ScreenSpot (second only to Gemini-3-Pro at 93.7%) and 91.8% on ScreenSpot v2 (behind Gemini-3-Pro's 93.7% but ahead of OpenAI CUA at 87.9%, Claude 3.7 at 87.6%, Fara-7B at 89.3%, and Holo1-7B at 89.9%). MolmoWeb-4B, the full agent (not a grounding specialist), achieves 87.2% on ScreenSpot and 89.5% on ScreenSpot v2 — only 1.5 and 2.3 points behind the specialist, suggesting that the agent training retains strong grounding capability while adding task-completion behavior.

Ablation Studies and Robustness Checks

Data scale ablation (Table 5a): Performance follows a saturating curve — 1% of data yields roughly 65% of full-data WebVoyager performance (44.5% vs. 68.5%), and 10% yields roughly 92% (63.2%). This suggests the mixture composition (which is preserved proportionally in the subset) provides most of the learning signal, with the remaining 90% of data adding only incremental coverage of rare websites, actions, and instruction formats.

Human vs. synthetic data contribution (Table 5b): Human data alone produces weak agents (27.8% on WebVoyager vs. 67.8% for synthetic-only), and adding human data to synthetic mixtures provides negligible benefit (68.5% vs. 67.8%) or slight degradation on Online-Mind2Web (21.4% vs. 22.0%). This is a negative result with practical implications for data collection strategy — expensive human demonstrations may not be cost-effective relative to scaling synthetic generation, at least for the benchmark distribution studied.

Controlled human-vs-synthetic comparison (Table 6): When task distribution is held constant (same 2,700 task instructions), synthetic trajectories provide substantially more effective training data across all three benchmarks (DeepShop: 24.4% vs. 19.8%; WebVoyager: 53.0% vs. 35.4%; Online-Mind2Web: 16.8% vs. 9.0%). The consistency across benchmarks strengthens the claim that synthetic data quality, not merely task distribution differences, drives the advantage.

Sampling strategy (Table 7): Greedy decoding substantially underperforms stochastic strategies (61.4% vs. 67.4–68.5%). This is not a hyperparameter sensitivity issue — the gap likely reflects a genuine behavioral difference (loop-stuck behavior under greedy) that would affect any deterministic decoding approach.

Step budget vs. parallel rollouts (Figure 6): Increasing the per-trajectory step budget from 30 to 100 provides modest gains for MolmoWeb-8B on WebVoyager (from approximately 75% to 78.2% — the paper does not quote the 30-step number separately, but it's visible in Figure 6), while parallel rollouts with 30 steps each provide substantially larger total-effective-compute gains. This robustness check confirms that the benefit of parallel rollouts is not simply an artifact of increased total step count.

Critical Assessment

Does the paper demonstrate that MolmoWeb agents are "state-of-the-art" among open-weight models?

Yes, but with important caveats about what "state-of-the-art" means. Table 4 shows MolmoWeb-8B leading all open-weight models on all four benchmarks by margins ranging from 1.2 points (Online-Mind2Web vs. Fara-7B) to 30.7 points (DeepShop vs. UI-TARS-1.5-7B). This is a clear quantitative lead. However, several factors complicate the comparison:

First, the evaluation conditions are not fully standardized across baselines. The paper notes that for WebTailBench, "it's unclear what judge was used by [Fara] for WebTailBench we used the WebVoyager judge" — meaning the numbers for Fara and UI-TARS on WebTailBench may not be directly comparable to MolmoWeb's because the success criterion differs. The paper marks these with † in Table 4, which is appropriate disclosure but makes the WebTailBench comparison less definitive.

Second, the step budget varies across baselines. Holo1-7B is evaluated at 30 steps (55.4% on WebVoyager), while MolmoWeb-8B at 100 steps achieves 78.2%. The paper does not report MolmoWeb-8B at 30 steps on WebVoyager separately (the number is visible in Figure 6 but not quoted in Table 4), making it difficult to assess how much of the gap is model capability versus step budget.

Third, Fara-7B includes WebVoyager data in its training, and UI-TARS-1.5-7B may as well (training data is not fully disclosed). MolmoWeb's training includes WebVoyager tasks as in-context examples for task generation, which means task distribution overlap exists but is indirect — the model was not trained on WebVoyager trajectories themselves. The paper's claim of benchmark-like task generation (Appendix C.1.2) acknowledges this: "we generated tasks that match the distribution of WebVoyager." How much this inflates WebVoyager performance relative to models without such task-generation alignment is unknown and not ablated.

Does the paper demonstrate that MolmoWeb agents "outperform set-of-marks agents built on much larger closed frontier models like GPT-4o"?

Yes, on specific benchmarks — but the comparison is asymmetric in important ways. On WebVoyager, MolmoWeb-8B (78.2%) convincingly exceeds SoM Agent (GPT-4o) at 65.1% — a 13.1 point gap at presumably 100× or more parameter disadvantage. On DeepShop, the gap is even larger (42.3% vs. 16.0%). These are genuine achievements.

However, the comparison has structural asymmetries that the paper acknowledges but does not fully emphasize in the main text. The SoM baselines are zero-shot prompted — they receive a SoM-annotated screenshot and are asked to predict the next action using only in-context examples, with no fine-tuning on web interaction data. MolmoWeb is extensively fine-tuned on 278.5K web trajectories. This is a fundamentally different regime of task-specific training investment. A fairer comparison would be: (1) fine-tune GPT-4o-level models on MolmoWebMix and compare, or (2) evaluate MolmoWeb zero-shot (without fine-tuning) against the SoM baselines. Neither comparison is performed.

The paper implicitly acknowledges this asymmetry by noting that "a well-trained, vision-only MolmoWeb agent achieves stronger task completion rates — suggesting that data quality and targeted training can compensate for raw model scale and additional inputs." This framing (training compensates for scale) is accurate but incomplete — it doesn't isolate whether the advantage comes from the training data, the fine-tuning process, or the architecture. An experiment where the same training data (MolmoWebMix) is used to fine-tune GPT-4o via API (if permitted) would disentangle these factors, but such an experiment is not feasible given API access constraints.

Furthermore, the SoM baselines receive BOTH AxTree (for element identification) AND visually annotated screenshots. This means they have access to strictly more information than MolmoWeb's screenshot-only input. On DeepShop, where MolmoWeb-8B dramatically outperforms SoM Agent (GPT-4o) (42.3% vs. 16.0%), the information advantage is irrelevant because the model cannot use it effectively without task-specific training — a finding that actually strengthens the paper's argument about the importance of training data over input richness.

Does the paper demonstrate that test-time scaling via parallel rollouts provides "substantial further gains"?

Yes, the gains are large and well-measured (Figure 6), but two limitations temper the practical significance.

First, the pass@k estimator (Equation 1) uses m=5 total rollouts and k up to 4. This means pass@4 is estimated from only 5 trajectories per task — a small sample size that could produce noisy estimates, especially on harder benchmarks like Online-Mind2Web where the per-task success probability is low. The paper does not report confidence intervals on the pass@k estimates, making it difficult to assess whether the 94.7% pass@4 on WebVoyager is distinguishable from, say, 92% or 97%. With m=5 rollouts of which c succeed, the possible pass@k estimates are discretized — the estimator can only take specific values, and the uncertainty around those values depends on both m and the true success probability. For high true success probabilities (as on WebVoyager), the variance is relatively low; for low probabilities (Online-Mind2Web), it could be substantial.

Second, the best-of-N selection mechanism uses the same LLM judge that is used for ground-truth evaluation. In a real deployment, the agent must select the best rollout without access to the ground-truth evaluation judge. The paper does not address this circularity — it's unclear whether a separate VLM judge trained or prompted for rollout selection (without knowing the correct answer) would achieve comparable selection accuracy. If rollout selection requires task-specific success evaluation (which is what the benchmark judge does), then pass@k is an oracle upper bound that may not be achievable in practice.

Do the data ablation experiments genuinely support the claim that synthetic data is more effective than human data?

The evidence is strong for the specific comparison performed, but the comparison is narrow in ways that limit generalizability. Table 6's controlled comparison (same 2,700 tasks, different data collection methods) is well-designed and shows a clear 17.6-point WebVoyager gap favoring synthetic. However, several factors warrant caution:

First, the human demonstrations were collected by crowdworkers following structured subtask decompositions and quality-control protocols that may not be representative of "natural" human browsing. The annotation tool imposed structure (checking off subtasks, waiting for screenshots) that likely made human trajectories more artificial than genuine user behavior. The paper's hypothesis that humans are "noisier" may partially reflect annotation tool constraints rather than inherent human behavior.

Second, the synthetic trajectories were filtered for success using the WebVoyager judge, while human trajectories were only reviewed for general quality and subtask completion — not subjected to the same automated success filtering. If some human trajectories that passed human review would have failed the LLM judge, this introduces a systematic quality difference unrelated to collection method.

Third, the table's results (35.4% vs. 53.0%) are on the earlier dataset version, which the paper notes had "fewer human trajectories as well as fewer synthetic trajectories." The absolute numbers are lower than the final model, and it's unclear whether the relative advantage of synthetic data would persist at the full data scale.

What experiments would have strengthened the paper?

Difficulty-stratified evaluation. The paper reports aggregate benchmark performance but never analyzes which tasks MolmoWeb succeeds or fails on, or whether certain task types (form-filling, navigation, information extraction, multi-constraint shopping) disproportionately drive the performance numbers. A breakdown by task category, website, or difficulty level would reveal whether the model's capabilities are broad or concentrated in specific task families.

Ablation of data source interactions. The paper ablates human-vs-synthetic (Table 5b) but does not systematically ablate other mixture components: What happens if you remove Screenshot QA? What if you remove grounding data? What if you remove node-traversal trajectories? What if you remove the multi-agent trajectories but keep single-agent? Understanding which data sources are essential and which are supplementary would provide actionable guidance for future data collection.

Comparison to fine-tuned baselines. The strongest open-weight baselines (Fara-7B, UI-TARS-1.5-7B) are evaluated as-is, without fine-tuning on MolmoWebMix. Fine-tuning these models on the same data and evaluating under identical conditions would isolate whether MolmoWeb's advantage comes from the base model (Molmo2), the training data (MolmoWebMix), or the training recipe — currently these factors are confounded.

Evaluation on non-benchmark websites. All four benchmarks use specific, known websites. Whether MolmoWeb's capabilities transfer to arbitrary websites not represented in training or benchmark distributions is unmeasured. A zero-shot evaluation on held-out website categories or newly encountered sites would test generalization, which is the core promise of a vision-only approach.

Wall-clock latency measurement. The paper claims vision-only design reduces token consumption compared to AxTree-based approaches but never measures actual inference latency. For a practical web agent, the time between screenshot and action matters — if the vision encoder is slow or if coordinated denormalization adds overhead, the token savings may not translate to latency improvements.

Confidence intervals on benchmark results. All results are reported as point estimates from 3–5 evaluation runs, with no variance estimates. Given the stochasticity of live-website evaluation (network latency, page load times, CAPTCHAs, A/B testing by websites), the true uncertainty around benchmark scores could be substantial. Without confidence intervals, it's impossible to assess whether, for example, the 1.2-point gap between MolmoWeb-8B and Fara-7B on Online-Mind2Web (35.3% vs. 34.1%) is statistically meaningful or within noise.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for and Dominates the Practical Budget

The assumption or constraint. The compute-optimal framework conditions strategy selection on estimated prompt difficulty, but the paper's difficulty estimation method requires generating 2,048 samples per question and scoring them with the PRM. The paper acknowledges this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. Generating 2,048 samples to estimate difficulty consumes more compute than the largest test-time budgets studied (256–512 generations). In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. The 4× figure should therefore be understood as an upper bound on achievable efficiency in the current formulation, not a realized deployment gain. The paper frames this as a research artifact that "future work" can address, but the absence of a practical difficulty estimation mechanism means the headline efficiency claim does not reflect what a deployed system would actually achieve today.

What evidence exists in the paper. The difficulty estimation procedure is described in Section 3.2, which specifies 2,048 samples per question. The paper's oracle and predicted difficulty bins (Figures 4 and 8) are computed using this expensive procedure and are the foundation for all compute-optimal scaling results. No experiment measures the total cost including difficulty estimation, and no experiment evaluates a cheaper difficulty estimator that could make the approach practical at deployment scale.

Mitigation status. The paper does not attempt to solve this problem. It acknowledges the limitation and suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) or adaptive difficulty estimation that amortizes the cost into the problem-solving process. Without such a mechanism, the compute-optimal framework remains a valuable analytical tool but is not directly deployable as described.

Hard Problems Remain Essentially Unsolved — Test-Time Compute Amplifies Capability But Cannot Create It

The assumption or constraint. The compute-optimal framework assumes the base model has some non-trivial probability of producing correct solutions. For problems where the base model's pass@1 is near zero, no amount of search, revision, or adaptive allocation can help — there are no correct solutions in the proposal distribution to find or refine. The paper is transparent about this in the Section 7 takeaway, noting that test-time compute "cannot compensate for fundamental capability gaps that larger pretraining would address."

The consequence. On difficulty bin 5 (the hardest quintile of MATH problems), accuracy hovers at 1–3% regardless of method or budget (Figure 3, right; Figure 7, right). In the FLOPs-matched comparison, the hardest problems show a −52.9% relative disadvantage for PRM search versus the 14× larger model at high inference-to-pretraining ratios (Figure 9, right). The approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems — which may be precisely the ones where users most need assistance — pretraining remains the only viable path.

What evidence exists in the paper. The difficulty-bin analyses consistently show near-zero performance on bin 5 across all methods and budgets. Figure 3 (right) shows bin 5 accuracy at 1–3% for both beam search and best-of-N at all budget levels. Figure 7 (right) shows bin 5 accuracy at roughly 2–3% across all sequential-to-parallel ratios. Figure 9 shows the bin 5 scaling line essentially flat near 0–5% even as test-time compute increases, while the 14× larger model's performance (stars) is consistently above it.

Mitigation status. The paper does not attempt to address hard problems. The compute-optimal policy is fundamentally bounded by what the base model can produce, and no allocation strategy can create capability that does not exist. The paper's framing — that test-time compute amplifies existing capability rather than creating it — is honest but leaves a clear boundary condition: this approach is suitable for problems within the base model's approximate capability range and unsuitable for problems outside it.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate with Only Patchwork Mitigations

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect followed by a correct target. As a consequence, at test time, when the model produces a correct answer in an early revision step, it may "revise" that correct answer into an incorrect one because it has never seen an example of what to do when the current answer is already correct. The paper reports (Section 6.1) that approximately 38% of correct answers get converted back to incorrect ones using a naive approach.

The consequence. This reversion rate fundamentally limits the effectiveness of long revision chains. If each step has a non-trivial probability of corrupting a correct answer, then longer chains do not monotonically improve — they oscillate between correct and incorrect answers. The paper mitigates this with within-chain selection (majority voting or verifier-based selection across the entire chain, not just taking the last revision), but these are post-hoc patches. They require the system to evaluate every answer in the chain and pick the best one, which adds selection overhead and may itself be imperfect. The underlying problem — that the model was never trained to recognize when no revision is needed — is not addressed.

What evidence exists in the paper. The 38% reversion rate is reported in the main text (Section 6.1). Figure 6 (left) shows the revision model's pass@1 trajectory: accuracy improves from roughly 18.2% at step 1 to roughly 24–25% by steps 15–20, but the curve is not monotonically increasing — it oscillates around 23–25% for the remainder of the chain (out to 64 steps). The plateau and oscillation are consistent with the reversion phenomenon. The ReSTᵉᴹ experiment (Appendix K, Figure 16) provides further evidence: attempting to optimize the revision model with RL-style training caused performance to degrade substantially with sequential revisions, suggesting the revision capability is fragile and sensitive to training methodology.

Mitigation status. Partial mitigation via within-chain selection, but the underlying problem is not solved. The paper acknowledges that a "more principled solution — such as training the model to recognize when no revision is needed — is not explored" (Section 6 limitations discussion). The selection mechanisms (majority voting, verifier-based selection) are effective enough to show net benefits from revisions in aggregate (Figure 6, right), but they add complexity and do not prevent the model from wasting computation generating incorrect revisions to correct answers.

Single Benchmark (MATH), Single Model Family (PaLM 2-S*), and a Small Test Set Limit Generalizability

The assumption or constraint. All experiments in the paper use only the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The paper states it "believe[s] this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an empirical claim that is not verified. Several aspects of the findings could be specific to this model-benchmark combination rather than general properties of test-time compute scaling:

• The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties or different error patterns might exhibit different difficulty-dependent scaling curves.
• The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families.
• MATH consists exclusively of competition-level math problems requiring symbolic reasoning. It is unclear whether the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems) generalize to other reasoning domains (code generation, logical reasoning, scientific QA) or to tasks requiring factual knowledge rather than inference.

The consequence. Without replication on other models and benchmarks, we cannot distinguish universal findings (e.g., "test-time compute strategies should be conditioned on difficulty") from model-specific or domain-specific artifacts. The paper's key insights about difficulty-dependent allocation are compelling and well-evidenced within the studied setting, but a practitioner working with a different model family, a different reasoning domain, or a different difficulty distribution may find that the optimal strategies differ. The 500-question test set, split into five difficulty quintiles of roughly 100 questions each, and then further split by two-fold cross-validation, means the compute-optimal policy is selected based on approximately 50 questions per fold per bin. This is a small sample that could introduce substantial variance in the learned policy, and the paper does not report confidence intervals on the compute-optimal scaling curves.

What evidence exists in the paper. Section 4 states that all experiments use MATH and PaLM 2-S*. Appendix D describes PRM training on 12,000 MATH training questions. The FLOPs-matched comparison in Section 7 uses the same model family scaled to approximately 14× more parameters. No cross-model or cross-benchmark experiments are reported. The paper acknowledges "future work" on extending to other domains but does not provide any evidence that the findings transfer.

Mitigation status. Not addressed. The paper does not attempt to validate findings on other benchmarks or model families, and it does not discuss domain-specific confounders that might limit generalizability. A reader deploying these techniques on code generation tasks or with a different model architecture should treat the quantitative thresholds (difficulty bins, optimal strategies per bin) as suggestive rather than directly transferable.

Verifier Over-Optimization Is Documented But Not Solved, Setting a Hard Ceiling on Search-Based Scaling

The assumption or constraint. All search-based test-time compute strategies rely on the PRM to evaluate solution quality. However, the PRM is an imperfect proxy for ground-truth correctness, and aggressive optimization against it finds solutions that score highly under the PRM but are actually incorrect. The paper documents this as a central limiting factor (Section 5.3): beam search degrades easy-problem performance at high budgets (Figure 3, right), lookahead search — the strongest optimizer — paradoxically performs worst overall (Figure 3, left), and qualitative examples show search producing degenerate outputs (repetitive low-information steps, overly short 1–2 step solutions) that score highly under the PRM (Appendix M).

The consequence. This means that the current approach to test-time compute scaling via search has a hard ceiling determined by verifier quality, and that ceiling is reached well before the generation budget is exhausted. The paper's compute-optimal policy mitigates this by routing easy problems away from aggressive search (avoiding the over-optimization regime), but it does not eliminate the ceiling — on medium-difficulty problems where beam search is deployed, over-optimization still limits how far scaling can go. The beam search curves in Figure 3 flatten and sometimes decline at moderate budgets (64–256 generations), meaning that doubling the compute budget from 128 to 256 provides negligible or negative returns. Any further scaling of test-time compute via search is gated on improving verifier robustness, which the paper does not address.

What evidence exists in the paper. Figure 3 (right) shows beam search accuracy decreasing on bin 1 (easiest) as budget increases from 4 to 256 generations — the clearest quantitative evidence of over-optimization. Figure 3 (left) shows lookahead search, the most powerful optimizer, consistently underperforming simpler methods at the same generation budget. Appendix M (Figure 29 and surrounding examples) provides qualitative evidence of degenerate outputs that score highly under the PRM. The paper explicitly identifies over-optimization as a phenomenon and notes it as a bottleneck, but its compute-optimal strategy treats it as a constraint to route around rather than a problem to solve. The FLOPs-matched comparison in Section 7 shows test-time compute losing to pretraining on hard problems at higher R values, which is partly attributable to verifier quality limits.

Mitigation status. The paper mitigates over-optimization indirectly through the compute-optimal allocation policy — routing easy problems to best-of-N (weaker optimization) where the verifier is reliable, and using beam search only on medium problems where the verifier signal has more room to provide genuine guidance. This is a routing fix, not a verifier fix. The paper does not explore methods to improve verifier robustness (adversarial training, ensembles, KL-constrained search), and it identifies verifier over-optimization as a key area for future work in Section 8 without proposing concrete solutions.

Sequential Revision Strategies Are Fundamentally Serial, Creating a Latency-Versus-Accuracy Tradeoff the Paper Does Not Address

The assumption or constraint. The paper measures test-time compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock latency. Sequential revision strategies — where each revision depends on the output of the previous revision — are inherently serial: revision step t cannot begin until step t-1 completes. In contrast, parallel best-of-N strategies can execute all N samples simultaneously with sufficient hardware.

The consequence. A compute-optimal strategy that allocates 256 generations as 128 sequential × 2 parallel would take approximately 128× longer wall-clock time than a strategy that runs 256 parallel samples simultaneously, even though total FLOPs may be comparable. The paper's finding that sequential revisions are optimal on easy problems (Section 6, Figure 7, right) and that moderate sequential-to-parallel ratios are optimal on hard problems means that the recommended strategies are precisely those that maximize latency. For latency-sensitive applications — interactive assistants, real-time decision-making, any deployment where users wait for responses — the sequential-heavy strategies favored by the compute-optimal policy may be impractical regardless of their accuracy advantages. The paper does not discuss this tradeoff, measure latency, or evaluate strategies under a latency-constrained budget.

What evidence exists in the paper. Section 6 describes the sequential revision mechanism and reports the compute-optimal sequential-to-parallel ratios (Figure 7), including the finding that fully sequential dominates on easy problems. The paper measures everything in generation counts and reports accuracy, with no latency measurements or time-to-completion analyses. The test-time scaling experiments in Section 4.3 use parallel rollouts (which are latency-friendly since all rollouts run independently), but the revision experiments in Section 6 use sequential chains that are latency-unfriendly. The paper never compares these two approaches under equivalent latency constraints.

Mitigation status. Not addressed. The paper does not measure latency, does not frame any experiment in terms of wall-clock constraints, and does not discuss the latency implications of its recommended strategies. For a practitioner deciding between, say, parallel best-of-256 (high latency if fully parallelized on limited hardware, but potentially much lower if hardware is abundant) and 64 sequential revisions × 4 parallel (high latency from the serial chain regardless of hardware), this omission means the paper provides no guidance on the practical deployment tradeoff.

7. Implications and Future Directions

How This Work Changes the Landscape

MolmoWeb represents a reframing of what "state-of-the-art" means for web agents — shifting the axis of progress from raw benchmark scores achieved by opaque proprietary systems toward transparent, reproducible, and scientifically inspectable systems that the community can collectively build upon. This is not a paradigm shift in agent architecture (the model is a standard VLM fine-tuned with SFT) nor a theoretical breakthrough, but it is a methodological intervention that changes what evidence is required to claim progress in web agent research. Before MolmoWeb, a paper reporting 78.2% on WebVoyager could be published without disclosing training data, without releasing model weights, and without providing a reproducible evaluation pipeline. After MolmoWeb, such a paper faces a new baseline expectation: the community now has a fully open system achieving that performance level, and any claim of superiority must be weighed against the transparency gap — is a 2-point improvement meaningful if it comes from an undisclosed training mixture and an unreproducible pipeline?

This reframing is important because it addresses a specific, well-documented failure mode in AI research that the paper explicitly cites: the "insufficient reporting and artifacts [that] hinder reproducibility and scientific understanding" (Section 1, citing Gundersen and Kjensmo, 2018 and Pineau et al., 2021). In web agent research specifically, the opacity problem is acute because the most capable systems are API-gated services. Researchers cannot inspect GPT-4o's web navigation behavior to understand why it fails on certain tasks, cannot study its training data to identify coverage gaps, and cannot fine-tune it to test hypotheses about capability acquisition. MolmoWeb makes all of these scientific activities possible for the first time at a competitive performance level.

The paper also reconciles a latent tension in the web agent literature between two data generation philosophies. On one side, the dominant approach among proprietary systems has been to train on massive datasets of undisclosed composition, often involving distillation from human demonstrations or from other proprietary models. On the other side, the open-weight community has struggled with the chicken-and-egg problem of needing high-quality training data to build capable agents, but lacking capable agents to generate that data. MolmoWeb resolves this by demonstrating that non-visual teachers (AxTree agents) can generate effective training data for visual students — the teacher doesn't need to see screenshots to teach the student to see. Table 6's result (53.0% vs. 35.4% on WebVoyager for synthetic vs. human trajectories on identical tasks) provides concrete evidence that cross-modal policy learning from structured-representation teachers can be more effective than learning from visual human demonstrations. This finding makes synthetic data generation from AxTree agents a newly attractive research direction — previously it might have seemed like a compromise (using non-visual teachers because visual ones are unavailable), but the paper shows it may be an advantage.

The large pass@1-to-pass@k gap on Online-Mind2Web (35.3% → 60.5% at pass@4, a 71% relative improvement) changes the diagnostic landscape for web agent evaluation. Before this result, a reasonable researcher might have assumed that a 35.3% pass@1 model simply lacks the capability to complete 64.7% of tasks. The pass@k result reveals instead that the model can complete many of those tasks, but unreliably — the capability exists but is stochastically accessed. This shifts the research priority from "acquiring new capabilities" to "improving the reliability of existing capabilities," which calls for different methods (better error recovery, verifier-guided backtracking, RL fine-tuning for consistency) than the former (more data, larger models, new architectures). The parallel-vs-sequential step budget comparison reinforces this: increasing per-trajectory steps from 30 to 100 provides marginal gains, while parallel 30-step rollouts with best-of-3 selection dramatically outperforms 100 single-trajectory steps. This is a diagnostic that other web agent papers should adopt as standard reporting — pass@1 alone obscures the reliability-versus-capability distinction.

The paper also makes certain research directions less attractive. The finding that human demonstrations provide limited or negative benefit when added to synthetic mixtures (Table 5b: synthetic+human achieves 68.5% vs. synthetic-only 67.8% on WebVoyager) suggests that expensive human data collection, at least using the annotation protocols described in the paper, may not be cost-effective for improving benchmark performance. This doesn't mean human data is useless — it may be valuable for teaching rare actions (scroll_at, drag_and_drop) or for safety alignment — but as a primary driver of task-completion capability, synthetic generation appears superior. Similarly, the finding that the multi-agent pipeline (Planner + Operator + Verifier) provides only a 4.1-point improvement over the single-agent AxTree pipeline on WebVoyager (78.5 vs. 74.4) while requiring substantially more complex infrastructure and LLM calls per step suggests that sophisticated multi-agent orchestration may have diminishing returns for trajectory generation — simpler, cheaper single-agent generation at larger scale may be the more efficient investment.

Follow-Up Research This Work Enables

Decomposition of the synthetic-vs-human advantage: is it trajectory length, action consistency, or teacher knowledge? The paper's finding that synthetic trajectories substantially outperform human trajectories for training (53.0% vs. 35.4% on WebVoyager, Table 6) is well-documented but not fully explained. The paper hypothesizes two causes: (1) human trajectories are longer and noisier, and (2) the AxTree provides richer structural information that enables more direct trajectories. A controlled experiment could disentangle these: take the synthetic trajectories and artificially corrupt them — add random exploratory detours (to simulate human noise), remove thought annotations, or add variable scroll amounts — and measure how much each corruption degrades training effectiveness. Conversely, take human trajectories and clean them — remove detour steps, standardize scroll amounts, add structured-thought annotations post-hoc — and measure recovery. If noise alone accounts for the gap, cleaning human data should close it. If the AxTree's structural knowledge provides an irreducible advantage, the gap would persist even with perfectly clean human trajectories. This experiment would provide actionable guidance for future data collection: if noise is the primary factor, invest in better filtering and cleaning pipelines for human data; if the AxTree advantage is fundamental, invest in AxTree-based synthetic generation and treat human data as a supplementary source for specific skills.

Can MolmoWeb be fine-tuned with RL on live-website success signals to improve single-rollout reliability? The pass@1-to-pass@k gap (35.3% → 60.5% on Online-Mind2Web) reveals that the model possesses capability that it accesses unreliably. The paper explicitly suggests that "self-distillation from best-of-N rollouts and RL might be effective strategies for further improving single rollout performance." A natural follow-up would fine-tune MolmoWeb using outcomes from its own rollouts as a reward signal: run the agent on training tasks, collect trajectories, use an LLM judge (or programmatic success verifier where available) to label rollouts as successful or failed, and apply RL (e.g., PPO or DPO) to increase the probability of actions from successful trajectories and decrease the probability of actions from failed ones. This is made feasible by the paper's openness — MolmoWeb's weights are available, MolmoWebMix provides the training task distribution, and the evaluation harness provides standardized success evaluation. A strong result would demonstrate that RL fine-tuning closes a meaningful fraction of the pass@1-to-pass@k gap (e.g., improving pass@1 on WebVoyager from 78.2% to 85%+ on the same task distribution), and would analyze which specific failure modes (loop-stuck, misreading, premature exit) are ameliorated by RL.

Can a difficulty estimator be trained directly from instruction text to enable practical compute-adaptive deployment? The paper's test-time scaling results show dramatic pass@k improvements, but best-of-N selection currently requires running all N rollouts to completion and then selecting with an LLM judge — an expensive and latency-unfriendly approach. A more practical system would estimate, after a small number of initial steps or a single quick rollout, whether the current task is likely to succeed or fail, and allocate additional parallel rollouts only for tasks where the estimated success probability is low. This requires a "difficulty estimator" or "early-exit confidence classifier" trained on rollout features (e.g., the PRM's score trajectory over the first 3-5 steps, action entropy, the model's own thought-language sentiment). Because MolmoWeb produces explicit thoughts and actions, these features are extractable without additional infrastructure. A strong experiment would train a lightweight classifier on features from the first 5 steps of WebVoyager rollouts to predict final success, then use it to gate parallel rollout allocation: if the classifier predicts success with high confidence, stop; otherwise, launch additional parallel rollouts. The metric would be the accuracy-vs-compute Pareto frontier compared to uniform best-of-N — the adaptive strategy should achieve comparable pass@k with fewer total rollouts by avoiding unnecessary parallelism on easy tasks.

Quantify the visual-only generalization advantage: does MolmoWeb transfer to websites with zero training representation? The paper argues that vision-only operation avoids the brittleness of DOM-based agents, which depend on website-specific element structures. But this claim is not empirically tested — MolmoWeb is evaluated on benchmarks whose websites are represented in training (directly or through benchmark-like task generation) and whose tasks were used as in-context examples during data generation. A rigorous test of the vision-only advantage would evaluate MolmoWeb on a held-out set of websites from categories not represented in MolmoWebMix — for example, government services websites, non-English e-commerce sites, or niche professional tools — and compare its performance drop against that of an AxTree-based agent (or a fine-tuned DOM-based agent) on the same held-out sites. The hypothesis is that MolmoWeb's relative performance degradation should be smaller because visual patterns (buttons look like buttons, search bars look like search bars) transfer across websites more readily than DOM structures. This experiment requires careful control of task difficulty — the held-out tasks should be of comparable complexity to benchmark tasks — and would produce a "generalization gap" metric that directly quantifies the paper's central architectural motivation.

Systematic failure mode taxonomy: where does MolmoWeb's 21.8% WebVoyager failure come from? The paper reports aggregate accuracy but provides only anecdotal failure characterization (Section 6: "may sometimes get stuck in states where it incorrectly keeps predicting the same action," "thoughts sometimes do not correlate well with actions"). A systematic failure analysis on the WebVoyager test set would categorize every failed trajectory into error types — e.g., grounding error (clicked wrong element), planning error (chose correct element but wrong strategic action), OCR error (misread text), action-execution error (correct action but wrong coordinates), premature termination (exited with wrong answer or stopped too early), loop-stuck (repeated same action), timing error (acted before page loaded). The distribution of failure modes would reveal where investment is most needed: if grounding errors dominate, improve the grounding data or add a verification step; if planning errors dominate, improve multi-step reasoning training; if OCR errors dominate, increase screenshot QA data or improve vision encoder resolution. Because MolmoWeb is fully open (weights, data, evaluation harness are all released), this analysis can be conducted by any research group without access to proprietary infrastructure — the paper makes failure-mode science possible for the first time on a competitive web agent.

Cross-model ablation: does MolmoWebMix improve other VLMs comparably? MolmoWeb's performance is the product of two factors: MolmoWebMix (the training data) and Molmo2 (the base VLM). To understand whether MolmoWebMix is a broadly useful resource or specific to the Molmo2 architecture, a valuable experiment would fine-tune other open VLMs (e.g., Qwen2.5-VL-7B, LLaVA-1.6-7B, InternVL2-8B) on MolmoWebMix using the same training recipe and evaluate on the same benchmarks. If other base models achieve comparable performance to MolmoWeb-8B, MolmoWebMix is validated as a general-purpose web agent training resource (analogous to how ImageNet enabled computer vision progress across architectures). If performance varies substantially, the interaction between base model pretraining and web-agent fine-tuning becomes an important research question — what pretraining properties (resolution, OCR quality, instruction-following ability) predict web agent fine-tuning success? This experiment is feasible because the paper releases the full training data and code, not just model weights.

Practical Applications and Downstream Use Cases

Open-source web agent for academic research on human-agent interaction. Before MolmoWeb, a researcher studying how humans collaborate with web agents had few options: use a proprietary API (GPT-4o, Gemini) with opaque behavior and rate limits, or use an open-weight model with substantially lower capability. MolmoWeb-8B's 78.2% on WebVoyager — outperforming GPT-4o SoM agents (65.1%) — means that researchers can now deploy a locally-inspectable, fine-tunable, and free web agent for human-subject experiments. This enables studies that were previously infeasible: systematically varying the agent's instruction-following behavior via fine-tuning on different data subsets, logging complete internal reasoning traces (thoughts) for analysis, or deploying the agent in privacy-sensitive contexts (medical information lookup, financial planning) where sending screenshots to an external API is unacceptable. The release of the evaluation harness as a unified infrastructure further means that experimental conditions can be standardized and replicated across research groups.

Data generation for self-improving web agents. The pass@1-to-pass@k gap (35.3% → 60.5% on Online-Mind2Web with MolmoWeb-8B) means that running parallel rollouts and selecting the best outcome produces a large volume of successful trajectories (60.5% of tasks succeed in at least one of 4 rollouts, yielding successful demonstrations for those tasks). These successful trajectories can be fed back into training — either as additional supervised fine-tuning data (behavior cloning on successful rollouts) or as positive examples for RL fine-tuning. The key enabler is that MolmoWeb is open-weight: the training pipeline is fully controllable, and the trajectory generation is not gated by API costs or rate limits. An organization could run MolmoWeb-8B at scale on a corpus of web tasks, collect best-of-4 successful trajectories, and fine-tune the model on those trajectories, potentially closing the pass@1-to-pass@k gap without any human annotation. The paper's finding that parallel rollouts with a 30-step budget substantially outperform single 100-step rollouts suggests this data generation approach would be compute-efficient: many short, diverse trajectories provide better training signal than few long ones.

Low-cost deployment for batch web automation tasks. For use cases involving batch processing of web tasks — price monitoring across e-commerce sites, data extraction from government databases, automated testing of web applications — MolmoWeb-4B provides a viable open-source alternative to proprietary APIs. At 75.2% on WebVoyager, it outperforms all open-weight models of comparable size, and its 4B parameter count means it can run on consumer-grade GPUs or even CPUs with quantization (the paper doesn't report latency, but 4B models are typically deployable with reasonable throughput on modest hardware). For a price monitoring service checking 10,000 product pages daily, using MolmoWeb-4B instead of GPT-4o API calls could reduce costs from hundreds of dollars per day (API pricing) to the electricity cost of running a local GPU, while maintaining competitive task completion rates on e-commerce benchmarks (35.6% on DeepShop, exceeding all open-weight baselines). The best-of-N test-time scaling strategy is directly applicable here: run 2–3 parallel rollouts per task for critical extractions and select the best using a lightweight verifier or majority-voting on extracted values.

Transparent web agents for regulated industries. In domains with regulatory requirements for algorithmic transparency — financial services, healthcare navigation, government benefits enrollment — the opacity of proprietary web agents is a legal barrier to deployment. A bank cannot use GPT-4o to autonomously fill loan applications if it cannot explain why the agent selected certain options or if the agent's training data is unknown (potentially containing biased examples). MolmoWeb's full openness — released training data, inspectable model weights, explicit thought traces at every step — makes it possible to audit the agent's behavior, trace decision pathways, and verify compliance with fairness and accuracy requirements. The thought traces provide an auditable decision record: for every form field filled, every link clicked, every product selected, the model's natural language rationale is preserved. This doesn't guarantee the agent is correct or unbiased, but it makes systematic auditing possible — a regulator or compliance officer can examine the agent's reasoning on representative tasks and identify systematic failures or biases. This is a qualitative advantage over proprietary systems where failures can only be detected through black-box testing, not structural inspection.