WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent

ArXiv: 2508.05748

🎯 Pitch

WebWatcher shows that multimodal deep research agents can more than double the performance of GPT-4o with RAG on visual reasoning benchmarks—but only if they’re trained first on action-grounded trajectories, without which reinforcement learning completely fails. The key is not just adding vision, but forcing the agent to coordinate search, browsing, code execution, and OCR in a single loop.

1. Executive Summary

This paper introduces WebWatcher, a multimodal agent for deep research that integrates visual-language reasoning with multi-tool interaction—including Web Image Search, Web Text Search, Webpage Visit, Code Interpreter, and internal OCR—to tackle complex information-seeking tasks requiring both perception and cross-modal inference. The authors train WebWatcher via supervised fine-tuning on high-quality synthetic ReAct trajectories followed by Group-Relative Policy Optimization (GRPO) reinforcement learning, and they introduce BrowseComp-VL, a challenging benchmark with BrowseComp-style, entity-obfuscated queries demanding multi-step visual-textual reasoning. WebWatcher-32B achieves state-of-the-art results across four challenging VQA benchmarks, including a 27.0% average on BrowseComp-VL (more than doubling the 13.4% of GPT-4o with RAG), 58.7% on LiveVQA, and 13.6% on Humanity's Last Exam, surpassing both proprietary reasoning models such as Gemini-2.5-Pro and o4-mini as well as the GPT-4o-based OmniSearch agent, with test-time scaling via pass@k yielding 41.9% on HLE at k=32. The paper establishes that effective multimodal deep research agents require both robust cross-modal reasoning and flexible multi-tool coordination, with cold-start SFT on action-grounded trajectories proving indispensable before reinforcement learning can yield meaningful gains.

2. Context and Motivation

The Core Gap: Deep Research Agents Are Blind to Visual Information

The paper addresses a fundamental asymmetry in the current landscape of AI research agents. Over the past two years, "deep research" systems—agents that autonomously search the web, read documents, plan multi-step tasks, and synthesize evidence into coherent answers—have demonstrated superhuman performance on information-seeking benchmarks like BrowseComp (Wei et al., 2025a) and Humanity's Last Exam (Phan et al., 2025). Proprietary systems such as OpenAI's DeepResearch (OpenAI, 2025a) and Google's Gemini Deep Research (Google, 2024) are now capable of near-expert performance across fact-finding, argumentative writing, and exploratory analysis. Open-source counterparts—WebDancer (Wu et al., 2025a), WebThinker (Li et al., 2025b), WebSailor (Li et al., 2025a), R1-Searcher (Song et al., 2025), and WebShaper (Tao et al., 2025)—have rapidly closed this gap through techniques like curriculum-driven SFT over ReAct traces, policy-gradient refinement, and formalization-driven data synthesis.

But the paper's opening observation is stark: all of these advances remain text-bound. Real-world information is not purely textual. Scientific diagrams, statistical charts, photographs in news articles, product images on e-commerce sites, architectural blueprints, and visually rich web interfaces all carry information that either cannot be expressed in text or loses critical nuance when flattened into captions. A deep research agent that cannot interpret a compound's spectral data embedded in a chemistry paper's figure, or a financial chart showing quarterly revenue trends, or a historical photograph whose details contradict the textual narrative surrounding it, is fundamentally limited in its ability to perform genuine research. As the authors put it in Section 1:

"most advances to date remain primarily text-centric, neglecting the rich visual information omnipresent in real-world scenarios."

This is not a niche concern. Many research-centric and everyday tasks—interpreting scientific diagrams (Hu et al., 2024), analyzing graphics (Wang et al., 2024b), navigating visually rich web interfaces (Hong et al., 2024)—demand integrated vision-language reasoning. The paper cites Dong et al. (2025) to emphasize that this integration is not a minor augmentation but a necessary capability for agents that aim to serve as genuine research assistants. Proprietary agents like GPT-4o and Gemini have made strides in multimodal reasoning, but the ecosystem of open, reproducible, and systematically evaluated multimodal deep research agents remains underdeveloped. This is the specific gap WebWatcher aims to fill.

Why the Gap Matters: Two Failure Modes of Current Approaches

The paper motivates the importance of this gap through a concrete example in Figure 2, drawn from the GAIA benchmark (Mialon et al., 2023). The question—"On the Wikipedia page for the animal in the provided image, how many revisions from before 2020 had 'visual edit' tags?"—requires a deceptively complex chain of reasoning. The agent must: (1) perceive and identify the animal from the image (an Atlantic puffin), (2) locate the correct Wikipedia page, (3) navigate to the page's revision history, (4) filter revisions by tag type ("visual edit") and date (before 2020), and (5) count the qualifying revisions.

The paper uses this example to illustrate two structural failure modes that motivate the entire WebWatcher design:

Failure Mode 1: Vision-only agents get trapped in perception. The "VL Agent" depicted in Figure 2 takes a visually dominated approach—edge detection, texture analysis, image search for visually similar entities. It identifies the bird as a "pelican" (misidentification) and returns an answer of 0. The paper's critique is not simply that the visual recognition was wrong; it's that the pipeline's architecture—excessive reliance on visual feature extraction without integrating textual reasoning—amplifies downstream errors. Perceptual mistakes at the identification stage propagate through every subsequent step without any mechanism for cross-modal verification. The answer 0 is confidently wrong because the agent never used text-based search to cross-check whether puffins or pelicans have "visual edit" tags on Wikipedia, nor did it visit the actual pages to verify.

This failure mode generalizes: agents that rely primarily on visual tools (OCR, bounding box extraction, image cropping, visual annotation) can handle straightforward perception tasks but struggle when perceptual outputs must feed into structured reasoning chains that cross modal boundaries. The paper explicitly argues in Section 1:

"While visual tools assist agents in handling perceptual tasks, they struggle to integrate visual reasoning with deep textual understanding and cross-modal inference, falling short in tackling high-difficulty tasks that require complex reasoning."

Failure Mode 2: Search-only agents cannot synthesize across modalities. The "Search Agent" in Figure 2 takes a text-dominant approach. It hypothesizes the animal might be a penguin or seagull based on vague textual reasoning, issues text-based search queries for "penguin" and "seagull," and arrives at answers of 2 or 8. The problem here is twofold. First, the agent fundamentally misidentifies the entity because it tries to reason about visual content through text alone—a form of modality blindness. Second, and more subtly, it lacks the tool repertoire to actually solve the problem: search returns snippets and URLs, but the agent never visits the Wikipedia page, navigates to the revision history, or executes a structured count. The agent's tool usage is "poor" and relies on "shallow problem decomposition"—it treats the task as a series of independent facts to retrieve, rather than a structured procedure requiring multiple coordinated tools (search, page visit, filtering, counting).

The contrast—"WebWatcher VL DeepResearch" on the right side of Figure 2—shows what multimodal deep research actually requires: a sequence of tool calls (ImageSearch, ImageVis, PageView, EvidenceSum, OCR, Wikipedia, VisualDiff, WebSearch, CrossValidate, Count) that interleave visual and textual reasoning, using the right tool at each decision point, with cross-validation steps that catch and correct potential errors.

This example is not anecdotal padding. It crystallizes the paper's thesis: effective multimodal deep research agents require both strong reasoning abilities across textual and visual information AND effective use of multiple external tools. Neither axis alone is sufficient.

Prior Approaches and Their Specific Shortcomings

The paper identifies three categories of prior work, each with limitations that WebWatcher is designed to overcome.

Category 1: VL Agents reliant on visual tools. The paper cites Zhao et al. (2025) and Su et al. (2025a;b) as representatives of VL agents that primarily leverage visual perception tools—OCR for text extraction, bounding box extraction for object localization, image cropping for region-of-interest isolation, and visual annotation for descriptive overlay. These systems are effective at perception-oriented tasks: given an image of a document, they can extract the text; given a complex diagram, they can identify and label components. But the paper argues that their reasoning capabilities are fundamentally bounded by the perceptual tools themselves. When a task requires integrating visual perception with deep textual understanding—for example, extracting the population of a city from a chart, noticing that the chart's data contradicts the accompanying article's claim, and reconciling the discrepancy through targeted web search—perception-only pipelines fall short. They lack the planning and tool-switching flexibility to move beyond what can be directly seen.

Category 2: Retrieval-augmented search agents. On the other side, the paper acknowledges that retrieval-augmented generation (RAG) approaches have become dominant for knowledge-intensive tasks, where agents retrieve relevant documents and synthesize answers. But the paper identifies specific failure modes. Citing Shen et al. (2024) and Gu et al. (2025), it argues that retrieval-augmented reasoning fails "when answers are implicit, require structured interactions, or demand additional computation." The key phrase is structured interactions. Retrieval assumes that the answer (or information sufficient to derive it) exists in the retrieved documents and can be surfaced through semantic similarity. But many real-world tasks require procedural engagement with the information environment: executing code to analyze data, clicking through links on a dynamic webpage, filling out forms, or performing step-by-step calculations that transform retrieved information into something that was never directly stated. The paper gives the example that "some problems require executing code to interpret charts, performing step-by-step calculations, or browsing dynamic webpages to extract up-to-date or structured content."

The distinction between retrieval and genuine research is central to the paper's intellectual contribution. Retrieval can tell you what a Wikipedia page says; research requires visiting the page, navigating its history, filtering, and counting—operations that retrieval pipelines do not support.

Category 3: Open-source deep research agents (text-only). The paper positions itself relative to a rapidly growing body of open-source work on text-based deep research agents: WebDancer (Wu et al., 2025a) introduces curriculum-driven SFT over ReAct traces; WebThinker (Li et al., 2025b) augments SFT with policy-gradient refinement; R1-Searcher (Song et al., 2025) leverages self-play for tree-structured exploration; WebSailor (Li et al., 2025a) uses structured task obfuscation and the DUPO algorithm; and WebShaper (Tao et al., 2025) proposes formalization-driven data synthesis. These systems have demonstrated remarkable performance on text-only benchmarks, but the paper's critique is direct: "nearly all leading deep-research agents are still text-bound."

This observation is not merely taxonomic—it points to a genuine technical gap. The architectures, training pipelines, and evaluation frameworks of text-based deep research agents were not designed to handle visual inputs, visual reasoning, or the coordination of visual and textual tools. Extending them to the multimodal setting is not a minor engineering tweak; it requires rethinking data construction, trajectory generation, tool design, and evaluation. The paper frames this as a necessary next leap:

"Integrating vision, layout, and cross-modal grounding is therefore not a minor tweak but the necessary next leap, multimodality will fundamentally redefine what deep research can achieve."

How WebWatcher Positions Itself

Against this backdrop, the paper positions WebWatcher not as an incremental improvement to existing text agents but as a unified framework purpose-built for the multimodal deep research challenge. The positioning operates along several axes:

Unified vision-language reasoning + multi-tool interaction. Rather than treating vision and search as separate capabilities bolted onto a text agent, WebWatcher is designed from the ground up to interleave visual perception, textual reasoning, and tool execution within a single reasoning loop. The ReAct-style trajectories (Section 3.1.2) enforce this integration at the training level: each think-act-observe cycle can seamlessly invoke vision tools (Web Image Search, OCR) followed by text tools (Web Text Search, Visit) followed by computation (Code Interpreter), with the agent's thought process explicitly reasoning across modalities.

Scalable data pipeline, not manual curation. The paper introduces two complementary data generation pipelines that are designed for scale and automation. Level 1 questions, inspired by CRAWL-QA from WebDancer, harvest real-world knowledge through recursive hyperlink traversal over authoritative sources (arXiv, GitHub, Wikipedia), using GPT-4o to synthesize QA pairs from aggregated content. Level 2 questions, following WebSailor's obfuscation approach, deliberately mask entities and attributes to force synthetic reasoning rather than direct retrieval. The QA-to-VQA conversion pipeline (Section 2.2.2) then transforms these text-only QA pairs into multimodal items by grounding them in authentic web-retrieved images, using entity masking and question rewriting to ensure that the image carries genuine information necessary for answering.

What distinguishes this pipeline from prior work—where prompting LLMs to generate VQA questions directly from images "often yields shallow, single-hop queries"—is that it preserves the multi-hop reasoning complexity of text-based QA while adding genuine visual grounding. The three-stage quality control (selector, examiner, minimum tool usage) ensures that the resulting data demands substantive, process-driven reasoning rather than single-step perception.

Action-grounded trajectories, not template-based reasoning. The paper's trajectory generation pipeline (Section 3.1) is designed to address a specific failure mode of prior reasoning agents: "recent reasoning agents generate traces that often tend to be long and templated, with limited diversity or adaptability across tasks" (citing Rose et al., 2023; Bi et al., 2025). Rather than hand-crafting chain-of-thought style traces or using template-based rationales, WebWatcher's trajectories are "grounded in actual tool-use behavior and reflect procedural decision-making aligned with complex reasoning demands." The automated annotation pipeline uses GPT-4o to simulate how a human would explore and reason through a problem by trying different tools step by step, with each action-observation pair verified through step-by-step consistency checks and final answer matching. This produces diverse, realistic trajectories that teach the model how to deploy tools adaptively rather than following a fixed recipe.

Cold-start SFT + RL, not RL from scratch. The paper's empirical analysis (Section 4.3, Figure 6) makes a case that is both methodological and principled: reinforcement learning from an instruction-tuned initialization fails catastrophically for multimodal agentic tasks, with the model "staying near zero for many steps" due to format errors wiping out rewards. Only after SFT on tool-use trajectories does RL produce meaningful gains. This finding has implications beyond WebWatcher itself—it suggests that for complex, tool-augmented reasoning tasks, supervised exposure to tool-use patterns is a necessary precondition, not just a helpful warm start.

New benchmark to match the capability gap. Finally, the paper introduces BrowseComp-VL, a benchmark specifically designed to evaluate the capabilities that existing benchmarks miss. Existing VQA benchmarks—OK-VQA, A-OKVQA, MMT-Bench, MicroVQA, Open3DVQA, Dyn-VQA, MMMU, MMMU-Pro—are characterized by the paper as emphasizing "single-step perception or shallow retrieval, with limited support for integrated multimodal reasoning and planning." BrowseComp-VL, by contrast, requires "complex information retrieval involving both visual and textual information," with long, entity-obfuscated queries demanding cross-modal inference, thorough information-seeking, and high-level planning. The two difficulty levels—Level 1 with explicit entities requiring multi-hop but retrievable reasoning, Level 2 with intentionally fuzzified entities and attributes—create a gradient that tests both retrieval efficiency and genuine synthesis capability.

Summary of the Intellectual Arc

The paper's motivation traces a clear arc: (1) Deep research agents have become remarkably capable, but only for text. (2) Real-world research tasks inherently demand visual understanding, cross-modal reasoning, and tool coordination. (3) Existing approaches fall into two camps that are individually insufficient: vision-only agents lack reasoning depth, and search-only agents lack perceptual grounding and procedural tool use. (4) No existing multimodal VQA benchmark evaluates the combination of multi-step planning, cross-modal synthesis, and flexible tool use. (5) Therefore, a unified framework—WebWatcher—with integrated vision-language reasoning, coordinated multi-tool interaction, scalable data generation, and action-grounded training is necessary to advance the field. The BrowseComp-VL benchmark provides the evaluation substrate to measure progress along these dimensions.

3. Technical Approach

3.1 Reader Orientation

WebWatcher is a vision-language agent—a single fine-tuned model (based on Qwen2.5-VL) that, given an image and a question, autonomously plans, executes multi-step tool calls (web search, image search, page visit, code execution, OCR), reasons across retrieved visual and textual evidence, and produces a final answer. The problem it solves is that real-world information-seeking tasks require coordinating visual perception with deep textual reasoning and procedural tool use—capabilities that current text-only search agents and perception-only visual agents each lack in isolation—and the solution shape is a unified model trained first via supervised fine-tuning on high-quality, action-grounded ReAct trajectories to establish basic tool-use competence, then optimized via reinforcement learning (GRPO) to improve decision-making and exploration, with a scalable synthetic data pipeline that converts multi-hop textual QA into genuinely visual VQA problems.

3.2 Big-Picture Architecture (Diagram in Words)

The WebWatcher system has six major components, each with a distinct responsibility:

1. Data Generation Pipeline (Section 2): Produces the training and evaluation data. It takes real-world web content (Wikipedia, arXiv, GitHub) as input, generates complex multi-hop QA pairs via recursive hyperlink traversal and entity obfuscation, converts these into multimodal VQA pairs by grounding them in authentic web-retrieved images, and applies multi-stage filtering to ensure quality. Output: the BrowseComp-VL dataset and additional training data.

2. Tool Suite (Section 3.1.1): A predefined set of five external tools—Web Image Search, Web Text Search, Visit (webpage summarization), Code Interpreter, and internal OCR—each with a structured input-output interface specified in the model's prompt. The agent can invoke any tool at any reasoning step.

3. Automated Trajectory Annotation Pipeline (Section 3.1.2-3.1.3): Uses GPT-4o to simulate human-like exploration for each VQA instance, producing ReAct-style trajectories (think-act-observe cycles) that ground correct reasoning in actual tool-use behavior. A three-stage filter (answer matching, step-by-step consistency, minimum tool usage) ensures trajectory quality.

4. Supervised Fine-Tuning Stage (Section 3.2): The cold-start phase. The Qwen2.5-VL base model is fine-tuned on the filtered trajectories to predict correct tool-use actions given the image, question, and interaction history. This teaches the model the basic mechanics of tool invocation and multi-step reasoning patterns.

5. Reinforcement Learning Stage (Section 3.3): GRPO further optimizes the SFT model. The model generates groups of trajectories, each receives a composite reward (format correctness + semantic accuracy), and the policy is updated to favor trajectories with higher relative advantage within each group.

6. Inference Engine: At deployment, the trained model receives an image and question, then autonomously loops through think-act-observe cycles—invoking tools, processing their outputs, and reasoning across modalities—until it decides to emit a final answer. Test-time scaling via pass@k (multiple independent rollouts) further boosts accuracy.

Information flows through these components sequentially during training—data generation → trajectory annotation → SFT → RL—and cyclically during inference (agent loop).

3.3 Roadmap for the Deep Dive

• First, the BrowseComp-VL data generation pipeline (QA construction and QA-to-VQA conversion), because it creates the training curriculum and evaluation benchmark that all subsequent training depends on, and its design choices (entity obfuscation, visual grounding with authentic images, multi-stage filtering) directly shape the agent's capabilities.

• Second, the tool definitions and interface, because all trajectories and training stages are built around these tools—understanding their input-output contracts is prerequisite to understanding trajectory construction.

• Third, the automated trajectory annotation and filtering pipeline, which bridges the semantic gap between raw VQA pairs and the action-grounded sequences that teach the model how to reason with tools.

• Fourth, the supervised fine-tuning objective as a cold-start, including why RL from an instruction-tuned initialization fails and what cold-start provides that RL cannot bootstrap on its own.

• Fifth, the GRPO reinforcement learning stage, covering the reward design, the group-relative advantage mechanism, the KL regularization, and the training dynamics that differentiate HLE, BrowseComp-VL, and LiveVQA scaling behavior.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that effective multimodal deep research requires (1) a scalable pipeline for generating training data that demands cross-modal, multi-hop reasoning, (2) an action-grounded trajectory construction process that teaches diverse, adaptive tool use rather than templated reasoning, and (3) a two-stage training regime (SFT cold-start followed by GRPO) that is necessary because RL alone cannot bootstrap tool-use competence from an instruction-tuned initialization.

BrowseComp-VL Data Generation Pipeline

The BrowseComp-VL dataset is not a manually curated collection of questions. It is the output of a multi-stage synthetic generation pipeline designed to produce VQA instances that are simultaneously (a) knowledge-intensive and multi-hop in their reasoning demands, (b) genuinely reliant on visual information for successful answering, and (c) resistant to shortcut retrieval. The pipeline has two major phases: (1) constructing complex textual QA pairs and (2) converting them into multimodal VQA items.

Phase 1: QA Pair Construction

The paper defines two difficulty levels with different construction methods, summarized in Figure 4.

Level 1: Multi-hop QA from Web Traversal

Level 1 questions are built from authoritative, knowledge-rich web sources. The procedure is:

• Root URL collection: The authors seed the pipeline with URLs from arXiv, GitHub, and Wikipedia—domains chosen for their high information density and reliable cross-linking.
• Recursive hyperlink traversal: Starting from each root URL, the system recursively follows accessible hyperlinks, simulating human browsing behavior. The traversal builds a directed graph of interconnected pages, each containing factual content, structured data, and embedded references.
• QA synthesis via GPT-4o: From the aggregated content across multiple linked pages, GPT-4o is prompted to synthesize question-answer pairs. The key design constraint is that answers must require integrating information from at least two distinct pages in the traversal graph—enforcing multi-hop reasoning. The synthesized QA pairs have concrete, explicitly named entities and retrievable answers, but the retrieval path is non-trivial because the information is distributed across sources.

This Level 1 procedure yields 110,000 raw QA pairs. The rationale for using recursive web traversal rather than prompting LLMs to hallucinate questions is explicitly stated: direct LLM-to-VQA generation "often yields shallow, single-hop queries that lack ambiguity, structured planning, and deeper reasoning." By grounding the generation in real web content with actual traversal structure, the resulting questions inherit the complexity of the underlying information topology.

Level 2: Entity-Obfuscated QA for Synthesis

Level 2 questions are designed to be substantially harder by removing the ability to answer through direct retrieval. The construction follows WebSailor's obfuscation approach and operates in two sub-stages:

Stage 1: Nodes Selecting (Building a Reasoning Graph). Given an initial Wikipedia page whose title serves as the root entity node $B_{root}$, GPT-4o first generates a base QA pair grounded in that page's content. Then, starting from $B_{root}$, the system recursively expands a hyperlink graph by traversing outgoing links to build a tree. The tree has depth $d = 3$ and branching factor $k = 3$, yielding a total of:

$\frac{k^{d+1} - 1}{k - 1} = \frac{3^4 - 1}{3 - 1} = \frac{81 - 1}{2} = 40 \text{ nodes}$

This tree represents all accessible knowledge paths reachable from the root within three link-hops. To generate diverse reasoning paths, the system samples multiple subgraphs: it randomly selects a subset of nodes and continues expansion until each subgraph contains $N$ entities. Each subgraph defines a unique multi-hop reasoning path from $B_{root}$ to a newly selected target entity $B$, forming a knowledge graph encoding the relations between the entities along the path. These subgraphs become the "ground-truth reasoning paths" that the generated questions will implicitly test.

Stage 2: Query Generating (Fuzzing for Synthesis). For each subgraph and its associated ground-truth answer, GPT-4o is prompted in two steps. First, it generates a "standard form" question that explicitly references all entities and relations along the reasoning path—this is a well-formed, unambiguous question that a retrieval system could in principle answer. Second, it transforms this standard form into a "fuzzed version": key references are replaced with partial, ambiguous, or qualitative descriptions (concrete dates become vague periods, entity names are masked, quantitative properties are fuzzified). For example, "the 2017 FIFA World Cup final" might become "the championship match of a major international football tournament held in the late 2010s."

The purpose of fuzzing is stated clearly: "This design encourages diverse reasoning patterns, pushing models to infer answers via synthesis rather than surface matching." A fuzzed question cannot be answered by a single search query; the agent must first disambiguate the fuzzy references (which may require multiple search-and-verify cycles), then trace the reasoning path across multiple knowledge sources, and finally synthesize the answer.

This Level 2 procedure yields 70,000 raw QA pairs. The key distinction from Level 1 is not just difficulty—it is the type of reasoning demanded: Level 1 tests efficient multi-hop retrieval, while Level 2 tests the ability to resolve ambiguity through exploration and synthesis.

Phase 2: QA-to-VQA Conversion

Converting textual QA pairs into multimodal VQA instances is non-trivial because the image must carry genuine information—not just be decorative—and the question must be rewritten so that answering it requires reasoning about the image in conjunction with external knowledge.

Visual Context Construction

For each QA pair $(q_t, a)$ where $q_t$ contains a reference to a target entity $\hat{B}$, the system first filters out entities unsuitable for visual grounding. The filtering eliminates "trivial or excessively ambiguous target entities, such as those that denote temporal references or domain-external concepts, which lack sufficient visual grounding." For example, an entity referring to "the year 1848" or "the concept of justice" would be filtered because no single image captures these concepts in a way that supports unambiguous visual reasoning.

For each retained entity $\hat{B}$, the system retrieves $K = 2$ web images via Google SerpApi: $\mathcal{I}(\hat{B}) = \{I_1^{\hat{B}}, I_2^{\hat{B}}\}$. The paper emphasizes that these images are "strictly authentic, thus minimizing noise and maximizing relevance for real-world tasks"—they are not synthetic renderings, composited images, or artist's illustrations (unless such images authentically exist on the web for that entity). This authenticity constraint is important because it means the visual content reflects the distribution of images a deployed agent would actually encounter when searching the web.

Entity Masking and Question Transformation

Given a QA pair $(q_t, a)$ and retrieved images $\mathcal{I}(\hat{B})$, GPT-4o performs a prompt-based rewriting:

1. Mask the entity mention: The clear mention of $\hat{B}$ in $q_t$ is replaced with a visual reference token $r_{\text{vis}}$. This token can take forms such as demonstratives ("this entity," "the animal in the image") or descriptive phrases ("the object shown in the provided photograph"). The resulting transformed VQA query is $q$.

2. Generate an image query string: Simultaneously, GPT-4o produces an image query string $s_{\text{img}}(\hat{B})$ that describes what visual content to look for among the retrieved images. This string guides the filtering step that matches images to questions.

Each image $I_k^{\hat{B}} \in \mathcal{I}(\hat{B})$ is then paired with $(q, a)$ to form a distinct VQA instance. Since $K = 2$, each textual QA pair produces exactly 2 multimodal examples. Consequently, $n$ original textual QA pairs yield $2n$ VQA items.

The critical design choice here is that the image is not the answer. The question asks about properties, relationships, or facts related to the entity in the image—not "what is this?" but rather "based on what you see, what can you determine about X?" This forces the agent to use the image for identification, then pursue additional information through tools. The transformation ensures that visual perception is a necessary intermediate step, not the terminal one.

Quality Control: Three-Stage Filtering

The raw VQA pairs from Phase 2 are filtered through a selector-examiner pipeline designed to eliminate several categories of bad data:

Stage 1: Selector Filtering. GPT-4o is prompted to evaluate each pair $(s_{\text{img}}(\hat{B}), \mathcal{I}(\hat{B}))$ along three dimensions:

• Trivial identity check: Cases where the transformed VQA query $q$ is identical to the original question $q_t$ (indicating masking failure) are discarded.

• Entity leakage check: Cases where the entity name $\hat{B}$ or any of its known aliases appear explicitly in $q_t$ (indicating failed masking that would let the agent bypass visual reasoning) are discarded.

• Relevance scoring: GPT-4o assesses each image $I_k^{\hat{B}} \in \mathcal{I}(\hat{B})$ with respect to both the original QA pair $(q_t, a)$ and the transformed VQA query $(q, a)$, scoring contextual alignment, semantic fit, and the plausibility of visual reasoning. Low-relevance images are filtered out.

Stage 2: Examiner Filtering. For each retained image-query pair, GPT-4o is prompted to answer the synthesized query $s_{\text{img}}(\hat{B})$ using only the visual content and associated captions of $\mathcal{I}(\hat{B})$. If the model cannot answer accurately under this closed-book setting—meaning the image alone provides insufficient information for the reasoning task—the query is filtered out. To mitigate false negatives from genuinely missing world knowledge, the model is granted access to available image captions during validation. This ensures that the filtered set contains only instances where the image carries necessary information—it is a genuine visual reasoning task, not a text task with a decorative picture.

Stage 3: Minimum Tool Usage (applied later, during trajectory construction). Trajectories with fewer than three tool calls are removed to ensure that training data reflects "substantive, process-driven interactions rather than trivial or one-step completions."

After all filtering, the BrowseComp-VL training set contains 60,000 Level-1 and 40,000 Level-2 high-quality VQA examples, distilled from 110,000 and 70,000 raw QA pairs respectively. The evaluation set is drawn from these but undergoes additional manual verification by PhD-level AI experts: 100 Level-1 instances and 200 Level-2 instances, ensuring "high accuracy and consistency."

Tool Definitions and Interface

The WebWatcher agent operates within a discrete action space $\mathcal{T}$ containing five tools. Each tool is defined through a structured prompt that specifies both its callable format and its semantic capability, ensuring the language model can reason about when and how to invoke it within the <tool_call>...</tool_call> block during interaction. All tool definitions are included in the model's system prompt during both training and inference.

Tool 1: Web Image Search. This tool performs reverse image search via Google SerpApi. Its interface takes an image URL as input and returns the top 10 search results, each consisting of a caption and the webpage URL of the matching page. The intended use case is: when the agent encounters an image it needs to identify or contextualize, it searches for visually similar images on the web to find pages that describe or contain the depicted entity.

Input: image_urls (array of image URL strings—though the description notes it "should only be used once" with the input image, since the purpose is to ground the visual input in external web context).

Output: A ranked list of up to 10 results, each with a caption (text description of the matching image) and webpage_url (the page hosting the matching image).

The constraint that it "should only be used once" is not a technical limitation but a design choice embedded in the prompt to prevent the agent from repeatedly querying the same image and wasting compute budget on redundant results.

Tool 2: Web Text Search. This tool performs open-domain text search, also via Google SerpApi. It takes one or more search query strings and returns the top 10 text excerpts, each with a title and webpage URL. It is the primary tool for information seeking across the open web—finding Wikipedia pages, news articles, documentation, and any other text-based source.

Input: queries (array of strings)—the search terms to issue.

Output: A ranked list of up to 10 results, each with a title and webpage_url.

The ability to issue multiple queries in a single call is a deliberate design for efficiency: the agent can explore several search directions in parallel rather than sequentially, which matters when the reasoning budget is capped at 15 tool calls.

Tool 3: Visit. This tool navigates to a specific URL and returns a summary of the webpage's content, tailored to a goal specified by the agent. It is powered by Jina.ai's webpage summarization service. The critical design element is the goal parameter: rather than returning a generic summary of the entire page, the Visit tool extracts and synthesizes content relevant to the agent's specific information need. For example, visiting a Wikipedia page with the goal "revision history and visual edit tags" would return a summary focused on that aspect, not the full article.

Input: url (the target webpage URL) and goal (a natural language string describing what the agent seeks from the page).

Output: A summarized, goal-conditioned extraction of the webpage's content.

This goal-conditioned design addresses a practical bottleneck: full webpage content is often too long for the model's context window, and generic summarization may omit the specific detail the agent needs. By making the retrieval goal-directed, the tool returns information that is both concise and relevant to the current reasoning step.

Tool 4: Code Interpreter. This tool executes Python code for symbolic computation, numerical reasoning, or structured data manipulation. It is invoked when the agent needs to perform calculations (e.g., verifying a KenKen puzzle solution, computing time differences, analyzing tabular data), parse structured outputs, or programmatically extract information that text-based tools would struggle with.

Input: code (a string containing valid Python to execute).

Output: The result of code execution—standard output, return values, or error messages.

The tool enables computation that would be unreliable if done purely through natural language reasoning (arithmetic errors are a known failure mode of LLMs) and supports operations (sorting, filtering, counting across large result sets) that are impractical for text-only reasoning.

Tool 5: OCR (Internal). This is an optical character recognition tool invoked via prompt rather than a separate API. It extracts text content from a given image, useful for reading embedded visual text such as charts, screenshots, scanned documents, or images containing text that visual-language models might misread or hallucinate.

Input: image_url (the URL of the image to extract text from).

Output: Extracted text strings from the image, including layout and positional information where relevant.

The paper notes that the OCR tool is "internal"—it is part of the model's prompted capabilities rather than a separate service—and its training data includes specialized OCR-activating examples from Huang et al. (2025) to ensure the model knows when and how to invoke it.

Tool Invocation Format. All tools are invoked through the ReAct framework implemented via Qwen-Agent. Each action cycle produces a <tool_call> block containing a JSON object with "name" (the tool identifier) and "arguments" (a dictionary of parameter names to values). The environment returns results in a <tool_response> block. The model's reasoning between tool calls is captured in a thinking block (the thought component of the ReAct cycle). The complete trajectory format is:

 thinking [intermediate reasoning or plan]  response
<tool_call>
{"name": "tool name", "arguments": {"param1": value1, "param2": value2, ...}}
</tool_call>
<tool_response>
[tool output]
</tool_response>
[repeat until final answer]
 thinking [final reasoning]  response
<answer> [final answer] </answer>

The maximum number of tool calls per trajectory is capped at 15. This constraint prevents the agent from entering infinite loops and forces efficient reasoning within a bounded compute budget.

Automated Trajectory Annotation and Filtering

The trajectory annotation pipeline is the bridge between raw VQA pairs and the action-grounded supervision signal that teaches WebWatcher how to reason. The paper explicitly motivates this design against the backdrop of prior work: "recent reasoning agents generate traces that often tend to be long and templated, with limited diversity or adaptability across tasks" (Rose et al., 2023; Bi et al., 2025). The key design principle is that trajectories should be "grounded in actual tool-use behavior and reflect procedural decision-making aligned with complex reasoning demands," not hand-crafted CoT templates.

Generation Procedure

Given a VQA instance $(I, q, a)$ from the filtered BrowseComp-VL training set, GPT-4o is prompted to simulate the reasoning process of a human researcher tackling the problem. The model operates in an ReAct-style loop:

1. Context accumulation: At each step $t$, the model receives the accumulated history of all previous actions and observations: $H_t = \{(a_0, o_0), (a_1, o_1), ..., (a_{t-1}, o_{t-1})\}$ plus the original image $I$ and question $q$.
2. Thought generation: The model produces a thinking block containing its intermediate reasoning—what it currently knows, what remains unknown, what it plans to do next, and why. This is the explicit reasoning trace.
3. Action selection: The model emits a <tool_call> block selecting one tool from $\mathcal{T}$ with appropriate arguments.
4. Observation retrieval: The tool is actually executed (or its output is simulated by GPT-4o based on realistic web search results and webpage summaries), and the result is placed in a <tool_response> block.
5. Loop or finish: Steps 1–4 repeat until the model emits a Finish action—an <answer> block containing the final answer—or hits the 15-call limit.

Formally, a trajectory $\tau$ of length $L$ is denoted:

$\tau = \{(t_0, o_0), (t_1, o_1), \dots, (t_L, o_L)\}$

where each action $t_i \in \mathcal{T}$ and each observation $o_i$ reflects the environment feedback after tool execution.

What this formalism means operationally: a trajectory is a sequence of interleaved decisions and outcomes. At each step, the model chooses an action, the environment responds, and this pair informs subsequent decisions. The trajectory serves as a demonstration of plan execution—it shows not just what the final answer is, but the sequence of information-gathering and reasoning steps that led to it, including dead ends, corrections, and cross-validation.

This generation process is fundamentally different from both (1) prompting an LLM to write a chain-of-thought explanation post-hoc (which can be rationalization rather than genuine reasoning) and (2) collecting human demonstrations (which is expensive and limited in scale). By having GPT-4o simulate the process with actual tool calls, the trajectories encode realistic exploration patterns—trying a search query, examining results, refining the query, visiting pages, cross-checking—that teach the agent to navigate information environments adaptively.

Three-Stage Trajectory Filtering

Raw trajectories from GPT-4o can contain several failure modes: correct final answers reached through flawed or hallucinatory intermediate steps, logically inconsistent tool usage, trajectories that shortcut with a single tool call, or trajectories where the "reasoning" is actually a post-hoc fabrication that doesn't connect to the tool outputs. The paper's three-stage filter addresses each:

Stage 1: Final Answer Matching. This is a binary filter: keep trajectory $\tau$ only if the final extracted answer (from the <answer> block) matches the ground truth $a$. Trajectories that conclude with an incorrect answer are discarded regardless of intermediate quality, because they would teach the model to reach wrong conclusions.

This ensures "the entire sequence of tool-use steps leads to a correct and complete solution." It is a necessary but not sufficient condition—a trajectory can have a correct answer and still be useless as training data if the intermediate steps are nonsensical.

Stage 2: Step-by-Step Consistency Check. This is the more nuanced filter. GPT-4o is prompted (with the tool call rationality evaluation prompt shown in Appendix C) to verify each intermediate step $(t_l, o_l)$ for three criteria:

1. Information Non-Redundancy: "The requested information or action in the tool call is not already provided or easily derivable from prior dialogue, the user's current question, or the assistant's previous answers." This eliminates trajectories where the agent repeatedly searches for the same information or asks questions it should already know the answer to.

2. Goal Alignment: "The tool call's purpose and expected result directly serve the user's explicit intent or core need in this turn." This ensures that each tool invocation meaningfully advances the task, rather than being tangential or distracting.

3. Logical Reasoning and Accuracy: "The assistant's thought process shows clear, correct logic and reliable grounding—no unfounded guesses or fabrications." The thinking block is evaluated for consistency with available evidence and logical soundness.

Crucially, the evaluator checks each step, not just the aggregate. A trajectory passes only if all criteria are met at every step. Trajectories with "hallucinated content, contradictions, or unjustified tool calls are discarded." This prevents "the common failure mode where correct answers are reached by lucky guessing rather than meaningful tool use."

The specific evaluation prompt (Appendix C) shows that GPT-4o is instructed to output simply "A" if all criteria are met, "B" otherwise—a forced binary choice that avoids the ambiguity of scalar ratings.

Stage 3: Minimum Tool Usage Requirement. Trajectories with fewer than three tool calls are removed. This operationalizes the intuition that BrowseComp-VL and similarly challenging tasks require substantive multi-step exploration. A single tool call (e.g., one web search that happens to return the answer) does not represent the kind of process-driven reasoning the agent needs to learn. The threshold of three is a heuristic—it is low enough to not eliminate genuinely simple-but-still-structured problems, but high enough to filter trivial or one-shot completions.

Scale and Quality. After filtering, the SFT training set contains 8,000 high-quality tool-use trajectories. An additional 2,000 VQA samples are reserved for GRPO (the reinforcement learning stage). The combined data—60,000 Level-1 + 40,000 Level-2 raw VQA reduced to 8,000 trajectories—represents a roughly 92% reduction, reflecting the stringency of the three-stage filter. The paper does not report per-stage filter rates, but the implication is clear: most raw trajectories fail at least one quality criterion, and achieving high-quality supervision requires aggressive filtering.

Supervised Fine-Tuning as Cold Start

The SFT stage trains the Qwen2.5-VL model (7B or 32B variants) on the filtered trajectories. Formally, given a dataset of $K$ filtered trajectories, each trajectory $i$ having length $L_i$ steps, the objective is:

$\max_{\theta} \sum_{i=1}^K \sum_{l=1}^{L_i} \log P_{\theta} \left( t_l^{(i)} \mid I^{(i)}, q^{(i)}, t_{<l}^{(i)}, o_{<l}^{(i)} \right)$

where:

• $\theta$ denotes the model parameters;
• $I^{(i)}$ is the input image for the $i$-th trajectory;
• $q^{(i)}$ is the question;
• $t_l^{(i)}$ is the $l$-th tool-use action;
• $t_{<l}^{(i)}$ and $o_{<l}^{(i)}$ are all previous actions and observations in that trajectory.

What this equation computes operationally: For every step of every trajectory, the model is trained to predict the correct next action—the specific tool call (name and arguments) that the filtered trajectory used at that step—conditioned on the image, the question, and the entire interaction history up to that point. The log-likelihood is summed over all steps and all trajectories, and the optimizer maximizes this sum. At test time, this means the model can generate the most likely next tool call given what it has seen and done so far.

Why this form: Maximum likelihood estimation under a next-action prediction model is the standard formulation for behavioral cloning—learning a policy from demonstrations. The sequential conditioning $t_{<l}^{(i)}, o_{<l}^{(i)}$ is essential because tool-use decisions are path-dependent: whether to visit a webpage depends on what the search results showed; whether to run code depends on what the OCR extracted. Training on full trajectories rather than isolated (image, question) → final answer pairs teaches the model the process of reasoning, not just the mapping from inputs to outputs. Alternative formulations—such as training only on the final answer or on individual tool calls without history—would fail to capture the conditional structure of multi-step exploration.

Training Hyperparameters: SFT uses Llama-Factory with batch size 32, learning rate $5 \times 10^{-6}$ with a minimum of $1 \times 10^{-10}$, warmup plus cosine decay schedule, and weight decay of 0.1. These are relatively standard fine-tuning settings for vision-language models; the learning rate is moderate to avoid catastrophic forgetting of the base model's visual and linguistic capabilities.

Why Cold-Start is Indispensable: The paper makes a strong empirical claim in Section 4.3 and Figure 6: RL without SFT cold-start fails catastrophically. Under the "Instruct" initialization (the base Qwen2.5-VL model with only instruction-following training), the agent "stays near zero for many steps" because tool-call format errors wipe out the reward and the strict grader "further suppresses partial answers." The failure mode is specific: the model has never seen tool-call syntax in its training, so its early rollouts produce malformed <tool_call> blocks that score $r_f = 0$ on the format reward, and even when it eventually stumbles toward correct answers, the semantic reward $r_a$ is too low. RL's credit assignment mechanism cannot efficiently explore the space of tool-use patterns from scratch—the action space is too large and the reward signal too sparse.

SFT cold-start solves this by providing explicit demonstrations of correct tool-use syntax and multi-step reasoning patterns. After SFT, the model reliably produces valid tool calls and achieves initial scores of 0.12 on HLE, 0.30 on BrowseComp-VL, and 0.45 on LiveVQA. GRPO can then build on this foundation, because the exploration space is now bounded to meaningful variations of tool-use strategies rather than the vast space of all possible text completions.

The paper also reports a negative result that reinforces this finding: "Injecting CoT chains from a larger reasoner made the small model unstable, format violations, repetitions, and context overflow spiked." This means that simply providing textual reasoning traces (without actual tool execution and observation feedback) is insufficient—the model needs exposure to the structured action-observation loop format, not just abstract reasoning text. The SFT trajectories, by including concrete tool responses, teach the model to expect and process <tool_response> blocks, which the chain-of-thought traces lack.

GRPO Reinforcement Learning

After SFT cold-start, WebWatcher applies Group-Relative Policy Optimization (GRPO), a ranking-based variant of Proximal Policy Optimization (PPO). The goal is to further refine the agent's decision-making: SFT teaches the model to imitate trajectories; RL teaches it to optimize for outcomes, exploring strategies that may be better than those in the training data.

Trajectory Sampling and Group Construction

For each VQA query $q$ in the GRPO training set (2,000 reserved examples), the current policy $\pi_\theta$ generates a group $G = \{\tau_1, \dots, \tau_K\}$ of $K = 16$ complete trajectories. Each trajectory $\tau^{(i)}$ is a full sequence of tool calls and observations ending in a final answer. The 16 trajectories are generated with temperature 1.0 and $top\_p = 1.0$ (maximum diversity) to ensure the group contains a range of strategies, from optimal to suboptimal to failing.

Reward Design

Each trajectory $\tau$ receives a scalar total reward $R$ computed from two components:

$R = w r_f + (1 - w) r_a$

where:

• $r_f \in \{0, 1\}$ is a binary format score: 1 if all tool calls in the trajectory conform to the required schema (valid JSON, correct argument types, proper tag structure), 0 otherwise;
• $r_a \in [0, 1]$ is a semantic accuracy score from an LLM grader that compares the final answer with the ground truth;
• $w = 0.2$ is the weight balancing format correctness and semantic accuracy.

What this equation computes operationally: The total reward is a weighted combination of two signals. The format score $r_f$ provides a hard constraint—any trajectory with malformed tool calls is severely penalized (since $w = 0.2$ means format accounts for 20% of the total reward). The semantic accuracy $r_a$ is the primary optimization target (80% weight), evaluating whether the final answer matches the ground truth. The reward is assigned only once at the end of each trajectory—there is no per-step reward shaping.

Why this form: The hybrid reward design balances two competing concerns. On one hand, the model must produce syntactically valid tool calls for the system to function—a trajectory with format errors cannot actually execute its tools, so $r_f$ penalizes this failure mode. On the other hand, the ultimate goal is answer correctness, so $r_a$ dominates the weight. The $w = 0.2$ setting is chosen to strongly discourage format violations (a binary 0 on $r_f$ drops the total reward by 0.2) while still making semantic accuracy the primary driver of policy improvement.

The LLM grader for $r_a$ uses the prompt shown in Appendix C (Response Accuracy Evaluation), which extracts the final answer from the model's response, compares it to the ground truth with allowance for "small numerical margins," and produces a binary correct/incorrect judgment. This is the standard LLM-as-judge approach, which is more robust to surface-form variations than exact string matching.

Group-Relative Advantage

The key innovation of GRPO over standard PPO is the elimination of a separate value function (critic model). Instead, the advantage of each trajectory is computed relative to the mean reward within its group:

$A_{\text{rel}}(\tau^{(i)}) = R^{(i)} - \frac{1}{K} \sum_{j=1}^K R^{(j)}$

where $R^{(i)}$ is the total reward of trajectory $\tau^{(i)}$ and $K = 16$ is the group size.

What this equation computes: The group-relative advantage measures how much better (or worse) trajectory $i$ is than the average trajectory in the same group. If $\tau^{(i)}$ achieved a higher reward than the group mean, $A_{\text{rel}}(\tau^{(i)})$ is positive; if lower, it is negative. The magnitude reflects the degree of outperformance or underperformance.

Why this form: Standard PPO requires training a value function (critic) to estimate expected future returns, which doubles the model size and introduces additional optimization instability because the critic must be trained concurrently with the policy. GRPO's group-relative advantage eliminates the critic entirely: by sampling a batch of trajectories and normalizing within the batch, it provides a zero-mean baseline that serves the same variance-reduction purpose without needing to learn value estimates. The normalization $\frac{1}{K} \sum R^{(j)}$ acts as a Monte Carlo estimate of the expected return under the current policy. This is especially suitable for the WebWatcher setting because trajectories can have highly variable lengths and tool-use patterns, making value function learning particularly challenging.

The group size of $K = 16$ is chosen to provide "sufficient diversity for computing meaningful relative advantages while maintaining computational efficiency during training." Smaller groups would give noisier advantage estimates; larger groups would increase the computational cost of each update.

GRPO Objective

The policy is updated to maximize the clipped surrogate objective:

$\mathcal{L}_{\text{GRPO}}(\theta) = \mathbb{E}_{\tau^{(i)} \in G} \left[ \min \left( \rho^{(i)} A_{\text{rel}}(\tau^{(i)}), \text{clip} \left( \rho^{(i)}, 1 - \epsilon, 1 + \epsilon \right) A_{\text{rel}}(\tau^{(i)}) \right) \right] - \beta D_{\text{KL}}(\pi_\theta \| \pi_{\theta_{\text{old}}})$

where:

• $\rho^{(i)} = \frac{\pi_\theta(\tau^{(i)})}{\pi_{\theta_{\text{old}}}(\tau^{(i)})}$ is the importance sampling ratio—the probability of trajectory $\tau^{(i)}$ under the current policy divided by its probability under the old policy (the policy that generated the trajectories);
• $A_{\text{rel}}(\tau^{(i)})$ is the group-relative advantage;
• $\epsilon$ is the clipping threshold (standard PPO value, typically 0.2);
• $\beta$ controls the strength of the KL penalty; and
• $D_{\text{KL}}(\pi_\theta \| \pi_{\theta_{\text{old}}})$ is the Kullback-Leibler divergence between the current and old policies, acting as a regularizer.

What this equation computes operationally: The objective has two parts. The first part (the $\min$ and $\text{clip}$ terms) is the standard PPO clipped surrogate loss: it encourages the policy to increase the probability of trajectories with positive advantage and decrease the probability of trajectories with negative advantage, but the clipping prevents the update from being too large—if the probability ratio $\rho^{(i)}$ moves outside $[1 - \epsilon, 1 + \epsilon]$, the gradient is clipped (the $\min$ takes the smaller of the clipped and unclipped terms). This ensures stable, conservative policy updates. The second part ($-\beta D_{\text{KL}}$) penalizes the policy for deviating too far from the old policy, preventing catastrophic forgetting of the SFT-taught tool-use patterns.

Why this form: The core challenge in RL fine-tuning of language models is that over-optimizing the reward can cause the model to exploit reward artifacts (reward hacking) or drift into regions of poor language quality. PPO's clipping addresses the first issue—preventing large policy changes based on noisy advantage estimates—while the KL penalty addresses the second—maintaining proximity to a known-good policy (the SFT initialization via $\pi_{\theta_{\text{old}}}$). The combination is now standard in RLHF but is adapted here for tool-use trajectories rather than preference comparisons. The $\min$ operator is key: it takes the pessimistic bound, ensuring that the objective only increases when both the clipped and unclipped terms agree on the direction of improvement.

Training Configuration: GRPO uses Verl with rollout group size 8 (note: the paper states both $K = 16$ in Section 3.3 and group size 8 in Appendix D.3 for RL training—the former appears in the main text as the GRPO formulation, the latter in the implementation details), temperature 1.0, $top\_p = 1.0$, total batch size 128, mini-batch size 32, and learning rate $1 \times 10^{-6}$. The lower learning rate for RL (compared to SFT's $5 \times 10^{-6}$) reflects the need for more conservative updates when optimizing a reward signal rather than maximizing likelihood under a fixed dataset.

Training Dynamics Across Benchmarks

The paper's analysis in Figure 6 reveals that the effectiveness of GRPO depends on the benchmark:

• LiveVQA shows a "steady rise and keeps a 0.06–0.18 margin over the Instruct baseline throughout" after cold-start SFT, indicating that GRPO successfully discovers strategies that improve on imitation learning for this benchmark. LiveVQA's emphasis on up-to-date visual knowledge from news sources likely benefits from exploration—the SFT trajectories may not cover all effective search strategies for time-sensitive information.

• BrowseComp-VL and HLE "oscillate heavily with no clear upward drift" even with cold-start SFT. The paper does not fully explain this oscillation, but the implication is that these benchmarks are so challenging—requiring multi-page browsing, fine-grained visual grounding, and cross-modal synthesis—that even the SFT initialization provides only a weak starting point, and GRPO's exploration struggles to find reliably better strategies within the training budget. The reward signal may be too sparse or too noisy for stable optimization on these benchmarks.

This differential behavior is a significant finding: it suggests that RL's benefits are benchmark-dependent, with tasks that admit incremental improvement (better search queries, more efficient tool sequencing) benefiting more than tasks where success hinges on rare "insight" moments that are difficult to discover through random exploration. The paper's decision to train on a mixture of data sources (5:3:2 ratio of BrowseComp-VL, long-tail VQA, and hard VQA) may partially address this by providing a curriculum of varying difficulty.

4. Key Insights and Innovations

Innovation 1: Difficulty Obfuscation as a Mechanism to Force Genuine Synthesis, Not Retrieval

The dominant paradigm for constructing challenging VQA benchmarks has been to increase knowledge scope (broader domains, rarer facts) or perceptual difficulty (smaller objects, occluded views, ambiguous lighting). BrowseComp-VL's Level 2 construction introduces a fundamentally different axis: syntactic and semantic obfuscation of the question itself. By deliberately replacing precise entity names, dates, and quantitative properties with "partial, ambiguous, or qualitative descriptions," the benchmark tests not whether the agent knows something or sees something, but whether it can resolve ambiguity through multi-step exploration and cross-modal synthesis.

This is a conceptual shift from prior work. Datasets like OK-VQA and A-OKVQA demand external knowledge but give the agent a clear target to retrieve. MMMU and MMMU-Pro test domain expertise but state questions precisely. Even BrowseComp (Wei et al., 2025a), the text-only inspiration, uses unambiguous queries—its difficulty comes from the sheer breadth of required information and the need to navigate diverse web sources. WebWatcher's fuzzed Level 2 questions (Section 2.2.1) change the nature of the challenge: the agent must first discover what the question means before it can attempt to answer it. For example, a fuzzed question like "the championship match of a major international football tournament held in the late 2010s" requires the agent to hypothesize candidate events (2018 FIFA World Cup? 2019 Women's World Cup? 2016 Euros?), search for evidence about each, and disambiguate based on contextual clues from the accompanying image and retrieved information.

This is not merely a harder version of existing benchmarks—it is a different cognitive demand. Standard multi-hop QA requires tracing a known path through knowledge sources. Obfuscated QA requires constructing the path from ambiguous clues, which is closer to how real research problems present themselves: researchers rarely begin with precise, well-formed queries. The paper's decision to evaluate Level 1 (explicit entities, retrievable but multi-hop) alongside Level 2 (fuzzed entities, requiring synthesis) creates a gradient that disentangles retrieval efficiency from ambiguity resolution—two skills that are confounded in most existing benchmarks. The evidence that this distinction matters is in Table 2: even strong RAG baselines (GPT-4o with RAG: 16.8% on Level 1 vs. 7.0% on Level 2) show a sharper drop than WebWatcher-32B (28.4% vs. 25.0%), indicating that the agentic framework is more robust to question obfuscation than retrieval pipelines.

Innovation 2: The QA-to-VQA Conversion Pipeline as a General Mechanism for Scaling Multimodal Reasoning Data

The paper's data construction methodology solves a structural bootstrapping problem that has constrained multimodal reasoning research: how do you generate large-scale VQA data that demands multi-hop, cross-modal reasoning without manual curation? The standard approach—prompting LLMs to generate questions directly from images—"often yields shallow, single-hop queries that lack ambiguity, structured planning, and deeper reasoning" (Section 2.1). This is not a failure of prompting quality; it is inherent in the task. When an LLM sees an image and is asked to generate a question about it, the most natural questions are perceptual ("what color is...") or factual ("what species is...") because the image itself provides limited narrative context for constructing multi-hop reasoning chains.

The QA-to-VQA pipeline (Section 2.2.2) inverts this: start with complex textual reasoning chains (constructed via web traversal and entity obfuscation), then ground them in images post-hoc through entity masking and visual substitution. This decouples the reasoning complexity (which comes from the textual QA construction phase) from the visual grounding (which comes from the image retrieval and masking phase). The result is multimodal data that preserves the multi-hop, ambiguity-intensive structure of the textual QA while genuinely requiring visual perception—the image is not decorative because the masked entity must be identified from it to begin the reasoning chain.

This is more than a data augmentation trick. It is a general-purpose conversion framework that can theoretically transform any text-only QA dataset into a multimodal benchmark, provided the questions reference entities with visual instantiations. The paper explicitly notes this compatibility: the pipeline is "compatible with most existing QA datasets, enabling substantial scaling of multimodal datasets." If validated across other text QA benchmarks, this would shift the bottleneck for multimodal reasoning research from data scarcity (manually annotating complex VQA is expensive) to the quality of the underlying text QA and the reliability of the entity-image grounding. The three-stage quality control (selector for relevance, examiner for visual necessity, minimum tool usage for reasoning depth) provides a template for ensuring the converted data maintains both reasoning complexity and genuine multimodal dependence.

The practical significance is captured in the scale: 110,000 Level-1 and 70,000 Level-2 raw QA pairs distilled to 100,000 high-quality VQA examples after filtering. Manual curation at this scale—particularly for questions requiring multi-hop reasoning and entity obfuscation—would be prohibitively expensive. The pipeline makes large-scale multimodal reasoning data economically feasible.

Innovation 3: Cold-Start SFT on Action-Grounded Trajectories Is Necessary for RL in Tool-Augmented Settings—and Chain-of-Thought Is Not a Substitute

The paper's most consequential methodological finding is not that RL helps (that is expected from prior work like WebThinker and R1-Searcher), nor that SFT helps (standard practice), but rather the sharp necessity and insufficiency result in Figure 6: RL from an instruction-tuned initialization fails catastrophically for multimodal tool-use tasks, and injecting chain-of-thought reasoning traces from a larger model does not rescue it. The agent "stays near zero for many steps" under Instruct-only initialization because format errors in tool calls wipe out the reward signal before semantic learning can begin.

This finding challenges an implicit assumption in the RL-for-reasoning literature. Recent work—particularly DeepSeek-R1 (Guo et al., 2025) and its descendants—has demonstrated that pure RL on base models can elicit complex reasoning behaviors (chain-of-thought, self-verification, backtracking). The natural extrapolation would be that the same approach could elicit tool-use behaviors: give the model a reward signal for correct answers, and it will discover that using tools helps. WebWatcher's negative result shows this extrapolation fails when the action space includes structured outputs (tool calls with specific JSON formats, argument types, and tag structures) that must be syntactically valid to receive any reward. The exploration space is too sparse: the model must simultaneously learn (a) that tools exist, (b) what each tool does, (c) the correct invocation syntax, and (d) when each tool is appropriate—all from a binary reward that arrives only at the end of a multi-step trajectory.

The insufficiency of chain-of-thought as a substitute (reported in Section 4.3) deepens this insight: "Injecting CoT chains from a larger reasoner made the small model unstable, format violations, repetitions, and context overflow spiked." CoT traces teach reasoning patterns but not the structured action-observation loop that defines tool interaction. The model exposed to CoT learns to think about tool use but not to execute it, because the traces lack <tool_call> and <tool_response> blocks that carry the syntactic and interactional regularities of the environment.

This has implications beyond WebWatcher: any agent system that uses structured tool calls and hopes to optimize via RL likely needs an SFT stage that explicitly demonstrates the tool interaction format. The SFT trajectories in WebWatcher provide not just demonstrations of correct answers, but demonstrations of the mechanics of interaction—how to format a call, how to parse a response, how to chain calls based on observations. This is a different kind of knowledge than what CoT provides, and the paper's empirical evidence suggests it cannot be bootstrapped from reward alone when the action space requires syntactic precision.

The benchmark-dependent RL dynamics (LiveVQA improves steadily with GRPO; BrowseComp-VL and HLE oscillate without clear upward drift) add nuance: even with SFT cold-start, RL's benefits are not uniform. Tasks where success depends on incremental improvements to tool-use strategy (better search queries, more efficient sequencing) benefit more than tasks where success hinges on rare insight moments that random exploration is unlikely to discover. This tempers the narrative that RL is a universal improver for agent capabilities and suggests that SFT quality—the coverage and diversity of the demonstrated trajectories—is the primary driver of performance on the hardest benchmarks.

Innovation 4: The BrowseComp-VL Benchmark Tests a Capability Not Evaluated by Any Existing VQA Dataset: Cross-Modal, Multi-Tool Information Seeking Under Ambiguity

Existing VQA benchmarks, by the paper's own characterization (Section 5), assess one or more of: single-step perception (standard VQA, OK-VQA), domain-specific knowledge (MMMU, MMMU-Pro), spatial reasoning (Open3DVQA), or visual search (MMSearch, LiveVQA). What none of them evaluates—and what BrowseComp-VL is explicitly designed to test—is the integration of all these skills plus tool coordination under entity obfuscation. The benchmark demands that an agent (a) perceive and identify an entity from an image, (b) resolve ambiguous textual references through web search and cross-referencing, (c) navigate multiple webpages to gather structured information about the entity, (d) potentially execute code to perform calculations or verify constraints, and (e) synthesize the results into a concise answer—often requiring the agent to notice and resolve contradictions across sources.

This is not merely a "harder" VQA benchmark using more obscure facts or finer-grained visual details. It is a different evaluation philosophy. Traditional VQA asks: "Given an image and a question, can the model produce the correct answer using its internal knowledge and visual understanding?" BrowseComp-VL asks: "Given an image and an intentionally ambiguous question, can the agent autonomously plan and execute a multi-tool information-gathering strategy to resolve the ambiguity and arrive at the correct answer?" The distinction is between testing knowledge and testing research capability.

The difficulty gradient across baselines in Table 2 supports this distinction. Direct inference models (GPT-4o, Gemini-2.5-flash, Qwen-2.5-VL family) score below 12% on BrowseComp-VL—far below their scores on LiveVQA (23.7–35.0%) and SimpleVQA (30.7–63.0%). This is not because BrowseComp-VL requires more arcane knowledge (SimpleVQA also tests factual knowledge) but because it demands a process—searching, browsing, cross-referencing—that direct inference models simply cannot perform. The RAG workflow improves performance (GPT-4o goes from 5.5% to 13.4% average), but even the best RAG system (GPT-4o at 13.4%) remains far below WebWatcher-32B (27.0%). The remaining gap is attributable to the agent's ability to use multiple coordinated tools (including Visit, Code Interpreter, and OCR) in a flexible, adaptive loop rather than following a fixed retrieve-then-answer pipeline.

The benchmark's two-level design is itself a methodological contribution: by providing both explicit-entity (Level 1) and obfuscated-entity (Level 2) versions of conceptually similar tasks, BrowseComp-VL enables researchers to measure not just overall capability but robustness to ambiguity—the performance gap between Level 1 and Level 2 isolates the cost of entity resolution. WebWatcher-32B's gap (28.4% vs. 25.0%, a 3.4 percentage point drop) is substantially smaller than GPT-4o+RAG's gap (16.8% vs. 7.0%, a 9.8 point drop), quantifying the agent's advantage in handling ambiguity through exploration rather than being blocked by unclear queries. This diagnostic capability is absent from benchmarks that only test at a single difficulty or ambiguity level.

5. Experimental Analysis

Evaluation Methodology

• Dataset. WebWatcher is evaluated on five benchmarks: (1) BrowseComp-VL, the paper's newly introduced benchmark with 100 Level-1 and 200 Level-2 instances, manually verified by PhD-level AI experts (Section 4.1, Appendix D.4); (2) Humanity's Last Exam (HLE) (Phan et al., 2025), using the 330 multimodal questions from the full 2,500-question set; (3) LiveVQA (Fu et al., 2025), using a 300-example subset from the full 3,602 instances; (4) SimpleVQA (Cheng et al., 2025), using 300 randomly sampled English QA pairs from the 1,013 available; and (5) MMSearch (Jiang et al., 2024), using the 171 image-paired examples from the 300 total. The diversity of benchmarks spans knowledge-intensive multi-hop reasoning (BrowseComp-VL, HLE), time-sensitive visual knowledge (LiveVQA), factual visual perception (SimpleVQA), and multimodal search (MMSearch).

• Base model(s). All WebWatcher variants are built on Qwen2.5-VL, with experiments reported for the 7B and 32B parameter versions (Bai et al., 2025). The paper also reports baseline results for the 72B variant under direct inference and RAG workflows (Tables 1 and 2). The Qwen2.5-VL family is chosen as a strong open-source vision-language foundation with demonstrated multimodal reasoning capability, and the 7B/32B/72B scaling allows analysis of how WebWatcher's training pipeline interacts with base model scale.

• Metrics. The primary metric is accuracy, specifically pass@1 and pass@k (Section 4.1). For pass@1, the model generates a single answer per question, and accuracy is computed as the fraction of questions answered correctly. For pass@k, the model generates k independent answers per question (with temperature 0.6 and top-p 0.95), and pass@k is the fraction of questions for which at least one of the k answers is correct. Answer correctness is judged using an LLM-as-judge approach (Liu et al., 2024; Wang et al., 2024a) with the evaluation prompt provided in Appendix D.5, which extracts the final answer, compares it to the ground truth with allowance for "small numerical margins," and outputs a binary correct/incorrect judgment. The pass@k metric follows the standard definition from Chen et al. (2021). For HLE, results are additionally broken down by academic subfield (Biology, Chemistry, CS/AI, Engineering, Humanities, Math, Physics, Other) in Table 1. All reported scores represent the average of three inference runs to account for sampling variance.

• Baselines. The paper evaluates four categories of baselines (Section 4.1):

  • Direct Inference: Models generate answers using only internal knowledge, without external retrieval or planning. Evaluated models include GPT-4o (OpenAI, 2024), Gemini-2.5-flash (DeepMind, 2025), Claude-3.7-Sonnet (Anthropic, 2025), and Qwen-2.5-VL at 7B, 32B, and 72B scales.
  • RAG Workflow: Two-stage retrieve-then-answer pipelines where relevant information is first retrieved (via web search) and then used to condition answer generation. The same set of models (GPT-4o, Gemini-2.5-flash, Claude-3.7-Sonnet, Qwen-2.5-VL 7B/32B/72B) is evaluated under this paradigm.
  • Reasoning Baselines: OmniSearch (Li et al., 2025c), a search-oriented reasoning agent built on GPT-4o, representing the prior state-of-the-art among open-source agentic approaches. Additionally, Gemini-2.5-Pro (DeepMind, 2025) and o4-mini (OpenAI, 2025b) represent proprietary multi-step reasoning models.
  • WebWatcher variants: WebWatcher-7B and WebWatcher-32B, both trained with the full pipeline (SFT cold-start on filtered trajectories followed by GRPO).

  The paper does not include an ablation where WebWatcher is evaluated without RL (SFT-only), which would isolate the contribution of GRPO, nor does it compare against the base Qwen2.5-VL models equipped with the same tool suite but without the specialized trajectory training, which would isolate the contribution of the data pipeline.

• Generation budget / compute accounting. The paper does not measure compute in FLOPs or wall-clock time. Instead, the implicit compute budget is the number of tool calls per trajectory (capped at 15; Section 3.1.2, Appendix D.2) and the number of independent rollouts k for pass@k evaluation (Section 4.2, Figure 7). Each trajectory can invoke up to 15 tool calls, each of which may involve API calls (Google SerpApi, Jina.ai) or local computation (Code Interpreter, OCR). For the pass@k scaling analysis on HLE (Figure 7), k ranges from 1 to 32, with each additional rollout linearly increasing the total inference cost. There is no FLOPs-matched comparison across methods analogous to the pretraining-vs-inference tradeoff analysis in some prior work. The computational cost of difficulty estimation or trajectory generation is not included in any reported efficiency metric.

• Cross-validation / statistical protocol. The paper reports that all main results represent the average of three inference runs (noted in Tables 1 and 2 captions), which accounts for stochasticity from sampling but does not provide confidence intervals, standard deviations, or statistical significance tests. For BrowseComp-VL evaluation, the test set (100 Level-1 + 200 Level-2 instances) is manually verified by PhD-level AI experts but is fixed for all models—there is no cross-validation or multiple test splits. HLE evaluation uses the official 330-question multimodal subset without modification. For LiveVQA, SimpleVQA, and MMSearch, the paper evaluates on randomly sampled subsets (300, 300, and 171 examples respectively), but does not report whether these subsets are fixed across models or resampled, and does not report variance across different random subsets. The lack of confidence intervals makes it difficult to assess whether reported differences (e.g., WebWatcher-32B at 27.0% vs. OmniSearch at 16.3% on BrowseComp-VL average) are statistically significant or within sampling noise for these small test sets.

Main Quantitative Results

Results on Humanity's Last Exam (Table 1)

WebWatcher-32B achieves an average score of 13.6% on the HLE multimodal subset, outperforming all direct inference models (GPT-4o: 6.5%, Gemini-2.5-flash: 4.9%, Claude-3.7-Sonnet: 2.8%, Qwen-2.5-VL-72B: 4.9%), all RAG workflow baselines (GPT-4o: 12.3%, Gemini-2.5-flash: 11.4%, Qwen-2.5-VL-72B: 8.6%), and the OmniSearch agent (9.3%, built on GPT-4o). The margin over the strongest RAG baseline (GPT-4o at 12.3%) is 1.3 percentage points, which is modest. The margin over OmniSearch is 4.3 points. WebWatcher-32B remains below the proprietary reasoning models Gemini-2.5-Pro (15.8%) and o4-mini (16.0%).

Breaking down by subfield, WebWatcher-32B shows particular strength in Biology (33.8%, substantially exceeding GPT-4o direct at 13.8%, GPT-4o RAG at 9.8%, and OmniSearch at 15.5%) and achieves the highest score among open-source agents in CS/AI (6.7% for WebWatcher-7B), Engineering (7.7% for WebWatcher-7B), and Other (17.2% for WebWatcher-7B). However, it scores 0.0% in CS/AI for the 32B variant, performs below OmniSearch in Physics (14.3% vs. OmniSearch's 21.4%), and trails GPT-4o RAG substantially in Chemistry (9.7% vs. 24.1%). The 7B variant (10.6% average) outperforms the 32B variant on CS/AI and Engineering but underperforms on Biology and overall average. This non-monotonic scaling—where the 7B model sometimes beats the 32B model on specific subfields—is noted but not explained in the paper. WebWatcher-7B's average of 10.6% exceeds OmniSearch's 9.3%, meaning even the smaller WebWatcher variant surpasses the GPT-4o-based agent baseline overall.

A critical observation: the HLE subset contains only 330 multimodal questions, and each subfield contains between approximately 20 and 60 questions (the exact per-subfield counts are not reported, but can be inferred from the percentage denominators). For instance, a score of 33.8% in Biology for WebWatcher-32B could represent roughly 10–20 correct answers out of 30–60 questions. With such small per-subfield sample sizes, the subfield breakdowns should be interpreted cautiously—a difference of 2–3 questions correct can produce large percentage swings.

The paper also reports a pass@k scaling curve on HLE for WebWatcher-32B (Figure 7). Pass@1 is 13.6%. Pass@3 rises to 20.3%, pass@8 reaches approximately 28% (interpolated from the curve), pass@16 reaches 35.7%, and pass@32 reaches 41.9%. The curve is "smooth" and "jump-free" (Section 4.3), with monotonically increasing but diminishing returns. The 41.9% at k=32 substantially exceeds both Gemini-2.5-Pro (15.8%) and o4-mini (16.0%) at single-inference. The paper attributes the smoothness to "de-correlated sampling strategy" that generates "mutually informative trajectories rather than redundant variants," though no evidence is presented that the trajectories are genuinely de-correlated beyond the use of temperature 0.6 and top-p 0.95.

Results on BrowseComp-VL, LiveVQA, MMSearch, and SimpleVQA (Table 2)

BrowseComp-VL. WebWatcher-32B achieves 28.4% on Level 1 and 25.0% on Level 2, averaging 27.0%. This more than doubles the best RAG baseline (GPT-4o at 13.4% average) and substantially exceeds the prior best agent OmniSearch (16.3%). The 7B variant (21.2% average) also significantly outperforms OmniSearch. The performance gap between WebWatcher and baselines is largest on Level 2, where entity obfuscation makes retrieval pipelines less effective: GPT-4o RAG drops from 16.8% (Level 1) to 7.0% (Level 2), a 9.8 percentage point decline, while WebWatcher-32B drops only from 28.4% to 25.0%, a 3.4 point decline. This quantifies the agent's relative robustness to question ambiguity.

The 27.0% average on BrowseComp-VL—despite being the best reported result—means that WebWatcher-32B still fails on nearly three-quarters of BrowseComp-VL questions. The paper does not analyze error modes on BrowseComp-VL, so it is unclear whether failures stem from visual misidentification, inadequate search coverage, reasoning errors, tool-use failures, or the 15-call budget being insufficient. Direct inference models score below 10% on this benchmark (GPT-4o: 5.5%, Qwen-2.5-VL-72B: 7.1%), confirming that the benchmark genuinely requires external information gathering beyond internal knowledge.

LiveVQA. WebWatcher-32B achieves 58.7%, exceeding all direct inference models (Gemini-2.5-flash: 35.0%, GPT-4o: 29.7%), all RAG baselines (Gemini-2.5-flash RAG: 41.3%, GPT-4o RAG: 34.0%), and OmniSearch (40.9%). The margin over the best RAG baseline is 17.4 points, and over OmniSearch is 17.8 points—both substantial. Even WebWatcher-7B (51.2%) outperforms all non-WebWatcher baselines. LiveVQA tests "up-to-date visual knowledge" from recent global news, where information freshness matters—retrieval pipelines that capture stale or irrelevant results would be penalized. The strong performance suggests WebWatcher's flexible tool-use loop effectively navigates time-sensitive information environments.

MMSearch. WebWatcher-32B achieves 55.3%, exceeding OmniSearch (49.7%) and all direct inference and RAG baselines (the best RAG is Gemini-2.5-flash at 43.9%). WebWatcher-7B (49.1%) is roughly competitive with OmniSearch. MMSearch includes both recent news and rare knowledge across 14 subdomains, with the evaluated subset containing 171 image-paired examples—the smallest test set among the benchmarks, making the reported scores more sensitive to sampling variance.

SimpleVQA. This benchmark primarily tests "fine-grained visual perception and reasoning rather than external knowledge." WebWatcher-32B achieves 59.0%, which is competitive but not dominant: Gemini-2.5-flash RAG reaches 68.6%, GPT-4o RAG reaches 61.6%, and OmniSearch reaches 63.0%. The paper acknowledges this result shows WebWatcher "attains a leading score" but the numbers indicate it is mid-pack among RAG and agentic baselines. This is not surprising given WebWatcher's design emphasis on tool-augmented information seeking—SimpleVQA's focus on perceptual reasoning means external tools provide less advantage, and the overhead of the agent loop may introduce failure modes (incorrect tool calls, unnecessary searches) that pure perception models avoid.

The WebWatcher-7B variant (54.3%) underperforms several RAG baselines on SimpleVQA, suggesting that the smaller model may struggle more with the tool-use overhead on perception-heavy tasks where tools provide marginal benefit.

A notable pattern across all four benchmarks in Table 2: the absolute performance levels are low to moderate across the board, even for the best systems. On BrowseComp-VL, 27.0% is state-of-the-art. On HLE, 13.6% is state-of-the-art among open-source agents. These are genuinely hard benchmarks, and the paper is transparent that significant headroom remains. The pass@k scaling on HLE (reaching 41.9% at k=32) demonstrates that computational scaling can substantially close the gap, but at significant inference cost.

Tool Usage Distribution (Figure 5)

Figure 5 shows the percentage of tool calls made to each of the five tools across the four benchmarks where tool usage is most relevant. The distribution varies sharply by benchmark:

• HLE: A relatively balanced distribution—Web Text Search, Web Image Search, and Code Interpreter each account for approximately 25% of calls, with Visit making up most of the remaining ~25%. This reflects HLE's diverse task composition, which includes multimodal search, numerical computation, and visual reasoning.

• BrowseComp-VL: Web Text Search dominates at 62% of all calls. Web Image Search is used much less (~15%), and Code Interpreter is almost never invoked. Visit accounts for approximately 20%. This aligns with BrowseComp-VL's design as an information-seeking benchmark where the primary challenge is navigating web sources to resolve entity-obfuscated queries—text search and page visits are the core operations.

• LiveVQA, SimpleVQA, MMSearch: Web Image Search jumps to approximately one-third or more of all calls across these three benchmarks, reflecting their emphasis on visual content. Web Text Search and Visit serve as auxiliary tools providing textual evidence and navigation. Code Interpreter is invoked sparingly across all benchmarks, appearing "only when genuine calculation is needed."

The paper interprets this distribution as evidence that the "agent is cost and context aware" and "able to flexibly choose the right tool chain rather than defaulting to a single strategy." This interpretation is plausible but should be tempered: the tool distribution could also reflect biases in the SFT trajectories that taught the model which tools to prefer for different task types, and the paper does not compare the tool distribution of WebWatcher against what an optimal policy would use or what human experts would employ.

Cold-Start Analysis (Figure 6)

Figure 6 compares RL training dynamics under two initializations—Instruct (no SFT cold-start) and Cold-start (with SFT on tool-use trajectories)—across HLE, BrowseComp-VL, and LiveVQA. The results are stark:

• Instruct initialization: The agent remains near zero reward for many training steps on all three benchmarks. The paper explains that "tool-call format errors wipe out the reward and the strict Qwen-2.5-72B grader further suppresses partial answers."

• Cold-start initialization: Initial scores lift to 0.12 on HLE, 0.30 on BrowseComp-VL, and 0.45 on LiveVQA after SFT alone (before GRPO begins). The subsequent GRPO dynamics diverge by benchmark:

  • LiveVQA: Shows a "steady rise and keeps a 0.06–0.18 margin over the Instruct baseline throughout" the RL training.
  • HLE and BrowseComp-VL: "Oscillate heavily with no clear upward drift" during GRPO training.

This is a critical finding. The necessity of cold-start SFT is unambiguously demonstrated—without it, RL cannot bootstrap tool-use competence. However, the sufficiency of cold-start SFT + GRPO is benchmark-dependent: on LiveVQA, the combination yields clear gains; on HLE and BrowseComp-VL, GRPO does not produce reliable improvement over the SFT-only policy (which already achieves the initial scores of 0.12 and 0.30). The paper does not report the final GRPO-trained scores on these benchmarks in Figure 6—the x-axis shows training steps, not converged performance—so the eventual RL benefit on HLE and BrowseComp-VL remains unclear from this figure alone. The main results in Tables 1 and 2 presumably use the fully trained (SFT + GRPO) models, but the Figure 6 dynamics suggest that on the hardest benchmarks, most of the performance comes from SFT.

The paper also reports a negative result: "Injecting CoT chains from a larger reasoner made the small model unstable, format violations, repetitions, and context overflow spiked." This confirms that textual reasoning traces are not a substitute for action-grounded SFT trajectories—the model must learn the structured interaction format, not just abstract reasoning patterns.

Pass@k Scaling on HLE (Figure 7)

The pass@k curve for WebWatcher-32B on HLE shows the following values (read from the curve in Figure 7):

• k=1: 13.6%
• k=2: ~17% (interpolated)
• k=3: 20.3%
• k=4: ~23% (interpolated)
• k=8: ~28% (interpolated)
• k=16: 35.7%
• k=32: 41.9%

The curve rises steeply for small k (k=1 to k=4 roughly doubles the score from 13.6% to ~23%) and continues rising at a slower rate for larger k. The paper highlights that at k=3, the score of 20.3% already exceeds all single-inference baselines including reasoning models (Gemini-2.5-Pro at 15.8%, o4-mini at 16.0%). The curve is "smooth, jump-free" and "monotonically increasing," which the paper presents as evidence that the trajectories are "mutually informative" rather than redundant.

This interpretation warrants scrutiny. A smooth pass@k curve is consistent with independent, identically distributed samples where each sample has a fixed probability p of being correct—the pass@k curve would follow 1 - (1-p)^k. The fact that the curve is smooth does not, by itself, demonstrate that trajectories are "mutually informative" (which would imply that later samples build on insights from earlier ones, or that the sampling strategy actively diversifies reasoning paths). A smooth curve is equally consistent with simple independent sampling. The paper does not compare the observed pass@k curve against the theoretical curve for independent sampling at the base pass@1 rate, which would test whether the trajectories are genuinely complementary or merely independent. If the observed pass@k rises faster than 1 - (1-p)^k with p=0.136, that would indicate complementarity; if it matches the independent-sampling curve, the gains come purely from increased sample count.

The practical implication is that "practitioners may cap the budget at 8–16 roll-outs to secure a 2 to 3 fold accuracy boost while containing compute costs." At k=16, the score is 35.7% (2.6× the k=1 score), and at k=8, the score is ~28% (2.1×). The diminishing returns after k=8 suggest that additional rollouts beyond this point become increasingly expensive per unit of accuracy gained.

Ablation Studies and Robustness Checks

SFT cold-start vs. direct RL (Figure 6): As discussed above, RL from instruction-tuned initialization fails entirely, with the agent staying near zero reward due to format errors. Cold-start SFT lifts initial scores to 0.12 (HLE), 0.30 (BrowseComp-VL), and 0.45 (LiveVQA) before RL begins. The subsequent RL dynamics are benchmark-dependent: LiveVQA shows steady improvement; HLE and BrowseComp-VL oscillate without clear upward drift. This ablation is the strongest empirical result in the paper—it cleanly demonstrates necessity of SFT cold-start. However, it does not fully answer the question of sufficiency: the paper does not report final SFT-only performance (without any RL) on the main benchmarks, which would isolate how much GRPO contributes beyond SFT alone.

Model scale (7B vs. 32B): The paper trains and evaluates both WebWatcher-7B and WebWatcher-32B. The 32B model generally outperforms the 7B model (Tables 1 and 2), but with notable exceptions: on HLE, WebWatcher-7B achieves higher scores in CS/AI (6.7% vs. 0.0%) and Engineering (7.7% vs. 5.8%), while WebWatcher-32B dominates in Biology (33.8% vs. 18.6%). The paper does not analyze these scale anomalies—they could reflect genuine capability differences, training instability, or sampling noise on small per-subfield test sets. The consistent gap on most benchmarks (e.g., BrowseComp-VL: 21.2% vs. 27.0%, LiveVQA: 51.2% vs. 58.7%) indicates that model scale provides meaningful gains, but the non-monotonic subfield results on HLE suggest that larger scale does not uniformly improve all capabilities.

CoT chains as SFT substitute (Section 4.3): The paper reports a negative result: injecting chain-of-thought chains from a larger reasoner as a substitute for action-grounded SFT trajectories "made the small model unstable, format violations, repetitions, and context overflow spiked." This is described qualitatively in Section 4.3 but is not presented as a formal ablation with quantitative results in a table or figure. The paper states this finding to reinforce the claim that action-grounded trajectories—with actual tool calls and environment responses—provide a kind of supervision that textual reasoning traces cannot replicate.

Tool usage distribution (Figure 5): This analysis shows how WebWatcher adapts its tool selection to benchmark demands, but it is descriptive rather than causal. The paper does not ablate individual tools (e.g., removing Code Interpreter or OCR and measuring performance impact), so it is unclear whether all five tools contribute meaningfully or whether the agent could achieve similar performance with a subset. For BrowseComp-VL, where Code Interpreter is almost never used (Figure 5), removing it would likely have negligible impact; for HLE, where it accounts for ~25% of calls, its contribution might be substantial. Without tool ablation experiments, the necessity of each tool remains unquantified.

Training data mixture ratio (5:3:2): The paper states that the final training data ratio is 5:3:2 for BrowseComp-VL, long-tail VQA, and hard VQA data respectively (Section 4.1), but does not ablate this ratio. It is unknown whether the BrowseComp-VL-heavy mixture is optimal, whether the long-tail data prevents overfitting to BrowseComp-VL's specific distribution, or whether the hard VQA examples contribute to perceptual robustness. An ablation varying the data mixture—or training on BrowseComp-VL data alone—would clarify how each data source contributes to the diverse benchmark performance.

Trajectory filtering stages: The three-stage trajectory filter (final answer matching, step-by-step consistency, minimum tool usage) reduces 180,000 raw VQA examples (60,000 Level-1 + 40,000 Level-2 after VQA conversion and filtering, per Section 4.1) to 8,000 SFT trajectories—a ~95.6% reduction. The paper does not report per-stage filter rates, so it is unknown which stage is the primary bottleneck. If most trajectories fail the step-by-step consistency check (Stage 2), that would suggest GPT-4o struggles to generate logically coherent multi-step tool-use sequences; if most fail the final answer matching (Stage 1), the limitation is in reaching correct answers even with tool access. The paper also does not ablate the filtering stages—e.g., comparing model performance when trained on trajectories that pass only Stage 1 vs. Stages 1+2 vs. all three stages—which would quantify how much each filtering stage contributes to downstream agent quality.

GRPO reward weights (w = 0.2): The total reward balances format score (weight 0.2) and semantic accuracy (weight 0.8). The paper does not ablate this weight. A lower weight on format would permit more format errors but potentially faster exploration of semantic strategies; a higher weight would enforce stricter format adherence at the cost of exploration. The choice of 0.2 is stated but not justified empirically.

Number of GRPO rollouts per group (K = 16): The paper uses groups of 16 trajectories for computing relative advantages (Section 3.3) but states a rollout group size of 8 in the RL training details (Appendix D.3). This discrepancy is not explained. The group size affects the stability of the advantage estimate—smaller groups give noisier baselines—but no ablation is provided.

Maximum tool calls per trajectory (15): The 15-call budget is stated (Appendix D.2) but not ablated. It is unknown whether performance saturates before 15 calls (in which case a lower budget would be equally effective and more efficient) or whether additional calls beyond 15 would yield further gains. Given that BrowseComp-VL requires "multi-page web browsing plus fine-grained visual grounding" (Section 4.2), some questions may require more than 15 tool calls to fully resolve.

Temperature and top-p for pass@k (0.6 and 0.95): These sampling parameters are stated for pass@k evaluation but not ablated. Different temperature settings would trade off between trajectory diversity and quality—higher temperature increases diversity (potentially improving pass@k) but may reduce individual trajectory quality (potentially hurting pass@1). The chosen values are standard but their impact on the pass@k curve is not explored.

Critical Assessment

Claim 1: WebWatcher achieves state-of-the-art results on challenging VQA benchmarks, outperforming proprietary baselines and open-source agents.

The evidence partially supports this claim, with important qualifications. On BrowseComp-VL, WebWatcher-32B's 27.0% average genuinely represents a new state of the art among evaluated systems, more than doubling GPT-4o RAG (13.4%) and substantially exceeding OmniSearch (16.3%). On LiveVQA (58.7%) and MMSearch (55.3%), WebWatcher-32B similarly achieves the highest reported scores. However, on HLE, the 13.6% average is below proprietary reasoning models Gemini-2.5-Pro (15.8%) and o4-mini (16.0%), and the margin over GPT-4o RAG (12.3%) is only 1.3 points—small enough that it could fall within sampling variance on a 330-question test set. On SimpleVQA, WebWatcher-32B (59.0%) is mid-pack, below Gemini-2.5-flash RAG (68.6%), GPT-4o RAG (61.6%), and OmniSearch (63.0%). So the claim of "state-of-the-art" holds for three of five benchmarks, is ambiguous on HLE (state-of-the-art among open-source agents, but not overall), and does not hold on SimpleVQA.

Moreover, the baselines are not uniformly strong. The RAG workflow baselines use a generic two-stage retrieve-then-answer pipeline without iterative tool use, and the direct inference baselines use no external tools at all. A stronger baseline would be Qwen2.5-VL equipped with the same tool suite as WebWatcher but without the specialized SFT trajectories—this would isolate the contribution of the data and training pipeline from the contribution of simply having access to tools. The paper does not report such a baseline. OmniSearch, the most relevant agentic baseline, is built on GPT-4o, which introduces a confound: performance differences could reflect the underlying model (GPT-4o vs. Qwen2.5-VL-32B) rather than the agent architecture and training.

Claim 2: The BrowseComp-VL benchmark fills a gap by testing cross-modal, multi-tool information seeking under ambiguity.

The design of BrowseComp-VL is well-motivated and the two-level structure (explicit vs. obfuscated entities) provides a genuine diagnostic capability absent from existing benchmarks. The evidence that it tests something distinct is strong: direct inference models score below 10% (Table 2), confirming that external information gathering is necessary; RAG workflows improve but remain below 14%, confirming that simple retrieval is insufficient; and the Level 1 to Level 2 performance drop is larger for retrieval pipelines than for WebWatcher, confirming that entity obfuscation disproportionately challenges non-agentic approaches.

However, the benchmark has limitations that the paper does not fully acknowledge. The test set is small—100 Level-1 and 200 Level-2 instances—and while manually verified by PhD-level experts, a 300-question benchmark provides limited statistical power for differentiating models, particularly when performance differences are in the single-digit percentage range. The benchmark is constructed entirely through the automated pipeline described in Section 2, and while manual verification ensures correctness, it does not guarantee diversity or representativeness—the benchmark reflects the distribution of entities and reasoning patterns captured by Wikipedia hyperlink traversal and GPT-4o's QA generation, which may have systematic biases. The benchmark's difficulty (state-of-the-art is 27.0%) means that a large fraction of questions are not answered correctly by any system, limiting its ability to discriminate among stronger future models that may saturate the easier questions while still failing on the hardest ones.

Claim 3: Cold-start SFT on action-grounded trajectories is necessary before RL can yield meaningful gains for multimodal tool-use tasks.

The evidence in Figure 6 strongly supports this claim. The Instruct initialization fails entirely, with near-zero reward persisting for many training steps. Cold-start SFT lifts initial scores substantially (0.12 on HLE, 0.30 on BrowseComp-VL, 0.45 on LiveVQA). The necessity of SFT cold-start is the most robust finding in the paper.

However, the sufficiency of SFT + GRPO for achieving the final reported performance is less well-established. Figure 6 shows that on HLE and BrowseComp-VL, GRPO training oscillates without clear upward drift—meaning that the final models used for Tables 1 and 2 may derive most of their performance from SFT alone, with GRPO providing at most marginal gains. The paper does not report SFT-only performance on the main benchmarks, so the incremental contribution of RL is unknown. If SFT-only models achieve similar scores to the reported SFT+GRPO models, the claim that RL "further refines decision-making and adapts to complex tasks" would be undermined for the hardest benchmarks. The benchmark-dependent RL dynamics (LiveVQA improves; HLE and BrowseComp-VL do not) suggest that RL's benefits are not universal and depend on task characteristics that are not fully characterized.

The additional finding—that CoT chains cannot substitute for action-grounded SFT—is stated qualitatively but not presented with quantitative evidence. A formal ablation comparing SFT on CoT traces vs. SFT on action-grounded trajectories, with both evaluated on the main benchmarks, would substantially strengthen this claim.

Claim 4: The QA-to-VQA conversion pipeline enables scalable generation of multimodal reasoning data.

The pipeline is well-described and the scale is impressive (180,000 raw QA pairs converted to 100,000 VQA examples). However, the claim of scalability is not directly tested. The paper does not demonstrate that the pipeline works with QA sources other than the custom-built Level 1 and Level 2 data—for instance, applying it to existing text QA benchmarks like HotpotQA or StrategyQA and showing that the resulting VQA data yields useful training signal. The pipeline's dependence on GPT-4o for QA generation, question rewriting, and quality control means that the data inherits whatever biases and limitations GPT-4o has. The paper does not analyze the failure modes of the conversion—how often entity masking introduces unintended ambiguity, how often the retrieved images are insufficient for identification, or how often the filtering pipeline incorrectly accepts or rejects examples. Without such analysis, the pipeline's robustness and generalizability remain unvalidated.

Missing experiments that would strengthen the paper:

• SFT-only evaluation on all benchmarks. Reporting WebWatcher performance after SFT but before GRPO would isolate RL's contribution. If SFT-only scores are close to the reported scores, the paper's emphasis on RL would be overstated.
• Tool ablation study. Evaluating WebWatcher with individual tools removed (no Code Interpreter, no OCR, no Visit) would quantify each tool's contribution and identify which tools are essential vs. optional for different benchmarks.
• Ablation of trajectory filtering stages. Training models on trajectories that pass only Stage 1 filtering vs. Stages 1+2 vs. all three stages would quantify the value added by consistency checking and minimum tool usage requirements.
• Baseline with tool-equipped Qwen2.5-VL without WebWatcher training. Giving the base Qwen2.5-VL model access to the same tool suite (with appropriate prompting) but without the specialized SFT trajectories would isolate the training pipeline's contribution from the contribution of tool access alone.
• Larger-scale evaluation. The test sets are small (100–330 questions per benchmark), and confidence intervals are not reported. Running evaluations on larger subsets or with bootstrap confidence intervals would clarify which performance differences are statistically reliable.
• Error analysis on BrowseComp-VL. The paper does not categorize or analyze the 73% of BrowseComp-VL questions that WebWatcher-32B answers incorrectly, which would identify the primary bottlenecks (visual misidentification, inadequate search, reasoning errors, budget exhaustion) and guide future improvements.
• Human baseline on BrowseComp-VL. Without knowing how well human experts perform on this benchmark, it is difficult to interpret the 27.0% state-of-the-art score—is this benchmark approaching human-level difficulty, or is it simply poorly matched to current AI capabilities?

Assessment of the paper's core contribution:

The paper makes a genuine contribution in identifying and addressing the gap between text-only deep research agents and the multimodal reality of real-world information-seeking tasks. The BrowseComp-VL benchmark, with its entity obfuscation and multi-tool demands, evaluates a capability that no existing benchmark captures. The data generation and training pipeline—while not every component is rigorously validated—represents a coherent approach to scaling multimodal agent training data. The finding that SFT cold-start on action-grounded trajectories is necessary for tool-use RL is empirically solid and has implications beyond this specific system. However, the modest absolute performance on the hardest benchmarks (27.0% on BrowseComp-VL, 13.6% on HLE) and the unclear contribution of RL to these benchmarks suggest that the core challenge—building agents that can reliably perform multimodal deep research—remains largely unsolved, and WebWatcher represents an important step rather than a breakthrough.

6. Limitations and Trade-offs

Controlling Trajectory Quality Requires Extreme Filtering That Discards ~96% of Generated Data

The trajectory annotation pipeline generates supervision by having GPT-4o simulate tool-use reasoning, then filters the results through a three-stage process. The paper reports that 180,000 raw VQA examples (60,000 Level-1 + 40,000 Level-2 after initial conversion and filtering) are reduced to only 8,000 SFT trajectories after the full filtering pipeline is applied (Section 4.1). This represents a ~95.6% rejection rate. Another 2,000 examples are reserved for GRPO, but this only modestly changes the ratio—the combined supervision dataset (10,000 trajectories) represents roughly 5.6% of the raw data that enters the pipeline.

The consequence is that WebWatcher's training depends on GPT-4o being able to produce, through rejection sampling, a small subset of trajectories where every intermediate step is logically consistent and the final answer is correct. This has several implications. First, it means the SFT data covers only the subset of problems that GPT-4o can solve with correct intermediate reasoning when given access to tools—if GPT-4o systematically fails on certain types of visual reasoning, entity disambiguation, or multi-page navigation, those failure modes are excluded from the training data rather than being demonstrated (with corrections) for the student model to learn from. The student model never sees examples of error recovery because trajectories with errors are filtered out. Second, the extreme filtering rate suggests that GPT-4o's simulated trajectories are overwhelmingly low-quality—ridden with hallucinated content, logical inconsistencies, unjustified tool calls, or incorrect final answers—which raises questions about the reliability of the trajectory generation process itself. If 95% of simulated expert demonstrations are unusable, the remaining 5% may represent an idiosyncratic subset of problems and strategies rather than a representative sample of effective reasoning.

The paper does not report per-stage filter rates (Section 3.1.3), so it is unknown whether the bottleneck is final-answer correctness (Stage 1), step-by-step consistency (Stage 2), or minimum tool usage (Stage 3). If most trajectories fail at Stage 2—the consistency check—that would indicate a fundamental limitation in GPT-4o's ability to generate coherent multi-step reasoning sequences with tool use, which has implications for any pipeline that relies on LLM-simulated trajectories as supervision. The paper partially acknowledges the filtering cost by describing the three-stage process in detail, but does not frame the 95% rejection rate as a limitation, nor does it analyze what distinguishes the surviving 5% from the discarded 95%. Mitigation is not attempted; the paper does not explore alternatives such as iterative refinement of trajectories, human correction of flawed trajectories, or using the filtered-out trajectories as negative examples. This is left entirely to future work.

The Method Cannot Solve the Hardest Problems Regardless of Compute Budget

Across multiple benchmarks and both training stages, the paper presents evidence that WebWatcher (and all evaluated baselines) makes negligible progress on questions that lie beyond a certain difficulty threshold. On BrowseComp-VL—the benchmark specifically designed to test the capabilities WebWatcher targets—the state-of-the-art result is 27.0% average accuracy (Table 2), meaning 73% of questions are answered incorrectly. On HLE, WebWatcher-32B achieves 13.6% (Table 1), with subfield scores of 0.0% in CS/AI, 5.8% in Engineering, and 8.9% in Math. On the hardest BrowseComp-VL questions (Level 2, with entity obfuscation), even WebWatcher-32B scores only 25.0%. Direct inference baselines on BrowseComp-VL score below 10% (GPT-4o: 5.5%, Qwen-2.5-VL-72B: 7.1%), confirming that the benchmark genuinely requires external information gathering, but the best agentic system still fails on three-quarters of the tasks.

The paper does not analyze why these failures occur—whether from visual misidentification of the grounding image, inability to resolve obfuscated entity references through search, insufficient tool-call budget (capped at 15), fundamental knowledge gaps that no amount of web search can fill, or reasoning errors in synthesizing gathered information. Without error categorization, it is unclear whether the 73% failure rate on BrowseComp-VL reflects a ceiling that better training, larger models, or more compute could push past, or a more fundamental limitation of the agentic paradigm on tasks requiring specific types of reasoning.

The pass@k scaling curve on HLE (Figure 7) shows that increased sampling does improve coverage—pass@1 of 13.6% rises to 41.9% at k=32—but even at k=32, more than half of HLE questions remain unsolved. The curve shows diminishing returns: the jump from k=1 to k=8 roughly doubles the score (13.6% → ~28%), but the jump from k=16 to k=32 adds only ~6 percentage points (35.7% → 41.9%). This suggests that for a substantial fraction of questions, no trajectory sampled by the model—even with 32 independent attempts—reaches the correct answer. These are questions where the model's policy simply does not assign meaningful probability to successful reasoning paths, and no amount of sampling within that policy will surface a correct solution. The paper does not characterize this "unreachable" subset or estimate its size at higher k values.

Mitigation is discussed only indirectly. The paper frames the pass@k results as demonstrating "robust, scalable improvements," but does not address the irreducible failure rate. The GRPO training dynamics (Figure 6) provide a partial explanation: on HLE and BrowseComp-VL, RL training oscillates without clear upward drift, meaning the optimization process itself cannot discover better strategies through exploration. This suggests the bottleneck is in the SFT data coverage and the base model's capabilities, not in the RL optimization. The paper does not propose solutions for the hardest-problem regime—such as targeted data augmentation for specific failure modes, larger base models, or fundamentally different reasoning architectures—and this remains an open challenge.

Difficulty Estimation Cost Is Not Accounted for in the System's Practical Efficiency

The paper's training and evaluation pipeline relies on several expensive preprocessing steps that are externalized from the reported performance metrics. First, the BrowseComp-VL data generation pipeline (Section 2) uses GPT-4o extensively—for QA synthesis from traversed web content, for entity obfuscation in Level 2, for QA-to-VQA conversion (entity masking and question rewriting), for relevance scoring during selector filtering, and for closed-book answer attempts during examiner filtering. A single raw QA pair passes through GPT-4o at least three times (generation, conversion, filtering), and the 95% rejection rate means most of these calls produce discarded data. Second, trajectory annotation (Section 3.1.2) uses GPT-4o to simulate multi-step reasoning with tool calls for each of the 100,000 filtered VQA examples that enter the pipeline—and 95% of these trajectories are subsequently discarded. Third, the SFT trajectories that survive filtering are used to train the model, but the computational cost of generating and filtering them (including actual API calls to Google SerpApi, Jina.ai, and execution of Python code) is not amortized into any reported efficiency metric.

The consequence is that the headline results—27.0% on BrowseComp-VL, 13.6% on HLE—are achieved after an enormously expensive data preparation phase that the paper does not quantify in dollar cost, API calls, or FLOPs. For a practitioner considering deploying a similar system, the primary cost may not be inference-time tool calls (capped at 15 per trajectory) but the upfront investment in generating filtered training data. The paper's framing—that WebWatcher demonstrates effective "deep research" capability through a scalable training pipeline—is accurate in principle, but the "scalability" refers to the ability to generate large volumes of training examples automatically, not to cost-efficiency. If generating 8,000 high-quality SFT trajectories requires producing and filtering ~160,000 raw trajectories (the approximate ratio implied by the 5% survival rate), and each raw trajectory involves multiple GPT-4o calls and API queries, the total cost could be substantial. The paper does not report this cost or compare it to alternatives such as human annotation, distillation from a stronger model, or using fewer but higher-quality seed examples.

Additionally, the test-time pass@k scaling (Figure 7) shows that achieving 41.9% on HLE requires 32 independent rollouts per question—each rollout potentially involving up to 15 tool calls with API queries. For 330 HLE questions, this represents up to 330 × 32 × 15 = 158,400 tool calls for a single evaluation. The paper does not discuss the latency or monetary cost of this inference regime, which would be relevant for any deployment scenario. The pass@k curve is presented as evidence of the agent's capability, but the cost to achieve that capability is externalized.

The paper partially acknowledges this class of concern in the context of BrowseComp-VL construction—noting that manual verification by PhD-level experts was used for the evaluation set (Appendix D.4)—but does not discuss the broader cost implications of the data generation and filtering pipeline. No mitigation is attempted; the paper does not explore cheaper alternatives for trajectory generation (e.g., using smaller models, reducing the number of filtering stages, or using the filtered-out trajectories as negative examples) or analyze how performance varies with the scale of SFT data. This limitation is structural: the method's effectiveness depends on high-quality trajectories, and achieving those trajectories currently requires generating and discarding an order of magnitude more data than is ultimately used.

Single Model Family and Single Task Domain Leave Generalization Unverified

All WebWatcher variants are built on the Qwen2.5-VL model family (7B and 32B), and all training and evaluation is conducted on visual question answering tasks in the knowledge-intensive, information-seeking domain—primarily entity-centric factual questions that can be answered by navigating web sources. BrowseComp-VL, HLE, LiveVQA, MMSearch, and SimpleVQA are all VQA benchmarks, and while they vary in their emphasis (multi-hop reasoning, time-sensitive knowledge, factual perception, visual search), they share a core structure: an image is provided, a question is asked, and the answer is a discrete fact, number, or short phrase that can be graded by string matching or LLM-as-judge comparison against a ground truth.

The consequence is that the paper provides no evidence about WebWatcher's effectiveness on: (1) different base model architectures or model families (e.g., LLaMA-based VL models, proprietary models other than Qwen2.5-VL); (2) non-VQA tasks that require multimodal deep research, such as generating research reports with embedded figures, comparing products across e-commerce websites using both visual and textual specifications, analyzing scientific papers with charts and tables, or conducting open-ended investigative research without a single correct answer; (3) tasks where the answer format is not a short fact but a structured output (a table, a list of citations, a generated image, an executable plan); (4) languages other than English (SimpleVQA includes Chinese examples, but the paper evaluates only on English); or (5) domains where web search tools are insufficient because the relevant information is in proprietary databases, paywalled journals, or non-indexed sources.

The choice of Qwen2.5-VL is stated without justification beyond it being a strong open-source vision-language model (Section 4.1). The paper does not claim that the approach is specific to Qwen2.5-VL, and the training methodology (SFT on tool-use trajectories + GRPO) is in principle model-agnostic. However, several aspects of the pipeline could interact with model-specific properties: the base model's native visual recognition capability affects whether the grounding images in BrowseComp-VL can be correctly identified; the model's instruction-following ability affects whether it can learn the structured tool-call format from SFT; the model's in-context reasoning capacity affects whether it can maintain coherence over 15-turn trajectories; and the model's calibration and exploration behavior affects GRPO training dynamics. Without replication on other model families, it is unknown whether WebWatcher's performance reflects the Qwen2.5-VL's particular strengths or the general effectiveness of the training pipeline.

The task domain limitation is more fundamental. The paper's central claim is that WebWatcher advances "multimodal DeepResearch," but all evaluation is on closed-form VQA. Real deep research—as exemplified by the proprietary systems the paper cites (OpenAI DeepResearch, Gemini Deep Research)—involves open-ended synthesis, multi-page report generation, comparison and contrast across sources, and iterative refinement of research questions. These capabilities are not tested by any benchmark in the paper. The BrowseComp-VL benchmark, while more complex than prior VQA, still reduces to a single correct answer per question. The paper does not discuss this gap between the evaluated capabilities and the "deep research" framing, nor does it acknowledge that VQA benchmarks—even challenging ones—represent a narrow slice of what multimodal research agents would need to do in practice.

The paper does not attempt to mitigate these generalization limitations—no additional model families are tested, no non-VQA tasks are evaluated, and the scope of claims is not explicitly bounded by the task domain. The limitation is structural to the current evaluation methodology and would require new benchmarks and experimental protocols to address.

The Incremental Contribution of Reinforcement Learning over Supervised Fine-Tuning Is Unquantified

The paper presents WebWatcher as trained through a two-stage pipeline: SFT cold-start followed by GRPO reinforcement learning (Section 3). The main results in Tables 1 and 2 report performance of the fully trained (SFT + GRPO) models. However, the paper never reports the performance of SFT-only models on these benchmarks—despite presenting an entire figure (Figure 6) analyzing RL training dynamics that strongly suggests GRPO may contribute little or nothing on the hardest benchmarks.

Figure 6 shows that on HLE and BrowseComp-VL, after SFT cold-start lifts the initial scores (to 0.12 and 0.30 respectively), the subsequent GRPO training "oscillate[s] heavily with no clear upward drift." The x-axis shows GRPO training steps, and the Cold-start curves for HLE and BrowseComp-VL are essentially flat with noise—there is no visible trend toward higher reward over the course of training. On LiveVQA, the Cold-start curve does show a steady rise, indicating that GRPO provides genuine benefit for that benchmark. But for the two benchmarks most central to the paper's claims—BrowseComp-VL (the paper's own benchmark) and HLE (the most prestigious external benchmark)—Figure 6 provides no evidence that RL improves over SFT.

The consequence is that the paper cannot disentangle how much of WebWatcher's performance comes from the SFT trajectory data (the careful construction of high-quality tool-use demonstrations) versus the GRPO optimization (the exploration and reward-driven refinement). If the SFT-only models achieve performance close to the reported numbers in Tables 1 and 2—which is plausible given the flat GRPO curves in Figure 6—then the paper's emphasis on RL as a key component of the method would be overstated. The core innovation would reduce to the data generation pipeline and SFT training, with RL serving as a benchmark-dependent fine-tuning step rather than a necessary stage.

This missing ablation is particularly significant because the paper makes specific claims about RL's role: "Reinforcement learning is then applied to further optimize tool use and decision-making" (Section 3 introduction), and GRPO "further refine[s] decision-making and adapt[s] to complex tasks" (Section 3.3). These claims are supported only for LiveVQA based on the available evidence. For BrowseComp-VL and HLE, the evidence in Figure 6 contradicts these claims—or at minimum, fails to support them. The paper also reports that the ReST^EM-trained revision model in a prior experiment (not this paper's main method) degraded performance, noting that RL-based self-improvement can backfire (referenced in Appendix K context from prior work). This precedent makes the unquantified RL contribution in WebWatcher more concerning: if RL can harm as easily as help, knowing its incremental effect is essential.

The paper does not acknowledge this as a limitation. The missing SFT-only baseline is not discussed, and the flat GRPO curves in Figure 6 are described without connecting them to the question of whether RL is necessary. Mitigation would require simply reporting SFT-only performance on all benchmarks—an experiment that uses models already trained (the SFT checkpoint before GRPO begins) and requires only evaluation. This omission is a significant gap in the experimental methodology.

The 15-Turn Tool-Call Budget Is Not Justified and May Be a Binding Constraint on Hard Problems

The paper caps the number of tool calls per trajectory at 15 (Appendix D.2), a constraint enforced during both training (trajectories are generated within this limit) and inference (the agent cannot exceed 15 think-act-observe cycles). This budget is stated without justification—there is no ablation comparing performance at different budget levels (e.g., 10 vs. 15 vs. 20 vs. 30 calls), no analysis of how often the agent exhausts its budget without reaching an answer, and no discussion of how the budget interacts with task difficulty.

The consequence is that some fraction of WebWatcher's failures—particularly on BrowseComp-VL and HLE, which the paper describes as requiring "multi-page web browsing plus fine-grained visual grounding" (Section 4.2) and "complex, knowledge-intensive VQA settings requiring cross-modal reasoning and external information integration" (Section 4.2)—may stem from insufficient tool-call budget rather than fundamental reasoning failures. The motivating example in Figure 2 shows WebWatcher using a sequence like "ImageSearch, ImageVis, PageView, EvidenceSum, OCR, Wikipedia, VisualDiff, WebSearch, CrossValidate, Count"—10 distinct tool invocations before reaching an answer. A more complex question requiring deeper exploration, more disambiguation attempts, or verification across additional sources could easily exceed 15 calls. The tool usage distribution in Figure 5 shows that BrowseComp-VL concentrates 62% of calls on Web Text Search, suggesting that the agent typically issues multiple search queries—if each query requires a follow-up Visit to the most promising results, and multiple rounds of search are needed to resolve obfuscated entities, 15 calls may be tight.

The paper does not report: (1) the distribution of trajectory lengths at inference time (how many trajectories hit the 15-call limit vs. finish earlier), (2) whether the agent's accuracy differs between trajectories that exhaust the budget vs. those that finish within budget, (3) how performance varies if the budget is increased to 20 or 30 calls, or (4) whether the marginal benefit of additional calls diminishes after some point. Without this analysis, the 15-call limit is an arbitrary constraint that could be suppressing the agent's true capability ceiling. The choice of 15 may be driven by practical considerations—context window length, API cost, training efficiency—but these tradeoffs are not discussed.

Mitigation is not attempted. The paper could have conducted a budget ablation study, or at minimum reported how often trajectories terminate due to budget exhaustion vs. agent decision, which would allow readers to assess whether the budget is binding. The fact that GRPO training on HLE and BrowseComp-VL shows no clear improvement (Figure 6) could be partially attributable to the budget constraint: if the agent's exploration is limited by a fixed trajectory length, RL cannot discover strategies that require more steps than the SFT trajectories demonstrated, capping the potential gains from optimization. This interacts with the unquantified RL contribution (Limitation 5): if the budget is binding, even an improved policy would show flat reward because it cannot execute longer, more effective trajectories. The paper does not explore this interaction.

7. Implications and Future Directions

How This Work Changes the Landscape

WebWatcher forces the field to confront an uncomfortable truth that has been hiding in plain sight: the past two years of rapid progress in deep research agents—systems that autonomously search, read, and synthesize knowledge from the open web—has occurred almost entirely in a sensory-deprived environment. Text-only agents can now answer BrowseComp questions, pass sections of Humanity's Last Exam, and conduct multi-hour research sessions that rival human analysts. But the world is not made of text. Scientific diagrams, financial charts, product photographs, architectural renderings, and historical images carry information that either cannot be expressed in words or loses essential structure when flattened into captions. By demonstrating that an open-source agent can reach 27.0% on BrowseComp-VL while GPT-4o with RAG reaches only 13.4% (Table 2), the paper establishes that multimodal deep research is not a minor extension of text-only research agents—it is a qualitatively different capability that requires reconceiving how agents perceive, reason, and act.

This is not a paradigm shift on the order of the transformer or RLHF. It is better understood as a reframing that opens a new subfield: multimodal agentic research. Before WebWatcher, the default assumption in the open-source agent community was that vision could be bolted onto text-based deep research agents through image captioning, OCR, or visual question answering modules. The paper's evidence challenges this assumption directly. Figure 2 shows concretely that a vision-only agent (relying on edge detection, texture analysis, and visual search) misidentifies the animal and fails, while a search-only agent (using text queries for "penguin" or "seagull") never visits the Wikipedia revision history and fails differently. WebWatcher succeeds precisely because it interleaves visual and textual reasoning, switching modalities and tools dynamically based on the state of its investigation. The implication is that multimodal research agents cannot be built by composing existing unimodal systems; they require integrated architectures, integrated training data, and integrated evaluation.

The paper also resolves a latent tension in the literature between two competing narratives about agent capabilities. On one side, the success of text-based deep research agents (WebDancer, WebThinker, WebSailor, R1-Searcher) has created an impression that the core challenges of autonomous research are largely solved—browsing, retrieval, synthesis, and verification—with remaining work focused on scaling and refinement. On the other side, the visual reasoning community has documented persistent failures of multimodal models on tasks requiring cross-modal inference, pointing to fundamental limitations in how vision-language models integrate information across modalities. WebWatcher's results on BrowseComp-VL (27.0% state-of-the-art, meaning 73% failure rate) and HLE (13.6% vs. 16.0% for o4-mini) demonstrate that both narratives are partially correct: when given tools and trained to use them adaptively, multimodal agents can dramatically outperform static retrieval pipelines (more than doubling GPT-4o RAG on BrowseComp-VL), yet they still fail on the majority of genuinely hard multimodal reasoning tasks. The synthesis is that tool-augmented agency substantially closes the gap between vision-language models and complex real-world tasks, but the remaining gap is large and may require fundamental advances in cross-modal reasoning, not just better tool use.

Methodologically, the paper's most important contribution to the landscape is the QA-to-VQA conversion pipeline as a general mechanism for scaling multimodal reasoning data. The structural bootstrapping problem has been clear for years: manually annotating complex, multi-hop VQA data is prohibitively expensive, and prompting LLMs to generate VQA questions from images yields shallow, perception-heavy queries. The paper's inversion—start with complex textual reasoning chains from web traversal and entity obfuscation, then ground them in authentic images through entity masking—provides a template that is theoretically compatible with any text QA dataset. If validated across other QA sources (HotpotQA, StrategyQA, MuSiQue), this pipeline could shift the bottleneck from data scarcity (the current limiting factor for training multimodal agents) to the quality of image-entity grounding and the reliability of the filtering process. This is the kind of methodological infrastructure that enables a field to scale—analogous to how instruction tuning templates enabled the explosion of instruction-following models, or how RLHF pipelines standardized preference optimization.

The paper's cold-start necessity finding (Figure 6) also shifts the conversation around RL for agents. The recent excitement about pure RL for reasoning—sparked by DeepSeek-R1's demonstration that chain-of-thought behaviors can emerge from reward optimization alone—has created an implicit assumption that RL can bootstrap complex behaviors from base models. WebWatcher's negative result demonstrates a sharp boundary on this assumption: when the action space includes structured tool calls with syntactic constraints, RL from an instruction-tuned initialization fails catastrophically because the model never produces valid actions to receive reward. The additional finding that injecting chain-of-thought traces does not rescue this failure deepens the implication: reasoning traces teach thinking but not acting, and tool-use agents require demonstrations of the structured interaction format itself. This does not invalidate pure-RL approaches for reasoning, but it draws a clear line between reasoning in natural language (where RL can bootstrap) and reasoning through structured tool interactions (where it cannot). For the growing community building tool-augmented agents, this finding implies that SFT on action-grounded trajectories is not merely helpful but necessary—an architectural constraint, not an optimization choice.

Finally, the BrowseComp-VL benchmark introduces a new evaluation dimension—robustness to entity obfuscation—that existing benchmarks do not capture. The two-level structure (explicit vs. fuzzed entities) provides a diagnostic that disentangles retrieval efficiency from ambiguity resolution. The fact that GPT-4o RAG drops 9.8 points from Level 1 to Level 2 while WebWatcher-32B drops only 3.4 points (Table 2) quantifies something that has been qualitatively observed but never systematically measured: retrieval pipelines are fragile to query ambiguity in ways that agentic exploration is not. This diagnostic capability, if adopted by other benchmarks, could change how the field evaluates information-seeking systems—not just by their accuracy on well-formed queries, but by their robustness to the ill-formed, ambiguous, and context-dependent questions that characterize real research.

Follow-Up Research This Work Enables

Cheap difficulty estimation and adaptive tool-call budgeting. The paper caps all trajectories at 15 tool calls (Appendix D.2) without justification or ablation, and does not report how often this budget is binding. A natural follow-up would train a lightweight classifier—possibly a small model distilled from WebWatcher's own intermediate reasoning states—to predict, after the first 2–3 tool calls, whether the current trajectory is on track to succeed and how many additional calls are likely needed. Such a classifier could enable dynamic budget allocation: easy problems complete in 5–7 calls and hard ones get up to 20–30, rather than all problems sharing a fixed 15-call cap. The experiment would compare fixed-budget WebWatcher against an adaptive-budget variant, measuring both accuracy (does the extra budget on hard problems improve the 27.0% BrowseComp-VL score?) and efficiency (does early termination on easy problems reduce average calls per question?). The BrowseComp-VL benchmark's two-level difficulty structure provides a natural testbed: the optimal budget likely differs between Level 1 (explicit entities, retrievable) and Level 2 (obfuscated entities, requiring synthesis). If adaptive budgeting recovers a significant fraction of the 73% failure rate on BrowseComp-VL, it would demonstrate that the 15-call cap is actively suppressing WebWatcher's capability ceiling.

Error taxonomy on BrowseComp-VL to identify the primary failure modes and guide targeted improvements. The paper reports that WebWatcher-32B achieves 27.0% on BrowseComp-VL but provides zero analysis of the 73% of questions it answers incorrectly. A high-priority follow-up would manually annotate a random sample of 100 failed BrowseComp-VL questions, categorizing each failure into: (1) visual grounding failure (the agent misidentifies the entity in the image and pursues the wrong reasoning path), (2) search coverage failure (the agent correctly identifies the entity but web search fails to return pages containing the necessary information), (3) budget exhaustion (the agent's reasoning is on track but hits the 15-call limit before reaching the answer), (4) reasoning or synthesis failure (the agent gathers relevant information but draws incorrect conclusions), and (5) tool-use failure (the agent invokes the wrong tool, passes malformed arguments, or misinterprets tool outputs). The distribution of these error categories would directly inform research priorities: if 40% of failures are visual grounding errors, improving the base vision model matters most; if 30% are budget exhaustion, the 15-call cap is the bottleneck; if 25% are search coverage failures, better retrieval tools or source selection strategies are needed. This analysis would also reveal whether WebWatcher's failures are concentrated in specific BrowseComp-VL subdomains (Entertainment, Humanities, Technology, Natural Science, Other; Figure 3, Appendix B), which would indicate whether the training data distribution (5:3:2 ratio of BrowseComp-VL to long-tail to hard VQA) creates domain-specific blind spots.

Replication of the cold-start necessity result on other model families and tool suites. The finding that RL from instruction-tuned initialization fails catastrophically for tool-use tasks (Figure 6) is the paper's most consequential methodological result, but it is demonstrated on only one model family (Qwen2.5-VL) with one tool suite (five tools: Web Image Search, Web Text Search, Visit, Code Interpreter, OCR). The finding's generality is unknown. A follow-up study would replicate the cold-start vs. direct-RL comparison on at least two other vision-language model families (e.g., LLaMA-3.2-Vision, InternVL2) and with different tool configurations (e.g., adding a SQL query tool, removing the Code Interpreter, increasing to 10 tools). The key measurement is whether the Instruct initialization always stays near zero reward, or whether larger models (70B+) or models with different pretraining mixtures can occasionally produce valid tool calls early enough for RL to bootstrap. If the failure is universal, it establishes a principled boundary on pure-RL approaches for agent training and shifts research effort toward better SFT data generation rather than better RL algorithms. If some model/tool combinations succeed without cold-start, the boundary is more nuanced—perhaps dependent on whether the base model's pretraining included structured output formats, or whether the tool syntax is sufficiently natural-language-like. The experiment would also test whether the failure mode (format errors vs. semantic errors) varies across model families, which would inform mitigation strategies.

Applying the QA-to-VQA pipeline to existing text QA benchmarks to test generality. The paper claims the conversion pipeline is "compatible with most existing QA datasets" (Section 2.2.2) but validates it only on the custom-built Level 1 and Level 2 QA data. A direct test of generality would apply the pipeline to established multi-hop text QA benchmarks—HotpotQA (Wikipedia-based multi-hop), MuSiQue (multi-hop with unanswerable distractors), and StrategyQA (implicit reasoning over general knowledge)—and measure: (1) what fraction of text QA pairs survive entity filtering (i.e., reference entities with visual instantiations), (2) whether GPT-4o can correctly perform entity masking and question rewriting for diverse question types (comparison, temporal, causal), (3) whether the three-stage quality control maintains similar filtering rates across benchmarks, and (4) whether agents trained on the resulting VQA data show transfer to WebWatcher's original benchmarks. If the pipeline generalizes, it would unlock a massive source of training data for multimodal agents: the entire corpus of text QA research becomes convertible. If it fails on certain question types (e.g., questions about abstract concepts without visual grounding, or questions where entity masking introduces unresolvable ambiguity), that would define the scope of the approach and motivate alternative conversion strategies for those question categories.

SFT-only vs. SFT+GRPO ablation to quantify RL's contribution. The paper's omission of SFT-only evaluation on the main benchmarks is the single most significant gap in the experimental methodology. Figure 6 shows that on HLE and BrowseComp-VL, GRPO training oscillates without clear improvement over the SFT initialization, yet Tables 1 and 2 report only SFT+GRPO results. A direct follow-up would evaluate the SFT checkpoint (before any RL training) on all five benchmarks—BrowseComp-VL, HLE, LiveVQA, MMSearch, and SimpleVQA—and report the difference from the SFT+GRPO results. Four outcomes are possible, each with different implications: (1) If SFT-only matches or exceeds SFT+GRPO on BrowseComp-VL and HLE, then the paper's training pipeline should be simplified to SFT-only, and research attention should focus on improving trajectory quality rather than RL algorithms. (2) If SFT+GRPO substantially outperforms SFT-only on all benchmarks, the Figure 6 oscillation is misleading (perhaps the reward signal is noisy but the policy is improving along dimensions not captured by the reward curve). (3) If RL helps on LiveVQA and SimpleVQA but not BrowseComp-VL and HLE, this suggests RL's benefits are task-dependent—perhaps RL improves efficiency on tasks where SFT already provides a strong foundation, but cannot discover new strategies on tasks where even SFT performance is poor. (4) If GRPO actually hurts on some benchmarks (analogous to the ReST^EM degradation the paper references), this would indicate that the RL optimization is overfitting to the reward signal in ways that reduce generalization. Any of these outcomes would refine our understanding of when and why RL benefits tool-augmented agents, moving beyond the current "SFT + RL" recipe toward a more principled training methodology.

Human baseline on BrowseComp-VL to calibrate the difficulty scale. The paper presents 27.0% as a state-of-the-art result on BrowseComp-VL but provides no human performance reference, making the number difficult to interpret. Is BrowseComp-VL a benchmark where human experts would score 95%, revealing a massive capability gap? Or is it so difficult—with entity obfuscation, multi-page browsing requirements, and cross-modal reasoning—that even skilled human researchers would score 50–60%, meaning WebWatcher is closer to human-level than the raw numbers suggest? A human baseline study would recruit 5–10 PhD-level researchers (matching the profile of the benchmark's verifiers), give them the same tool access as WebWatcher (web search, image search, webpage browsing, code execution), and measure their accuracy and average completion time on a random subset of 50 BrowseComp-VL questions. The results would contextualize the 27.0% score: if humans achieve 80%+, the benchmark primarily measures AI limitations; if humans achieve 40–50%, the benchmark measures inherent task difficulty and WebWatcher's performance is more impressive. The human study would also reveal whether humans use fundamentally different strategies than WebWatcher—issuing more diverse search queries, spending longer on individual pages, or using tools in sequences the agent never discovers—which would inform trajectory data generation improvements.

Practical Applications and Downstream Use Cases

Cost-efficient batch processing for multimodal knowledge base construction. Organizations maintaining knowledge bases that integrate visual and textual information—museums cataloging artifacts with provenance research, pharmaceutical companies linking compound structures to research literature, legal firms connecting case documents to evidentiary photographs—could deploy WebWatcher-style agents for batch enrichment. The paper's data generation pipeline (180,000 raw QA pairs from web traversal, Section 2) demonstrates the feasibility of automated, large-scale multimodal data creation. The practical value is not in answering user questions interactively but in populating structured databases where each entry requires: identifying entities in images, searching for related documentation, extracting specific properties from web sources, and cross-validating across multiple references. At WebWatcher-32B's 27.0% accuracy on BrowseComp-VL, fully automated population would require human verification of ~73% of entries, but the pass@k scaling (Figure 7) offers a practical tradeoff: generating k=8 independent answers per entry and accepting the answer only when a supermajority agrees could yield high precision on the subset that passes consensus, with the remainder flagged for human review. This transforms the agent from a standalone answerer to a triage system that automates easy and medium cases while routing hard cases to humans—a deployment model that is viable at current accuracy levels.

Multimodal research assistance for academic literature review. The BrowseComp-VL benchmark's domain distribution (Natural and Formal Sciences, Engineering and Computer Science, Social Sciences and Humanities, Arts and Entertainment; Appendix B) maps directly onto academic disciplines where researchers routinely need to answer questions that cross visual and textual boundaries: "What was the sample size in the experiment shown in Figure 3 of this paper?" "Which architectural style does the building in this photograph exemplify, and what are three other examples from the same period?" "Compare the electrochemical performance of the catalyst in this SEM image to the state-of-the-art reported in the literature." Current tools force researchers to manually split these tasks—reverse image search to identify the figure, then text search to find related papers, then manual comparison of reported numbers. A deployed WebWatcher variant, fine-tuned on domain-specific literature (arXiv papers for CS, PubMed for biomedicine) rather than general web content, could automate the cross-modal, multi-source synthesis step. The key adaptation would be replacing the general Web Text Search and Visit tools with domain-specific search APIs (Semantic Scholar, PubMed, arXiv API) and adding a tool for extracting structured data (tables, charts) from PDFs. The paper's finding that WebWatcher-32B achieves 33.8% on HLE Biology questions (Table 1)—its strongest subfield—suggests that domain-specific fine-tuning on scientific VQA could push accuracy high enough for assistance (not replacement) in literature review workflows, where the cost of a missed relevant paper is high but the cost of a false positive is low (the researcher skims and dismisses).

Visual fact-checking and claim verification for journalism and intelligence analysis. BrowseComp-VL's Level 2 questions—where entities and attributes are deliberately obfuscated—are structurally similar to real-world verification tasks: a journalist receives a photograph claiming to show "a rare phenomenon observed in early spring of a recent even-numbered year" and must determine whether the image is authentic, when and where it was taken, and whether the accompanying textual claims are consistent with independent sources. WebWatcher's demonstrated robustness to entity obfuscation (28.4% Level 1 to 25.0% Level 2, a 3.4-point drop, vs. GPT-4o RAG's 16.8% to 7.0%, a 9.8-point drop; Table 2) directly translates to this use case: the agent is less fragile to vague or incomplete initial descriptions than retrieval pipelines. A practical deployment would integrate WebWatcher into existing verification workflows (such as those used by fact-checking organizations or open-source intelligence analysts), where the agent serves as a first-pass filter: given an image and an associated claim, it autonomously searches for corroborating or contradicting evidence, visits relevant pages, and produces a structured summary with source URLs. The current 27.0% accuracy on BrowseComp-VL means the agent would correctly verify or debunk slightly more than one-quarter of claims without human intervention, with the rest requiring analyst review. The pass@k scaling (41.9% on HLE at k=32) suggests that running multiple independent investigations in parallel and surfacing the consensus answer, with disagreement triggering escalation, could push the autonomous resolution rate higher. The critical deployment challenge is not accuracy alone but calibrated uncertainty: the agent must know when to escalate, and the Figure 6 GRPO oscillation on BrowseComp-VL and HLE suggests that current training does not produce well-calibrated confidence. Adding an explicit confidence estimation module—trained on whether the agent's eventual answer matches ground truth given intermediate features—would be a necessary engineering step.