Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos

ArXiv: 2603.22529

🎯 Pitch

Current web agents that must act on what a user sees in the real world (e.g., buying a snack spotted in an egocentric video) still fail roughly 40% of the time compared to an oracle. Ego2Web reveals this gap with the first benchmark pairing first-person video tasks with live web execution, and introduces an automatic judge that matches human ratings 84% of the time.

1. Executive Summary

This paper introduces Ego2Web, the first benchmark that bridges egocentric video perception and web-agent task execution, featuring 500 human-verified video–instruction pairs spanning e-commerce, media retrieval, knowledge lookup, and local/maps domains. To enable scalable evaluation, the authors propose Ego2WebJudge, a multimodal LLM-as-a-Judge framework that incorporates grounded visual cues from egocentric videos and achieves approximately 84% agreement with human judgment, substantially outperforming prior automatic evaluators like WebVoyager and WebJudge. Experiments with six state-of-the-art web agents—including Claude Computer-Use, GPT-5.4, and Browser-Use with Gemini-3-Flash—reveal a roughly 40% performance gap from oracle performance, with even the strongest agent (BU-Gemini-3-Flash) reaching only 58.6% success rate under human evaluation, establishing that current agents struggle with the visual grounding, temporal reasoning, and perception–action alignment required when web tasks depend on real-world egocentric video evidence.

2. Context and Motivation

The Core Problem: Web Agents Are Blind to the Physical World

The fundamental gap this paper addresses is deceptively simple: web agents today cannot see what their users see. When a person wants to buy a product they spotted in a store, find information about a landmark they're standing in front of, or look up a recipe for a dish they saw someone preparing, they naturally connect visual perception from the physical world to digital action — they see something, recognize it, and then act online. Current web agents cannot do this. They operate entirely within the digital realm, taking text instructions or screenshots as input, with no awareness of the user's physical surroundings.

This gap is not merely an inconvenience. As multimodal assistants become embedded in everyday devices — AR glasses, wearable cameras, smart home robots — users will increasingly expect agents to handle tasks that naturally span both physical and digital contexts. The paper gives concrete examples (Section 1): identifying a medication label before scheduling a delivery online, or recognizing an object in one's environment and purchasing it. These are tasks humans perform seamlessly but that no existing benchmark can evaluate because no benchmark connects egocentric vision to web execution.

The paper frames this as a missing link between real-world visual perception and online action execution (Section 1). Prior work has addressed the two halves independently — egocentric video understanding on one side, web agent capabilities on the other — but the connection point where an agent must perceive something in the physical world and use that perception to guide digital actions has been entirely unexplored.

Why This Matters: The Trajectory Toward Embodied Assistants

The paper's motivation is grounded in a clear technological trajectory. The rapid advancement of Multimodal Large Language Models (MLLMs) and web-based agents — the paper cites OpenAI Operator (OpenAI, 2025), Claude Computer-Use (Anthropic, 2025), and Google Project Astra — has made autonomous web execution increasingly viable. However, these systems share a critical architectural limitation: they perceive only digital content. As the paper puts it:

"unlike humans who can seamlessly perceive the physical world, reason over what they see, and act across both physical and digital contexts, current web agents remain confined to purely digital perception. They operate based only on screenshots or text instructions, lacking grounding in the user's surrounding physical world."

This limitation has concrete practical consequences for several reasons:

1. Real-world deployment scenarios inherently span physical and digital domains. The paper describes a user wearing AR glasses who sees an object and wants to buy it online, or who needs to verify a product label before completing a purchase — these are not edge cases but the default use case for assistive agents embedded in wearable devices. Without egocentric perception, agents cannot serve these scenarios at all.

2. Existing benchmarks cannot evaluate the capability that matters most. Because prior web-agent benchmarks (WebArena, VisualWebArena, WebVoyager, etc.) provide only on-screen information, they cannot measure whether an agent can correctly ground its actions in physical-world evidence. The paper argues that this creates a dangerous evaluation blind spot: an agent might excel on existing benchmarks by being good at web navigation while fundamentally lacking the perceptual grounding that real-world deployment demands.

3. The field lacks a testbed for perception–action integration. Even if researchers wanted to build agents that connect egocentric vision to web execution, there has been no standard way to measure progress. Without a benchmark, the capability remains undefined, unmeasured, and unimproved. Ego2Web fills this role by defining the task, providing the data, and establishing evaluation protocols.

Where Prior Work Falls Short: Two Separate Literatures, No Connection

The paper identifies two well-developed but entirely disconnected research areas, neither of which addresses the gap.

The Egocentric Video Understanding Literature: Perception Without Action

Egocentric video understanding has a rich history of datasets and benchmarks, including Ego4D (Grauman et al., 2022, over 3,600 hours of first-person video), EPIC-KITCHENS (Damen et al., 2018), and EgoExo4D (Grauman et al., 2024). These have driven progress on tasks like action recognition, object tracking, temporal segmentation, and video-language reasoning. More recent benchmarks like EgoSchema (Mangalam et al., 2023), EgoThink (Cheng et al., 2024), and EgoPlan (Chen et al., 2023) pushed toward higher-level reasoning — inferring intentions, temporal order, and causal relations from first-person videos.

However, as the paper explicitly notes (Section 2.1):

"all existing egocentric video benchmarks evaluate perception and reasoning in isolation, without connecting real-world video understanding to online web decision-making."

The shortcoming is not in the quality of these benchmarks — they are rigorous and have driven meaningful progress in video understanding — but in their scope. They ask questions like "what is the person doing?" or "what object did they pick up second?" but never "given what you observed, go complete a related task on the web." The evaluation stops at answer production, never requiring the model to translate perception into action in a live digital environment.

The Web Agent Benchmark Literature: Action Without Real-World Perception

On the other side, web-agent benchmarks have evolved from static, sandboxed environments to fully interactive online evaluation, but remain purely digital in their inputs.

Early sandboxed benchmarks. WebArena (Zhou et al., 2023) and Mind2Web (Deng et al., 2023) provided controlled environments for web navigation, with agents operating on DOM trees and rendered pages in simulated settings. These offered reproducibility but lacked the messiness and diversity of real websites.

Visual web benchmarks. VisualWebArena (Koh et al., 2024) and VideoWebArena (Jang et al., 2024) incorporated richer visual inputs — screenshots and video recordings of web interactions — to test visual reasoning during navigation. OSWorld (Xie et al., 2024) extended to open-ended tasks in real computer environments. These represent important progress in making web agents visually aware, but critically, the visual content is web content (pages, UI elements, screen recordings), not physical-world content.

Online evaluation benchmarks. WebVoyager (He et al., 2024), Online-Mind2Web (Xue et al., 2025), and WebCanvas (Pan et al., 2024b) moved evaluation to live, real-world websites via LLM-as-a-Judge approaches, emphasizing realism and scalability. WebJudge (Xue et al., 2025) specifically improved automatic evaluation with better trajectory understanding. These are the most advanced benchmarks in this lineage, but they share the same fundamental limitation: all inputs are digital. The agent receives text instructions and screenshots, never egocentric video or images of the user's physical environment.

The missing dimension. As the paper's comparison table (Table 1) makes clear, no existing benchmark combines egocentric video perception, web task execution, and online evaluation in a single setting. EgoThink has egocentric perception but no web tasks. VisualWebArena has web tasks and visual perception but the visuals are web pages, not real-world scenes. WebVoyager has web tasks and online evaluation but no egocentric perception. The gap is at the intersection — the ✓, ✓, ✓, ✓ column in Table 1 exists only for Ego2Web.

How Prior Approaches Fall Short in Detail

The paper identifies specific limitations that prevent existing benchmarks from evaluating the full capability chain that a real-world assistant requires:

1. Text-only or screenshot-only input provides no physical grounding. WebArena, Mind2Web, and OSWorld give agents text instructions and/or screenshots of web pages. This tests web navigation competence but cannot test whether the agent correctly perceived a physical object in the user's environment. An agent that navigates Amazon perfectly but misidentifies the product the user is looking at would score well on these benchmarks despite failing in a real deployment scenario.

2. Sandboxed environments lack real-world web complexity. Static environments like WebArena use simulated websites that don't change. Real websites have A/B testing, dynamic layouts, CAPTCHAs, login walls, and constantly evolving interfaces. The paper chooses live online evaluation specifically because sandboxed settings "cannot capture the dynamic, unpredictable nature of real web interactions" that agents would encounter when deployed (implicit in Section 2.2).

3. Prior visual web benchmarks confuse "visual" with "web-visual." VisualWebArena and VideoWebArena added visual inputs, but these are screenshots of web pages and video recordings of browser interactions, not egocentric video of the user's environment. The perceptual challenge is entirely different: recognizing UI elements versus recognizing real-world objects, tracking interface state changes versus tracking physical object interactions over time. The skills tested are orthogonal to the skills needed for egocentric grounding.

4. LLM-as-a-Judge evaluation methods are blind to visual evidence. Even the most advanced automatic evaluators like WebJudge (Xue et al., 2025) assess agent performance based on action trajectories, screenshots, and final outputs — all textual or web-screenshot data. They have no mechanism to verify that the agent's web result actually matches the visual evidence from the user's environment. As the paper demonstrates (Table 4), these evaluators achieve only 70-78% agreement with human judgment on egocentric tasks, compared to Ego2WebJudge's 84%, precisely because they cannot check visual grounding.

How This Paper Positions Itself

Ego2Web positions itself not as an incremental improvement to either the egocentric video understanding or web-agent literature, but as a new category of benchmark that creates a previously nonexistent intersection. The paper's Table 1 is the clearest statement of this positioning: it shows four columns (Visual Grounding, Egocentric Video Perception, Web Task, Online Evaluation) and makes explicit that Ego2Web is the only benchmark with checks in all four.

The paper does not claim to introduce new agent architectures, new training methods, or new models. Its contribution is entirely infrastructural: defining a task that didn't exist, building a dataset and evaluation framework for it, and using both to reveal concrete, measurable limitations in current state-of-the-art agents. The framing is explicitly that of a catalyst — providing "a critical new resource for developing truly capable AI assistants that can seamlessly see, understand, and act across the physical and digital worlds" (Section 1, closing paragraph).

The paper also positions its evaluation methodology as a contribution in its own right. The existing LLM-as-a-Judge paradigm for web agents (WebVoyager, WebJudge) was not designed for tasks with external visual grounding. The paper argues that naive application of these evaluators to Ego2Web tasks would produce unreliable results because the judge cannot see the egocentric video evidence that determines correctness. Ego2WebJudge is presented as a necessary methodological innovation — extending the LLM-as-a-Judge framework to incorporate visual evidence from the user's environment, thereby making automatic evaluation feasible and reliable for this new task class.

Finally, the paper positions its findings as a diagnosis of current agent capabilities. The experiments (Section 5) are not designed to showcase a new system's strengths but to reveal where existing systems fail. The roughly 40% gap from oracle performance, the domain-specific breakdown showing Knowledge Lookup as dramatically easier than E-Commerce, and the ablation showing that raw video input more than doubles performance over caption-only input — all of these serve to characterize the specific challenges of egocentric grounding for web agents, providing a roadmap for future work rather than claiming to have solved the problem.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

This paper is not building a new agent or training a new model; it is an infrastructure paper that constructs a benchmark — a standardized test — for evaluating multimodal AI agents on a previously nonexistent task. The core idea is to define, operationalize, and provide tooling for the problem of egocentric video-grounded web task execution: given a first-person video of what a user sees in the physical world, can an AI agent perceive the relevant objects or events, reason about them, and then take appropriate actions on live websites (searching, purchasing, retrieving information) to accomplish a goal? The "shape" of the solution is a dataset of 500 video–task pairs, a semi-automatic pipeline for creating more, and an automatic evaluation framework that can judge success reliably without requiring slow and expensive human annotation.

3.2 Big-Picture Architecture (Diagram in Words)

The Ego2Web system has three major components that operate sequentially:

1. The Data Generation Pipeline (Section 3.2): A semi-automatic LLM-plus-human workflow that takes raw egocentric videos from existing datasets, converts them into structured textual descriptions, prompts a large language model to invent web tasks that require visual information from those descriptions, and then has human annotators verify and refine the generated tasks for quality. This produces the 500 benchmark examples.

2. The Task Execution Environment: The live, unconstrained web, where agents (SeeAct, Browser-Use with various backends, Claude Computer-Use, GPT-5.4) receive an egocentric video and a task instruction, and must produce a sequence of web actions (navigating, clicking, scrolling, typing) in a real browser against real websites. Agents are not given ground-truth visual annotations; they must extract all necessary perceptual information from the raw video themselves.

3. The Ego2WebJudge Evaluation Framework (Section 4): An automatic evaluation pipeline that takes the task instruction, annotated visual evidence from the egocentric video, the agent's action history, and selected screenshots, and produces a binary success/failure judgment. This replaces human evaluation for scalable benchmarking. The framework has three internal stages: key-point extraction from the instruction, key-screenshot selection from the agent's trajectory, and final multimodal judgment that checks both web-task completion and visual grounding.

Information flows through these components as follows: raw egocentric videos enter the data generation pipeline and emerge as verified video–task pairs → these pairs are fed to the agent under evaluation, which produces action trajectories and screenshots → the trajectory, screenshots, and annotated visual evidence enter Ego2WebJudge, which outputs a success/failure binary verdict. The entire pipeline is designed so that, once the benchmark is built, evaluating a new agent requires only running the agent and then running the judge — no additional human annotation.

3.3 Roadmap for the Deep Dive

• First, the task definition and formal framing (Section 3.1), because everything else — what data is collected, how it is generated, how agents are evaluated — follows from what "success" means in this setting. The formalization pins down exactly what inputs the agent receives, what outputs it must produce, and what criteria determine correctness.

• Second, the semi-automatic data generation pipeline (Section 3.2), since the benchmark instances are the foundation on which everything else rests. We walk through the three-stage process — video parsing into structured captions, LLM-based task instruction generation, and human verification — explaining each stage's inputs, outputs, design choices, and quality control mechanisms.

• Third, the Ego2WebJudge evaluation framework (Section 4), because evaluating agents on this task is nontrivial: the judge must check not only whether the web task was completed but whether the web result matches the visual evidence in the video. We decompose Ego2WebJudge into its three internal stages, explaining how key-point extraction, key-screenshot selection, and multimodal judgment work together to produce reliable binary verdicts that achieve ~84% agreement with human evaluators.

• Fourth, the experimental setup details (Section 5.1) that are implicitly part of the technical approach: what agent configurations are tested, how videos are converted when agents cannot process raw video, and how human evaluation is conducted to establish ground-truth accuracy for calibration.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark construction and evaluation methodology paper. The technical contributions are not a new model architecture or training algorithm but rather (1) a task definition that bridges two previously disconnected research areas, (2) a data generation pipeline that efficiently creates high-quality video–task pairs, and (3) an evaluation framework that makes automatic, visually-grounded judgment reliable for this new task class. What follows is an exhaustive walkthrough of each mechanism.

Task Definition and Formal Framing (Section 3.1)

The paper defines Ego2Web's task as a mapping from multimodal input to a sequence of web actions with a verifiable goal state. Formally, the agent receives two inputs: an egocentric video and a task instruction, and must produce a sequence of web actions in a browser environment.

Input specification. Let the egocentric video be defined as:

$V = \{f_1, f_2, \ldots, f_t\}$

where each $f_i$ is an individual frame captured from the user's first-person perspective, and $t$ is the total number of frames (variable across videos, but most videos are approximately three minutes long, following the EgoSchema selection criteria described in Appendix A).

The task instruction is a natural-language string $I$ that describes what the agent must accomplish on the web using visual information from $V$. For example: "On Amazon, find the same snack as the fourth one picked up by the person in the video. And report the Calories per serving."

What it captures. The video $V$ provides egocentric visual evidence — the raw perceptual data that a human user would see through AR glasses or a wearable camera. The instruction $I$ specifies what web action to take and what visual information from the video is relevant. Critically, $I$ is designed (through the data generation pipeline) such that it cannot be fulfilled without visual perception of $V$; an agent that ignores the video and guesses based on the instruction text alone would almost certainly fail (as confirmed by the ablation in Table 5, where the no-visual-input condition achieves only 4.4% success rate).

Output specification. The agent must produce a sequence of web actions:

$A = \{a_1, a_2, \ldots, a_n\}$

where each $a_i$ is an atomic browser operation (e.g., navigate(url), click(element), input_text(field, text), scroll(direction)). The actions are executed sequentially in a real browser environment $E$ against live websites. The paper sets a maximum of 40 actions per episode (Section 5.1, Implementation Details).

Goal state and success criterion. The task is considered successful if and only if the final web state matches the goal state $G$. However, $G$ is not provided to the agent — it is implicitly defined by the task instruction $I$ and verified by either human annotators or Ego2WebJudge after the episode completes. The paper formalizes this verification as:

$O = \text{Ego2WebJudge}(I, v, A, S)$

where $O \in \{\text{Success}, \text{Failure}\}$ is the binary outcome, $v$ is the annotated egocentric visual evidence clip (a subset of $V$ containing the frames necessary for task solving), and $S = \{s_1, s_2, \ldots, s_k\}$ is the sequence of browser screenshots captured during the agent's actions.

What this equation computes. The judge takes four pieces of evidence — the instruction (what was asked), the visual ground truth (what the video actually shows), the agent's actions (what the agent did), and the resulting screenshots (what the agent saw and produced on the web) — and produces a single binary verdict: did the agent succeed or fail? The judge must check two things simultaneously: (1) that the web task was completed (e.g., the correct product page was reached, the required information was retrieved), and (2) that the web result matches the visual evidence from the egocentric video (e.g., the product on the webpage is indeed the same object that appears in the video).

Why this form. The inclusion of $v$ — the annotated visual evidence — is what distinguishes this evaluation from prior web-agent evaluators. In WebVoyager or WebJudge, the judge only sees $I$, $A$, and $S$, so it can check whether the web task was completed but cannot check whether the result is grounded in the user's physical environment. A naive judge might mark a task as successful because the agent reached a product page for "Trail Mix" when the instruction mentioned "Trail Mix," even if the video actually showed a different brand or flavor. The visual evidence $v$ closes this loophole, making the judge's verdict contingent on correct visual grounding.

The two capability dimensions tested. The paper explicitly decomposes the task into two required competencies (Section 3.1):

1. Visual Perceptual Understanding and Grounding: The agent must extract task-relevant semantic and visual information from the egocentric video — object categories, brands, colors, shapes, spatial relationships, temporal sequences of actions. This is not a multiple-choice recognition task but an open-ended extraction problem: the agent does not know in advance which visual details matter; it must infer from the instruction what to look for and then locate it in unstructured, often cluttered first-person footage.

2. Web Execution Reasoning: Given the visual information extracted from the video, the agent must plan and execute a sequence of web actions — navigating to the appropriate website, formulating search queries, scrolling through results, clicking on correct items, and extracting specific information (prices, nutritional facts, video titles, addresses, etc.). This requires understanding web interface conventions, handling dynamic page loads, and recovering from navigation errors, all within the 40-step action budget.

The implicit three-stage cognitive pipeline. While the paper does not enforce a specific agent architecture, the task definition implies a three-stage information-processing pipeline that any successful agent must implement (whether explicitly or implicitly): (1) perceive and ground visual cues in the video, (2) map those cues to symbolic web queries or navigation targets, and (3) execute the web interaction and verify that the result matches the original visual evidence. Failures can occur at any stage — misperceiving the video, failing to translate perception into an effective web search, or executing web actions incorrectly — and the benchmark's evaluation does not distinguish between these failure modes at the metric level (though the error analysis in Section 5.3 does break them down qualitatively).

Semi-Automatic Data Generation Pipeline (Section 3.2)

The core engineering contribution of the paper is the pipeline that produces the 500 benchmark instances. Building this pipeline required solving a chicken-and-egg problem: the benchmark needs video–task pairs where the task depends on visual information in the video, but manually creating such pairs by watching hundreds of hours of egocentric footage and inventing web tasks would be prohibitively expensive. The paper's solution is a three-stage model–human collaborative process that uses MLLMs and LLMs to generate candidate tasks at scale, then relies on human annotators for verification and refinement.

Stage 1: Egocentric Video Pool and Visual Parsing

Source material. The pipeline begins with a curated pool of egocentric videos sourced from Ego4D (Grauman et al., 2022), a large-scale dataset containing over 3,600 hours of first-person video with dense, timestamped narrations. The paper does not use Ego4D videos directly in their raw form; instead, it adopts the selection and trimming procedure from EgoSchema (Mangalam et al., 2023). EgoSchema filters Ego4D for non-overlapping three-minute clips containing at least 30 human-annotated narrations. This produces segments with the following desirable properties (explained in Appendix A):

• Consistent temporal structure: All clips are approximately three minutes, providing a standardized input length that avoids the confounding variable of video duration when comparing agent performance.
• High semantic density: The ≥30 narration threshold ensures each clip contains substantial human activity, object interactions, and scene changes, providing rich visual material for task generation.
• Reliable supervision: The human narrations provide timestamped ground-truth descriptions of what the camera wearer is doing, which is useful for quality control even though these narrations are not directly used in the benchmark tasks.

Visual captioning. Each selected video is processed by a frozen MLLM — specifically Qwen3-VL-7B (Team, 2025) — to extract structured, detailed visual metadata. The captioning is performed clip-by-clip, with each clip corresponding to a 5-second segment of the video. For each 5-second clip, the MLLM produces a structured description following a predefined JSON schema (the exact prompt is provided in Appendix B):

{
  "video description": "a detailed description of the video content",
  "objects": {
    "object_name_1": "detailed description including attributes",
    "object_name_2": "..."
  }
}

The "video description" field captures the overall scene context, actions, and temporal progression within that 5-second window (e.g., "The camera pans left to right across a glass-door refrigerator filled with various bottled and canned beverages"). The "objects" field provides a dictionary of notable objects with distinguishing attributes — color, shape, brand, state, and physical interactions (e.g., "blue can of Nos energy drink," "red shopping basket").

From clip captions to video profile. The per-clip captions are concatenated into a single video profile $V_{\text{meta}}$:

$V_{\text{meta}} = \{v_1^{\text{meta}}, v_2^{\text{meta}}, \ldots, v_k^{\text{meta}}\}$

where each $v_i^{\text{meta}}$ is the structured metadata for the $i$-th 5-second clip, and $k = \lceil \text{video\_duration} / 5 \rceil$ (for a three-minute video, $k \approx 36$). This video profile serves as a textual summary of the entire visual world observed by the camera wearer — a structured, machine-readable representation that captures what objects appear, what actions occur, and when they happen.

Why captions instead of raw video for task generation. The task generation step (Stage 2) uses a text-only LLM (GPT-5), which cannot process video directly. The captions act as a lossy but structured compression of the visual content, preserving object identities, attributes, and temporal order while discarding fine-grained visual details (exact pixel colors, precise spatial layouts, subtle textures). This is a deliberate trade-off: the captions need only be descriptive enough for the LLM to invent plausible task instructions; the actual benchmark tasks will be solved by agents that receive the raw video, so the captions' imperfections do not propagate into the final task quality (human verification in Stage 3 catches any cases where the caption was misleading or insufficient).

Design choice: Qwen3-VL-7B over other MLLMs. The paper uses Qwen3-VL-7B, a relatively small (7B parameter) open-weight model, for captioning. This choice reflects a practical consideration: captioning hundreds of videos at 5-second granularity is computationally intensive, and a smaller model can process the full dataset within a reasonable budget. The paper does not ablate the choice of captioning model, but the human verification stage provides a safety net — if Qwen3-VL-7B produces inaccurate captions for some clips, the resulting task would likely fail the human quality checks and be discarded or corrected.

Stage 2: LLM-Based Automatic Task Instruction Generation

The second stage takes the video profile $V_{\text{meta}}$ and a pool of predefined websites as input, and uses a text-only LLM (GPT-5) to generate task instructions that require visual information from the video.

Inputs to the LLM. The LLM receives two structured inputs concatenated into its prompt (the full prompt is provided in Appendix B):

1. The video profile: The complete $V_{\text{meta}}$ — all clip-level captions arranged chronologically, giving the LLM a textual "view" of the entire three-minute video. The prompt presents this as "Detailed Video Description (every 5 seconds): [Video Metainfo]".

2. The website pool: A predefined list of popular, actively maintained websites selected by the human annotators as appropriate targets for web tasks. The list includes 40 domains spanning e-commerce (amazon.com, ebay.com, walmart.com, etsy.com, ikea.com, nike.com, adidas.com, apple.com, bestbuy.com, target.com), media (youtube.com, vimeo.com, imdb.com, tiktok.com, bilibili.com), knowledge bases (wikipedia.org, stackexchange.com, quora.com), local/maps services (google.com/maps, tripadvisor.com, booking.com, yelp.com), and other categories (reddit.com, linkedin.com, nytimes.com, etc.).

The three-step generation procedure. The LLM is prompted to perform a structured reasoning-and-generation process in three sequential steps (explicitly enumerated in the prompt):

Step 1: Suitability Assessment. The LLM first determines whether the video is suitable for creating a video-grounded web task. The prompt specifies concrete criteria for suitability:

• The video must contain objects, activities, or scenes that naturally link to a web-based information or action need.
• The task must require visual information from the video (i.e., it cannot be solved from the caption text alone — this is a subtle point: the LLM must imagine a task where the caption describes the video content, but the task-solver would need to actually see the video, not just read the caption, to succeed).
• Visual cues must be clear and distinctive (e.g., a specific brand logo, a unique product shape, a recognizable landmark).

If the video is deemed unsuitable, the LLM outputs "suitable": false with a brief explanation, and no tasks are generated. This gate prevents the pipeline from producing tasks for videos where the visual content is too ambiguous, cluttered, or mundane to support meaningful web tasks.

Step 2: Visual Anchor Extraction. For suitable videos, the LLM identifies 1–2 visual anchors — specific objects, scenes, or events in the video that can serve as the perceptual basis for a web task. For each anchor, the LLM must specify:

• A name guess (what the anchor probably is, e.g., "a blue can of Red Bull").
• 3–5 visual cues that distinguish this anchor (e.g., "blue and silver can," "bull logo," "tall slim shape," "energy drink branding").
• Timestamps indicating when the anchor appears in the video (e.g., "00:55-01:00").
• Why the task is video-dependent — a justification that the task cannot be solved from captions alone, which forces the LLM to verify that the anchor involves visual details (color, shape, spatial layout, fine-grained text) that would not be fully captured in the textual description.

Step 3: Task Instruction Generation. For one selected anchor, the LLM generates 1–3 task instructions at different difficulty levels. Each instruction must satisfy specific constraints:

• Must depend on ≥2 visual cues from the anchor, ensuring that the task cannot be trivially solved by guessing a single attribute.
• Must include a "Must Match" list — an explicit enumeration of the visual cues that the correct web result must align with (e.g., ["blue color", "bull logo", "energy drink category"]). This list serves as an internal consistency check for the LLM and later as a reference for human verifiers.
• Must be phrased as a natural web instruction — clear, grammatically correct, and specifying an achievable goal (e.g., "On Amazon, find the same snack as the fourth one picked up by the person in the video. And report the Calories per serving").
• Must specify allowed domains from the website pool — typically 2–3 relevant websites (e.g., for a shopping task: ["amazon.com", "walmart.com"]).
• Must include a "why video dependent" justification — confirming that the task genuinely requires visual perception.

The LLM is also instructed to produce tasks at different difficulty levels (the output JSON schema includes a "difficulty" integer field), though the paper does not extensively use these difficulty labels in the final benchmark analysis, instead relying on the domain-based categorization.

Output format. The LLM's output is a strictly validated JSON object following a schema that includes the suitability flag, the list of generated tasks (each with difficulty, instruction, must_match list, timestamps, allowed domains, and why_video_dependent), and, if unsuitable, a reason string. The prompt explicitly instructs: "Ensure the output is strictly valid JSON with no extra text."

Why GPT-5 for task generation. The paper uses GPT-5 because the task generation step requires creative, open-ended reasoning: inventing plausible and interesting web tasks from a textual description of a video. This is fundamentally a language generation and planning problem, not a visual perception problem. GPT-5, as a state-of-the-art text LLM, is well-suited to generating diverse, coherent instructions that satisfy the specified constraints. The choice to separate visual captioning (done by Qwen3-VL-7B, a multimodal model) from task generation (done by GPT-5, a text model) reflects a pragmatic division of labor: the visual model handles perception, the text model handles creative synthesis.

Design choice: why not end-to-end with a single MLLM? A single MLLM could theoretically take the video directly and generate tasks end-to-end, but this would require a model with both strong video understanding and strong instruction-generation capabilities, and would likely be more expensive (processing full video tokens rather than text captions). The two-stage approach (video → captions, captions → tasks) is more modular, cheaper, and allows using the best available model for each subtask. The human verification stage ensures that errors in the captioning stage do not produce low-quality tasks — if the captions misrepresent the video, the generated task will likely fail the visual grounding check during human verification.

Stage 3: Human Verification and Refinement

The final stage involves human annotators reviewing each generated video–task pair and either accepting, editing, or rejecting it based on three explicit quality criteria.

The three verification criteria (Section 3.2):

1. Visual Grounding: The annotator watches the egocentric video and confirms that the task depends on information that is actually visible in the video. If the LLM-generated task references an object or attribute that does not appear in the footage, or that is too ambiguous to reliably identify, the annotator either corrects the task instruction or rejects the example. This is the most critical quality check, as it ensures the benchmark genuinely tests visual perception rather than linguistic guesswork.

2. Web Feasibility: The annotator verifies that the task can realistically be executed on the specified websites. This includes checking that the target website is still active and functional, that the required information (product details, video metadata, map locations) is likely to be findable through reasonable search and navigation, and that the task does not require actions that are technically impossible (e.g., purchasing an out-of-stock item, accessing geo-restricted content). This criterion prevents tasks that are theoretically video-grounded but practically unsolvable due to web constraints.

3. Instruction Quality: The annotator ensures the instruction is grammatically correct, unambiguous, and clearly specifies what the agent must accomplish. Vague or contradictory instructions are edited for clarity. This criterion ensures that agent failures on the benchmark are due to genuine capability limitations, not due to incomprehensible task descriptions.

The refinement process. When annotators identify issues, they edit the instruction or visual cue annotations rather than simply rejecting the example (unless the issues are unfixable, e.g., the video contains no distinguishable visual content). This hybrid model–human pipeline is designed to maximize the yield of high-quality examples: the LLM generates a large number of candidate tasks quickly and cheaply, and human annotators focus their effort on verification and refinement rather than creation from scratch. The paper reports that this process yields 500 verified video–task pairs from the initial pool of egocentric videos (the exact yield rate — what fraction of generated tasks pass verification — is not explicitly stated).

Why human verification is necessary rather than purely automatic filtering. The LLM-generated tasks are based on textual captions, not the raw video. Captions can be inaccurate, incomplete, or misleading in ways that only a human watching the actual video can detect. For example, a caption might describe "a person picks up a snack" without specifying the brand or flavor, and the LLM might generate a task requiring the agent to identify that brand — which may or may not be visually distinguishable in the video. Only a human can verify whether the visual information needed for the task is actually present and recognizable. The paper's use of human verification is thus not a weakness but a deliberate design choice to ensure benchmark quality, trading annotation cost for reliability.

Final benchmark composition. The 500 verified pairs are categorized into five task types based on the primary web domain involved (Section 5, Figure 4a): E-Commerce (50.3% of tasks, 230 examples), Media Retrieval (24.1%, 132 examples), Knowledge Lookup (17.0%, 92 examples), Local/Maps (6.0%, 31 examples), and Others (2.6%, 15 examples). The distribution across specific websites (Figure 4b) shows Amazon as the most frequent domain (approximately 175 tasks), followed by YouTube, Wikipedia, Walmart, eBay, Google Maps, and a long tail of less common sites. This distribution is intentionally realistic — reflecting the domains where users most commonly perform web tasks — rather than artificially balanced across websites.

Ego2WebJudge: Automatic Online Evaluation Grounded in Visual Cues (Section 4)

The second major technical contribution is the automatic evaluation framework that enables scalable, reproducible assessment of agent performance without requiring human annotators for every evaluation run. Prior work on LLM-as-a-Judge for web agents (WebVoyager, WebJudge) provided the foundational paradigm, but these evaluators were designed for tasks where all relevant information is contained in the text instruction, action trajectory, and web screenshots. Ego2WebJudge extends this paradigm to incorporate visual grounding — the judge must check not only whether the web task was completed but whether the web result matches the visual evidence in the egocentric video.

The Core Evaluation Function

Ego2WebJudge is formalized as a function that maps its inputs to a binary verdict (Section 4):

$O = \text{Ego2WebJudge}(I, v, A, S)$

where:

• $I$ is the task instruction (text).
• $v$ is the annotated egocentric visual evidence clip — the specific temporal segments (and corresponding keyframes) from the original video $V$ that contain the visual information necessary for solving the task. This is not the entire video; it is a curated subset identified during human annotation that pinpoints exactly which frames show the relevant object, action, or scene.
• $A = \{a_1, a_2, \ldots, a_n\}$ is the agent's action history — the sequence of browser operations it performed.
• $S = \{s_1, s_2, \ldots, s_k\}$ is the sequence of browser screenshots captured during the agent's execution.
• $O \in \{\text{Success}, \text{Failure}\}$ is the binary outcome.

What this function computes. It determines whether the agent's web actions and their final result satisfy two conditions simultaneously: (1) the task described in $I$ was completed (the agent reached the required webpage, retrieved the required information, applied the required filters), and (2) the web result is visually consistent with the real-world content shown in the egocentric video evidence $v$. The second condition is what makes Ego2WebJudge novel — it closes the grounding loop that prior evaluators leave open.

Why this form. The inclusion of $v$ in the evaluation function reflects the core insight that success on Ego2Web tasks is not purely a matter of web navigation competence. An agent could navigate to Amazon, search for "harvest cheddar chips," and report nutritional information, and a naive evaluator would mark this as success. But if the video actually showed a different brand of chips — say, "Lay's Cheddar" — the agent has failed, even though its web execution was flawless. By providing $v$ to the judge, Ego2WebJudge can detect this category of error: the mismatch between what the user saw in the physical world and what the agent retrieved on the web.

Stage 1: Key-Point Identification

Before evaluating the agent's trajectory, Ego2WebJudge first distills the task instruction into explicit key points — atomic, verifiable sub-goals that collectively define task success.

Procedure. Given the task instruction $I$, a text LLM (the paper does not specify which LLM is used for this stage, but it is presumably the same LLM backbone used as the judge in Stage 3) is prompted to extract the critical key points that must be achieved. For example, for the instruction "On Amazon, find the same snack as the fourth one picked up by the person in the video. And report the Calories per serving," the key points might be:

• Identify the fourth snack picked up in the video.
• Navigate to Amazon.
• Locate the product page for that exact snack (matching brand, flavor, packaging).
• Extract and report the Calories per serving from the product listing.

Why this stage is necessary. The task instructions in Ego2Web are natural language and can be complex, involving multiple sequential sub-goals and implicit constraints. Directly feeding the raw instruction to the judge and asking "did the agent succeed?" leads to unreliable judgments because the judge must simultaneously parse the instruction, identify what success means, and evaluate the trajectory against that definition — a cognitively demanding task that introduces variance. By pre-extracting key points, Ego2WebJudge decomposes the evaluation into explicit, checkable criteria, reducing the risk that the judge overlooks a sub-goal or applies inconsistent standards across different examples. This design is motivated by prior work (the paper cites WebJudge, Xue et al., 2025) and the observation that "instructions in Ego2Web often involve multi-step reasoning, and distilling the instruction into explicit key points helps the LLM evaluation with this prior" (Section 4).

Stage 2: Key Screenshot Selection

Web agent trajectories in Ego2Web can contain 5 to 20 steps (and up to 40, given the action budget), with each step producing a screenshot. Many of these screenshots are irrelevant to the evaluation — loading pages, intermediate navigation states, error pages, or backtracking steps. Feeding all screenshots into the multimodal judge would cause two problems: (1) context overflow, since MLLMs have limited context windows and packing in dozens of screenshots can exceed these limits, and (2) diluted judgment quality, since the judge's attention is spread across irrelevant visual information, reducing its ability to focus on the screenshots that actually matter.

Procedure. Ego2WebJudge addresses this with a filtering stage:

1. A multimodal LLM (the paper uses the same judge model that will perform the final evaluation, e.g., Gemini-2.5 Pro or GPT-4o) is prompted to summarize each screenshot $s_i$ in the trajectory.
2. The MLLM rates each screenshot's relevance to the task on a 1–5 scale.
3. Only screenshots whose relevance score exceeds a threshold $\delta$ are retained as key screenshots. The paper does not explicitly specify the value of $\delta$, but the principle is clear: screenshots that show task-critical states (e.g., the product page after a successful search, the map result showing the correct location, the Wikipedia article with the requested information) are kept; screenshots showing loading spinners, intermediate search result pages with incorrect items, or navigation errors are filtered out.

What this filter computes. It selects a subset of the trajectory screenshots that are most informative for judging success or failure. This is essentially an attention mechanism: rather than forcing the judge to process the entire trajectory, Ego2WebJudge highlights the parts that matter, improving both efficiency and accuracy.

Why this form. The alternative — feeding all screenshots to the judge — would be simpler to implement but would degrade evaluation quality, particularly for longer trajectories. The relevance-based filtering ensures that the judge's context window is used efficiently, focusing its multimodal reasoning capacity on the screenshots that contain evidence of task completion (or failure). This design also addresses a common failure mode in prior evaluators identified by the paper: "over-reliance on surface-level text matching or incomplete trajectory understanding" (Section 5.2).

Design choice: separate key-point and key-screenshot extraction. The paper extracts key points from the instruction before selecting key screenshots, rather than doing both jointly. This sequential design makes logical sense: knowing what to look for (key points) helps determine which screenshots are relevant. A joint model could theoretically be more efficient, but the sequential approach is more interpretable and allows each stage to be optimized independently.

Stage 3: Final Outcome Judgment

The final stage combines all the evidence — task instruction, LLM-extracted key points, selected key screenshots, agent action history, and annotated egocentric video evidence — and produces a binary success/failure verdict.

Procedure. A multimodal LLM (options evaluated in the paper include Qwen3-VL-Flash, Gemini-2.5 Pro, and GPT-4o) is prompted with all five inputs and asked to determine whether the agent successfully completed the task. The prompt (provided in full in Appendix B) is extensive and includes:

• A detailed system prompt defining the judge's role, the strict evaluation principles, and concrete criteria for success and failure.
• The egocentric video evidence — provided as sampled keyframes from the annotated visual evidence clip $v$, with frame indices or timestamps indicating when in the video each keyframe appears.
• The task instruction $I$.
• The key points extracted in Stage 1.
• The agent's action history $A$.
• The selected key screenshots $S_{\text{key}}$ with accompanying text explanations (generated during the relevance scoring in Stage 2).

The judge outputs exactly two lines:

Thoughts: <reasoning based on key points, webpage evidence, and ego video evidence>
Status: success or failure

The strict evaluation principle. The system prompt enforces a crucial asymmetry in judgment: false positives (marking a failure as success) are considered worse than false negatives (marking a success as failure). The prompt explicitly instructs:

"If there is any uncertainty, ambiguity, mismatch, or missing visual evidence, the task must be marked as failure."

This conservative bias is intentional. For a benchmark, it is more important to avoid incorrectly certifying agents as capable than to avoid incorrectly penalizing them. An agent that truly succeeds on a task but gets marked as a failure will still demonstrate its capability across other tasks; an agent that fails but gets marked as a success would inflate benchmark scores and obscure genuine limitations.

The visual grounding rules. The prompt includes ten specific rules for evaluating visual grounding (Appendix B, "Ego Video Grounding Rules"), which collectively enforce the requirement that the web result must match the egocentric video evidence. Key rules include:

• "The webpage result must be strictly grounded in ego video evidence, including object identity, category, brand, color, quantity, state, and action cues."
• "Apparent success is not sufficient. Even if the webpage looks correct, the task fails if it does not match the ego video."
• "Carefully verify fine-grained visual details: object identity, brand, color, material, shape, text, number of items, spatial relations, and actions."
• "For object retrieval tasks, the result must correspond to the same object or correct category supported by the video evidence."
• "If evidence conflicts, prioritize visual grounding over textual claims."
• "Near or partial matches are considered failure when precise identification is required."

These rules operationalize the benchmark's core requirement — that success depends on correct visual perception, not just web navigation competence — into explicit, machine-interpretable criteria.

The common failure cases checklist. The prompt also enumerates specific failure patterns to guide the judge's reasoning (Appendix B):

• Product mismatch (different brand, color, or type than shown in the video)
• Similar-looking but incorrect objects
• Related but different events or actions in retrieved media
• Titles suggesting correctness but screenshots contradicting it
• Insufficient visual evidence to confirm a match
• Claims not supported by video or webpage evidence
• Partial matches that do not fully satisfy the task

What the judge computes. It performs a multimodal alignment check: comparing the visual content of the egocentric video keyframes against the visual content of the key web screenshots, mediated by the task key points. The reasoning is not purely textual (comparing strings) nor purely visual (comparing pixel patterns); it is cross-modal, requiring the judge to determine whether the real-world object shown in the video is the same as the product, location, or media item shown on the webpage. This is a challenging judgment that leverages the MLLM's pre-trained understanding of object identity, visual similarity, and brand recognition.

Why this design outperforms prior evaluators. The paper demonstrates (Table 4) that Ego2WebJudge achieves 84.0% agreement with human judgment when using GPT-4o as the judge model, compared to 78.4% for WebJudge and 74.7% for WebVoyager. The improvement comes from the integration of egocentric visual evidence, which prior evaluators lack. WebVoyager and WebJudge can verify whether the agent performed the correct web actions and reached a webpage that textually matches the instruction, but they cannot verify whether the webpage's content corresponds to what the video actually shows. Ego2WebJudge can, because it has access to the visual ground truth $v$.

Multi-judge model evaluation. The paper instantiates Ego2WebJudge with three different MLLM backbones — Qwen3-VL-Flash, Gemini-2.5 Pro, and GPT-4o — to assess whether the evaluation framework is robust to the choice of judge model. The results (Table 2) show that all three judges produce consistent relative rankings across agents, though their absolute scores differ: Qwen3-VL-Flash tends to give higher scores, while Gemini-2.5 Pro and GPT-4o align more closely with human evaluation. GPT-4o achieves the highest agreement with human judgment (84.0% average across agents), making it the recommended judge backbone.

The agreement calculation. The agreement rate (AR) reported in Table 4 is computed per agent as the fraction of benchmark examples where the automatic judge's binary verdict matches the human evaluators' majority-vote verdict. For human evaluation, three annotators independently assess each agent's output, and majority voting determines the ground-truth success/failure label. This ground truth is then compared against each automatic judge's binary output to compute agreement.

Why agreement is high for weak agents and harder for strong agents. The paper observes (Section 5.2) that Claude-based agents exhibit relatively high agreement rates across all evaluation methods, despite having low success rates. This is because these agents frequently fail tasks outright — producing short, obviously unsuccessful trajectories — and both humans and automatic judges can easily agree on failure. In contrast, stronger agents like BU-GPT-4.1 and BU-Gemini-3-Flash produce more diverse and partially successful behaviors (e.g., finding the right product category but the wrong brand, navigating to the correct website but failing to extract the specific required information), making evaluation inherently more challenging. Ego2WebJudge maintains high agreement even in these nuanced cases, demonstrating robustness to intermediate outcomes that prior evaluators handle poorly.

Implementation Details for Ego2WebJudge

The paper specifies several concrete implementation choices:

• Base MLLM options: Qwen3-VL-Flash (a smaller, faster model variant), Gemini-2.5 Pro, and GPT-4o are all evaluated as judge backbones. The paper also references using Qwen3-VL-Flash "as the small model variants of Ego2WebJudge" (Section 5.1), suggesting that the framework is designed to work with faster, cheaper judges for rapid iteration, with larger judges used for final, high-stakes evaluation.
• Keyframe extraction from video evidence $v$: The annotated visual evidence clip $v$ is presented to the judge as sampled keyframes, not as raw video. The paper does not specify the sampling rate, but the use of keyframes (rather than full video) keeps the multimodal context within manageable limits for current MLLMs, which may have token or frame-count constraints.
• Screenshot relevance scoring: Each screenshot in the agent's trajectory is summarized and scored on a 1–5 relevance scale by the judge MLLM. The paper does not report the specific prompt used for this scoring, but the principle is to filter out irrelevant or redundant screenshots before the final judgment.
• Output format enforcement: The judge is required to output exactly two lines — "Thoughts:" and "Status:" — ensuring parseable, consistent outputs that can be automatically extracted for metric computation without manual inspection.

Experimental Setup Details Implicit in the Technical Approach (Section 5.1)

Several implementation choices described in the experimental setup are integral to understanding the Ego2Web technical approach, as they define how agents interface with the benchmark and how human evaluation is conducted for calibration.

Video input modality handling. Different agents have different capabilities for processing video. The paper handles this heterogeneity through a capability-adaptive input conversion:

• Agents with native video support (Gemini-based agents, Browser-Use with Gemini-3-Flash): Receive the egocentric video as raw video input, enabling dense temporal processing with access to all frames and fine-grained motion cues. These agents represent the strongest visual perception setting.

• Agents with keyframe-based input (GPT-4o-based agents, SeeAct): Receive the egocentric video as sampled keyframes — a sparse temporal representation that captures key moments but misses intermediate frames and continuous motion. This reflects the practical limitation that many API-accessible MLLMs do not yet support native video input.

• Agents with no video input (Claude Computer-Use, GPT-5.4): Cannot receive raw video or keyframes in their computer-use mode. For these agents, the video is converted into structured, detailed text captions generated by Gemini-3.1-Pro — a strong MLLM chosen to "capture video details as much as possible" (Section 5.1). The captions include timestamps and detailed descriptions of objects, actions, and scenes (an example caption covering a three-minute video is provided in Appendix B). This conversion inevitably causes information loss, particularly for fine-grained spatial-temporal cues and subtle visual interactions, but it is the only feasible way to evaluate these agents on Ego2Web without modifying their architectures.

Human evaluation protocol. To establish ground-truth success rates for calibrating the automatic evaluators, three human annotators independently assess each agent's output for every benchmark example. The final human-evaluated success rate is determined by majority voting across the three annotators. This protocol provides a more reliable gold standard than single-annotator evaluation, reducing the impact of individual annotator errors or inconsistencies. The annotation task involves watching the egocentric video, reading the task instruction, reviewing the agent's action trajectory and screenshots, and determining whether the agent successfully completed the task with correct visual grounding.

Action budget. All agents are limited to a maximum of 40 steps per episode. This budget is generous enough to accommodate multi-step web tasks (navigating, searching, scrolling, clicking, extracting information) while preventing agents from getting stuck in infinite loops or excessively long exploration. Trajectories that exceed 40 steps without completing the task are counted as failures.

Data generation models and configuration. The pipeline uses specific model versions and settings:

• Video captioning: Qwen3-VL-7B, processing videos in 5-second clips.
• Task instruction generation: GPT-5, prompted with the video profile and website pool.
• Video-to-caption conversion for non-video agents: Gemini-3.1-Pro, producing structured timestamped captions.

Dataset composition. The final benchmark contains 500 video–task pairs, with the distribution across task categories reflecting the natural prevalence of e-commerce and media-related tasks in the generated pool: E-Commerce (230), Media Retrieval (132), Knowledge Lookup (92), Local/Maps (31), and Others (15).

Summary of Design Choices and Their Justifications

Semi-automatic LLM+Human pipeline over fully manual creation. Manual creation of 500 video–task pairs, where each task requires watching a three-minute video and inventing a web task that depends on specific visual evidence, would take hundreds of annotator hours. The LLM generates candidate tasks from captions in seconds, and human annotators focus on verification and refinement — a division of labor that dramatically reduces cost while maintaining quality. The three verification criteria (visual grounding, web feasibility, instruction quality) provide systematic quality control that catches errors from the automated stages.

Qwen3-VL-7B for captioning over larger MLLMs. The 7B parameter model is computationally efficient enough to process hundreds of three-minute videos at 5-second granularity. The human verification stage provides a safety net — any captioning inaccuracies that survive to task generation are caught during the manual visual grounding check.

GPT-5 for task generation over other LLMs. Task generation requires creative reasoning and adherence to complex output constraints. GPT-5's strong instruction-following and generation capabilities make it well-suited for this role. The structured three-step prompt (suitability assessment → anchor extraction → task generation) constrains the generation to produce verifiable, video-dependent tasks rather than generic web instructions.

Ego2WebJudge key-point extraction before screenshot selection. Decomposing the evaluation into explicit sub-goals before identifying relevant screenshots improves judgment quality by providing clear criteria for relevance. This sequential design prevents the judge from being distracted by screenshots that appear task-relevant but do not actually contain evidence about the key sub-goals.

Strict evaluation principle with false-positive aversion. The conservative bias toward marking uncertain cases as failures, rather than successes, ensures that benchmark scores represent a lower bound on true agent capability. This is appropriate for a benchmark intended to reveal limitations and drive progress, where overestimating performance is more harmful than underestimating it.

Capability-adaptive input handling rather than forcing a single modality. Different agents have fundamentally different visual processing capabilities. Forcing all agents to use captions would obscure the benefit of native video understanding; forcing all agents to use raw video would exclude important baselines. The adaptive approach allows meaningful comparisons within each modality class while maintaining the overall benchmark's relevance to both current and future agents.

Live online evaluation over sandboxed environments. Real websites have dynamic content, A/B testing, CAPTCHAs, and constantly evolving interfaces that sandboxed simulations cannot replicate. Evaluating agents on live websites tests their robustness to the real-world web conditions they would encounter in deployment, providing more ecologically valid performance estimates.

4. Key Insights and Innovations

Innovation 1: Defining a Previously Invisible Task — Connecting Egocentric Perception to Web Action

The paper's most fundamental contribution is conceptual rather than algorithmic: it identifies and operationalizes a task that the field did not recognize as a distinct capability gap before. Prior to Ego2Web, egocentric video understanding and web agent execution existed as entirely separate research programs. Egocentric benchmarks (EgoSchema, EgoThink, EgoPlan) evaluated whether models could answer questions about first-person video, while web-agent benchmarks (WebArena, VisualWebArena, WebVoyager) evaluated whether agents could navigate websites given text instructions or screenshots. No benchmark asked: "Given what you see in the physical world, go do something about it on the web."

This is not merely a combination of two existing benchmarks — it is a qualitatively different task that exposes failure modes invisible in either precursor alone. An agent that scores perfectly on EgoSchema (demonstrating strong video understanding) and perfectly on WebVoyager (demonstrating strong web navigation) could still fail catastrophically on Ego2Web if it cannot correctly translate visual perception into appropriate web actions — misidentifying which product to search for, confusing temporal order (the second object vs. the fourth), or retrieving a semantically similar but physically different item. Conversely, an agent that succeeds on Ego2Web must possess both competencies and, critically, the integration layer between them — a capability no prior benchmark could measure or incentivize.

The paper's Table 1 makes this contribution explicit through its four-column taxonomy (Visual Grounding, Egocentric Video Perception, Web Task, Online Evaluation). The empty row — the configuration where all four columns are checked — represents a genuine gap in the literature, not a minor variant of existing work. By filling this row with a concrete, executable benchmark, the paper creates a new evaluation category that redefines what "multimodal agent capability" means: not just perception or action in isolation, but perception–action coupling grounded in real-world visual evidence.

Innovation 2: The Difficulty Diagnosis — Current Agents Are Stronger at Web Navigation Than at Visual Grounding

The experimental results in Section 5 provide a diagnostic finding with implications that extend beyond the specific benchmark: state-of-the-art web agents are substantially better at the web-execution half of the task than at the visual-grounding half. This is not an obvious result — one could have hypothesized that agents with strong web capabilities would also handle video perception reasonably well, or that the bottleneck would be in the integration between perception and action rather than in perception itself. The evidence suggests otherwise.

Several findings converge on this diagnosis:

The input modality ablation (Table 5) provides the clearest evidence. When the same agent (BU-Gemini-3-Flash) receives no visual input, it achieves only 4.4% success rate — essentially chance performance, confirming that visual information is strictly necessary. When it receives detailed text captions describing the video (generated by Gemini-3.1-Pro, a strong MLLM), performance rises to 23.6% — a substantial improvement, but still far below the 48.2% achieved with raw video input. The jump from captions to video (a factor of 2× improvement) dwarfs the jump from no visual input to captions, indicating that fine-grained visual information unavailable in text descriptions — precise object appearance, spatial relationships, temporal dynamics, subtle visual attributes — is the rate-limiting factor. If web navigation were the primary bottleneck, the caption-based setting (which provides all the semantic information needed to construct web queries) would perform much closer to the video setting. The persistent large gap reveals that current agents cannot extract actionable information from raw video as effectively as they can from text, and that text-based proxies leave critical visual detail on the table.

The domain difficulty hierarchy supports this interpretation. Knowledge Lookup tasks (Wikipedia, StackExchange) have the highest success rates (75.0% for the best agent, averaging 50.0% across agents), while E-Commerce tasks have the lowest (38.2% for the best agent, averaging 21.7%). Knowledge Lookup typically requires identifying a broad category or entity (a brand, a university, a board game) and then finding its Wikipedia article — a task where coarse-grained visual recognition suffices and web navigation is relatively structured. E-Commerce requires fine-grained identification of specific products (exact brand, flavor, packaging, color) and then navigating dynamic, visually heterogeneous shopping interfaces. The degradation from Knowledge Lookup to E-Commerce maps onto the increasing demand for precise visual grounding rather than increasing web complexity — Wikipedia articles have structured layouts that are arguably easier to navigate than product pages, but the primary gap is in the precision of visual identification required, not the difficulty of web interaction.

The error analysis (Section 5.3) quantifies this directly. Of 50 manually inspected failure cases from the strongest agent, 36% involve object misidentification (the agent correctly navigates the web but searches for the wrong item), 18% involve temporal or action misunderstanding (confusing the order of events), and 16% involve cross-modal retrieval failure (correctly identifying the object but failing to locate it on the web). Only 18% fall into a miscellaneous category including web-specific failures like CAPTCHAs or action limits. This distribution demonstrates that perceptual errors, not web-execution errors, dominate the failure profile — agents are failing primarily because they see the wrong thing, not because they cannot act on what they see.

This diagnosis is significant because it redirects research attention. The web-agent community has focused heavily on improving action execution — better planning, more efficient search, robust error recovery. Ego2Web's results suggest that for the next generation of physically-grounded agents, improving visual perception and grounding may yield larger gains than further optimizing web navigation. This is not an obvious prioritization from existing benchmarks, none of which isolate the perception–action coupling that Ego2Web tests.

Innovation 3: Ego2WebJudge — Extending LLM-as-a-Judge to Grounded Multimodal Evaluation

The paper introduces a methodological contribution in automatic evaluation that addresses a problem specific to grounded tasks: how can an automatic evaluator verify that a web result matches visual evidence from the user's environment, when the evaluator only sees the web output and the agent's trajectory? Prior LLM-as-a-Judge frameworks for web agents (WebVoyager, WebJudge) could verify that the agent completed the requested web actions and reached a relevant webpage, but they were blind to visual grounding — they could not check whether the product on the webpage was actually the same object shown in the user's video.

Ego2WebJudge solves this by incorporating the annotated visual evidence v directly into the evaluation function, treating egocentric keyframes as ground-truth reference data against which the agent's web result is compared. This is more than an incremental improvement to existing evaluators; it defines a new evaluation paradigm where the judge's verdict depends on cross-modal alignment between physical-world visual evidence and digital-world output. The judge must reason about object identity, visual similarity, brand recognition, and attribute matching across two domains — a fundamentally different capability from text-trajectory or screenshot-only evaluation.

The empirical validation of this approach is strong. Ego2WebJudge achieves 84.0% agreement with human judgment using GPT-4o as the judge backbone, compared to 78.4% for WebJudge and 74.7% for WebVoyager (Table 4). The 5.6–9.3 percentage point improvement is substantial given that agreement rates in this range represent a difficult ceiling — some fraction of remaining disagreement likely reflects genuine human annotator noise rather than judge error. More importantly, the improvement is concentrated in the cases where evaluation is hardest: stronger agents that produce nuanced, partially successful trajectories. For Claude-based agents that fail obviously and frequently, all evaluators agree. For BU-Gemini-3-Flash, which produces more ambiguous outcomes, Ego2WebJudge maintains higher agreement than competitors (78.0% vs. 74.2% for WebJudge with Gemini-2.5 Pro), demonstrating that the visual grounding component matters precisely when evaluation is most challenging.

The design principle underlying Ego2WebJudge — that automatic evaluation of grounded agents must have access to the same grounding evidence that defines task success — is conceptually significant and generalizable. Any future benchmark that connects real-world perception to digital action (e.g., an agent that watches a cooking video and orders ingredients, or observes a mechanical repair and purchases replacement parts) will face the same evaluation challenge. Ego2WebJudge provides a template: identify the grounding evidence (in this case, keyframes from an egocentric video), incorporate it into the judge's input alongside the agent's trajectory, and enforce explicit visual-consistency rules that penalize mismatches between the grounding evidence and the agent's output.

Two design choices in Ego2WebJudge are notable for their practical wisdom rather than theoretical novelty:

The key-screenshot selection stage (Stage 2) solves a pragmatic problem that would otherwise degrade judgment quality: web trajectories contain many irrelevant screenshots (loading pages, backtracking, errors) that dilute the judge's attention and consume context-window capacity. By pre-filtering screenshots via relevance scoring, Ego2WebJudge focuses the expensive multimodal reasoning on the frames that matter. This is an engineering choice with significant practical impact — it is likely responsible for part of the improvement over WebJudge, which does not perform this filtering.

The conservative evaluation principle — that uncertain cases are marked as failures, prioritizing false-negative avoidance over false-positive avoidance — is a deliberate asymmetry that makes the benchmark a lower bound on true agent capability. This is appropriate for a benchmark designed to reveal limitations: overestimating performance would obscure the very gaps the benchmark is meant to expose. The explicit "When in doubt, fail" instruction in the judge prompt operationalizes this principle, and the 84% agreement rate suggests that human annotators largely share this conservatism (since agreement is measured against their majority-vote judgments).

Innovation 4: A New Data Generation Paradigm — Semi-Automatic Pipeline for Grounded Task Creation

The semi-automatic LLM-plus-human pipeline (Section 3.2) represents a methodological contribution to benchmark construction that addresses a specific bottleneck: how to efficiently create tasks that require visual grounding, when the task generator (an LLM) cannot process video directly. This is not a trivial engineering convenience; it solves a chicken-and-egg problem that would otherwise make benchmarks like Ego2Web prohibitively expensive to construct.

The approach is two-stage modality decomposition: a multimodal model (Qwen3-VL-7B) converts video into structured text captions, and a text-only LLM (GPT-5) generates task instructions from those captions. This decomposition is conceptually elegant because it separates the problem into components where strong, off-the-shelf models exist: video-to-text description (a standard MLLM capability) and creative task generation from text (a standard LLM capability). Neither model alone could build the benchmark — the MLLM cannot invent web tasks, the LLM cannot see video — but the pipeline composition achieves what neither can alone.

The critical design choice that makes this pipeline work at quality is Stage 3 human verification with explicit grounding criteria. Without human verification, the pipeline would produce tasks based on captions that may not accurately represent the video, or tasks that reference visual details that are not actually discernible in the footage. The three verification criteria — visual grounding, web feasibility, and instruction quality — systematically catch errors from the automated stages. Visual grounding verification ensures that the task's perceptual requirements are genuinely satisfiable from the video. Web feasibility verification prevents tasks that are theoretically grounded but practically impossible (defunct websites, out-of-stock products). Instruction quality verification eliminates ambiguity that could make agent failures attributable to unclear task descriptions rather than capability gaps.

This pipeline is significant beyond Ego2Web because it defines a reusable pattern for grounded benchmark construction in any domain where the task requires linking perceptual evidence to action. The pattern — automated perception-to-text conversion, LLM-based task generation from text, human verification against the original perceptual data — could be applied to construct benchmarks for agents that perceive through other modalities (audio, depth sensors, thermal cameras) and act in other environments (mobile apps, operating systems, robotics). The specific models (Qwen3-VL, GPT-5) are incidental; the pipeline architecture is the reusable contribution.

The paper's decision to release the benchmark, code, and data generation prompts supports this interpretation. By making the pipeline transparent, the authors enable the community to extend Ego2Web with new videos, new websites, and new task types — or to adapt the pipeline for entirely different grounded-agent domains — without requiring the original authors' involvement.

Innovation 5: Input Modality as a First-Class Variable — Not All "Multimodal" Agents Are Equal

A finding that emerges from the experimental design rather than being explicitly claimed as a contribution is that the format in which visual information reaches an agent matters enormously, and current agent architectures vary dramatically in their ability to consume rich visual input. The paper's experimental setup reveals a clear performance hierarchy — raw video > keyframes > captions > no visual input — that is consistent across all task domains (Table 5) and has direct implications for agent design.

This finding is subtle because it is not about model quality in the conventional sense. GPT-5.4 and Claude 4.5 are state-of-the-art models with strong reasoning capabilities, but they underperform BU-Gemini-3-Flash on Ego2Web primarily because their architectures cannot accept raw video input (Table 2: GPT-5.4 achieves 30.6% human-evaluated success rate vs. 58.6% for BU-Gemini-3-Flash). The performance gap attributable to input modality dwarfs any reasonable estimate of the gap attributable to underlying model capability differences. This establishes that for grounded tasks, input modality is as important as model scale or training data — a conclusion that is not obvious from prior benchmarks where all agents receive text or screenshots.

The practical implication is clear but not widely internalized by the agent-building community: deploying an agent for physically-grounded tasks requires investing in native video understanding capabilities, not just strong language reasoning. Converting video to captions as a preprocessing step, even with a powerful captioning model, discards fine-grained visual information that is essential for precise grounding — the 2× performance gap between captions and raw video is too large to close through better captioning alone. This finding argues for architectural decisions (video-native multimodal encoders, dense temporal processing) that may seem unnecessary in text-centric or screenshot-centric benchmarks but become critical when agents must perceive the physical world.

The paper's capability-adaptive evaluation design — testing agents with their best available input modality rather than forcing a common format — makes this finding interpretable. If all agents had been forced to use captions, the input-modality effect would be invisible, and the benchmark would conflate agent capability with input-representation quality. By allowing each agent to use its native modality, the benchmark reveals that input modality is itself a design choice with measurable performance consequences, making it a legitimate axis for agent comparison and improvement.

5. Experimental Analysis

Evaluation Methodology

• Dataset. Ego2Web consists of 500 video–task pairs, with the videos sourced from Ego4D (Grauman et al., 2022) and curated following EgoSchema's (Mangalam et al., 2023) selection criteria: non-overlapping three-minute clips with at least 30 human-annotated narrations. The 500 examples are partitioned into five task categories: E-Commerce (230 tasks, 50.3%), Media Retrieval (132, 24.1%), Knowledge Lookup (92, 17.0%), Local/Maps (31, 6.0%), and Others (15, 2.6%). The benchmark uses the full 500 examples for evaluation (no train/validation/test split is reported, as this is an evaluation-only benchmark rather than a training dataset).

• Base model(s). Six web agents spanning diverse architectures and MLLM backbones are evaluated: SeeAct (Zheng et al., 2024b), Browser-Use with GPT-4.1 (OpenAI, 2026a; Müller and Žunič, 2024), Browser-Use with Gemini-3-Flash (Deepmind, 2026), Claude Sonnet 3.7 Computer Use (Anthropic, 2024), Claude Sonnet 4.5 Computer Use (Anthropic, 2025), and GPT-5.4 (OpenAI, 2026b). These agents were chosen to span the range of current state-of-the-art web-agent capabilities, including both proprietary frontier models (GPT-5.4, Claude 4.5, Gemini-3-Flash) and research systems (SeeAct). The diversity in MLLM backbones and input modalities (raw video, keyframes, captions) is an intentional feature of the evaluation design, not a confound — the benchmark aims to characterize how different agent architectures handle egocentric grounding, and input modality is itself a first-class variable.

• Metrics. The primary metric is Success Rate (SR), defined as the fraction of the 500 benchmark tasks for which the agent's final output is judged to have successfully completed the task. Success is determined by two evaluation methods: (1) Human Evaluation, where three annotators independently assess each agent's output and majority voting determines the binary success/failure label, and (2) Ego2WebJudge, the proposed automatic evaluation framework instantiated with three different MLLM judge backbones (Qwen3-VL-Flash, Gemini-2.5 Pro, and GPT-4o), which produces a binary success/failure verdict based on task instruction, annotated visual evidence, action history, and selected screenshots. Additionally, Agreement Rate (AR) is reported when comparing automatic evaluators against human judgments — it is the fraction of benchmark examples where the automatic judge's binary verdict matches the human majority-vote verdict.

• Baselines. The paper does not define a single "baseline agent" against which others are compared; rather, it evaluates six agents against each other and against the oracle performance ceiling (the maximum achievable success rate if all tasks were solved correctly, i.e., 100%). Each agent serves as a baseline for the others, with BU-Gemini-3-Flash emerging as the strongest. For the evaluation framework (Ego2WebJudge), the baselines are two prior automatic evaluators: WebVoyager (He et al., 2024) and WebJudge (Xue et al., 2025), both of which evaluate web agents using LLM-as-a-Judge but lack visual grounding to egocentric evidence. These baselines are compared against Ego2WebJudge on the Agreement Rate metric.

• Generation budget / compute accounting. The paper does not measure compute in FLOPs or token counts; instead, it uses a maximum action step budget of 40 per episode as a uniform constraint across all agents. An episode that exceeds 40 steps without completing the task is counted as a failure. This budget is generous enough to accommodate multi-step web interactions (navigating, searching, clicking, scrolling, extracting information) while preventing unbounded exploration. The paper does not report wall-clock time, API costs, or total tokens consumed per episode, which would be relevant for practical deployment cost comparisons but are orthogonal to the capability evaluation focus.

• Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals on success rates. The evaluation is conducted on the full 500-example benchmark for each agent. Human evaluation uses three annotators per agent output with majority voting, providing some robustness to individual annotator variance, but no inter-annotator agreement metrics (e.g., Fleiss' kappa, Krippendorff's alpha) are reported. For the agreement rate between automatic evaluators and human judgments, the metric is computed as a simple fraction of matching verdicts across the 500 examples, per agent, with an average agreement rate computed across the six agents. No statistical significance tests are reported for comparisons between agents or between evaluation methods.

Main Quantitative Results

The experimental results are organized around two axes: (1) agent performance comparison across six state-of-the-art systems, and (2) evaluation framework validation comparing Ego2WebJudge against prior automatic evaluators. The headline finding is that even the strongest agent (BU-Gemini-3-Flash) achieves only 58.6% success rate under human evaluation, leaving a roughly 40% gap from oracle performance, and that input modality (raw video vs. keyframes vs. captions) is a dominant factor in agent performance.

Agent Performance Comparison

Overall success rates across agents. Table 2 reports the success rate of all six agents under both human evaluation and Ego2WebJudge with three judge backbones. Under human evaluation, the performance hierarchy is:

• BU-Gemini-3-Flash: 58.6% — the strongest agent by a clear margin, roughly 14 percentage points ahead of the second-best agent.
• BU-GPT-4.1: 44.4% — the second-strongest, representing Browser-Use with a non-video-native MLLM backend.
• SeeAct: 34.2%
• Claude 4.5: 32.8%
• GPT-5.4: 30.6%
• Claude 3.7: 26.4% — the weakest agent.

The gap between the best agent (58.6%) and the oracle ceiling (100%) is approximately 41.4 percentage points. The gap between the best and worst agents is approximately 32.2 percentage points, indicating substantial variance in capability across current systems. Notably, the three weakest agents (GPT-5.4, Claude 4.5, Claude 3.7) cluster within a narrow range (26.4–32.8%), while BU-Gemini-3-Flash is an outlier on the high end, suggesting that native video understanding capability creates a qualitative performance jump rather than a smooth gradient.

Agreement between automatic and human evaluation (Table 4). Ego2WebJudge is evaluated on its ability to reproduce human judgments. The Agreement Rate (AR) between each automatic evaluator and human evaluation, averaged across all six agents, is:

• Ego2WebJudge with GPT-4o: 84.0%
• Ego2WebJudge with Gemini-2.5 Pro: 80.8%
• WebJudge with GPT-4o: 78.4%
• WebJudge with Gemini-2.5 Pro: 76.1%
• WebVoyager with GPT-4o: 74.7%
• WebVoyager with Gemini-2.5 Pro: 70.7%

Ego2WebJudge with GPT-4o achieves the highest agreement, outperforming the best prior evaluator (WebJudge with GPT-4o at 78.4%) by 5.6 percentage points. The improvement is consistent across both judge backbones: Ego2WebJudge outperforms WebJudge by 4.7 percentage points with Gemini-2.5 Pro (80.8% vs. 76.1%) and by 5.6 percentage points with GPT-4o (84.0% vs. 78.4%). This consistency suggests the improvement is attributable to Ego2WebJudge's design (incorporating visual evidence) rather than to idiosyncratic advantages of a particular judge backbone.

Per-agent agreement patterns. The agreement rates are not uniform across agents. For Claude 3.7 (the weakest agent), all evaluators achieve high agreement (85.4% for Ego2WebJudge with Gemini-2.5 Pro, 77.8% for WebVoyager with Gemini-2.5 Pro). For BU-Gemini-3-Flash (the strongest agent), agreement rates are lower across all evaluators (78.0% for Ego2WebJudge with Gemini-2.5 Pro, 66.2% for WebVoyager with Gemini-2.5 Pro). This pattern reflects the difficulty of evaluating nuanced, partially successful trajectories — weak agents fail obviously and frequently, making evaluation easy; strong agents produce ambiguous outcomes that challenge both human and automatic judges. Ego2WebJudge maintains higher agreement than competitors even for these difficult cases: with Gemini-2.5 Pro, it achieves 78.0% agreement on BU-Gemini-3-Flash vs. 74.2% for WebJudge and 66.2% for WebVoyager.

Fine-Grained Domain Analysis

Table 3 disaggregates success rates by task domain, evaluated by Ego2WebJudge with Gemini-2.5 Pro. The domain-average success rates (averaged across all six agents) reveal a clear difficulty hierarchy:

• Knowledge Lookup: 50.0% average — the easiest domain. The best agent (BU-Gemini-3-Flash) achieves 75.0%.
• Media Retrieval: 30.1% average — intermediate difficulty. The best agent achieves 50.7%.
• Local/Maps: 23.1% average — harder. The best agent achieves 48.3%.
• E-Commerce: 21.7% average — the hardest major domain. The best agent achieves only 38.2%.
• Others: 14.4% average — the hardest category, though with only 15 tasks, making estimates less reliable. The best agent achieves 13.3% (surprisingly low — BU-Gemini-3-Flash underperforms BU-GPT-4.1 which achieves 40.0% in this category).

The domain difficulty ranking is consistent with the hypothesis that fine-grained visual grounding is the primary bottleneck. Knowledge Lookup typically requires identifying a broad entity (brand, university, object category) and navigating to a structured article — coarse-grained recognition suffices. Media Retrieval requires finding a specific video matching an action or event — moderately fine-grained temporal and visual matching. E-Commerce requires identifying exact product variants (brand, flavor, size, packaging) across dynamic shopping interfaces — the most demanding visual precision requirements. The 75.0% vs. 38.2% gap for the best agent between Knowledge Lookup and E-Commerce quantifies how much harder precise product identification is than entity-level recognition.

Per-agent domain profiles. BU-Gemini-3-Flash dominates every domain but with varying margins:

• Knowledge Lookup: 75.0% (vs. second-best SeeAct at 43.4%) — a 31.6 percentage point gap, the largest relative advantage.
• Media Retrieval: 50.7% (vs. second-best BU-GPT-4.1 at 30.3%) — a 20.4 percentage point gap.
• Local/Maps: 48.3% (vs. second-best GPT-5.4 at 29.0%) — a 19.3 percentage point gap.
• E-Commerce: 38.2% (vs. second-best BU-GPT-4.1 at 26.9%) — an 11.3 percentage point gap.

The larger gaps in Knowledge Lookup and Media Retrieval suggest that native video understanding (BU-Gemini-3-Flash's key advantage) provides the greatest benefit in domains where temporal reasoning about actions and events matters, rather than in domains requiring purely static object recognition. This is consistent with the input modality ablation (Table 5), which shows that the video-to-caption jump is largest in Knowledge Lookup (39.1% → 75.0%) and Local/Maps (38.7% → 48.3%), domains where temporal sequence understanding is particularly important.

Evaluation Method Comparison: Ego2WebJudge vs. Prior Evaluators

Table 4's agreement rate comparison is the primary validation of Ego2WebJudge. The key pattern is that Ego2WebJudge's advantage over WebJudge and WebVoyager is largest for the strongest agents (where evaluation is hardest) and consistent across both judge backbones.

With Gemini-2.5 Pro as judge:

• For Claude 3.7 (weakest): Ego2WebJudge 85.4% vs. WebVoyager 77.8% — a 7.6 point advantage.
• For BU-Gemini-3-Flash (strongest): Ego2WebJudge 78.0% vs. WebVoyager 66.2% — an 11.8 point advantage.

With GPT-4o as judge:

• For Claude 3.7 (weakest): Ego2WebJudge 86.0% vs. WebJudge 74.2% — an 11.8 point advantage.
• For BU-Gemini-3-Flash (strongest): Ego2WebJudge 80.4% vs. WebVoyager 68.4% — a 12.0 point advantage.

The pattern is clear: Ego2WebJudge's relative advantage grows as agents become more capable and their trajectories become more nuanced. This is exactly where visual grounding matters most — a naive evaluator might see a trajectory that reaches a relevant-looking product page and mark it as success, but only a visually grounded evaluator can detect that the product is the wrong brand or variant. For obviously failing trajectories (common with weak agents), all evaluators agree on failure, so the visual grounding component provides no additional discrimination.

Why GPT-4o outperforms Gemini-2.5 Pro as a judge backbone. GPT-4o achieves higher agreement with human judgment (84.0% average) than Gemini-2.5 Pro (80.8% average) when used as the Ego2WebJudge backbone. The paper does not explain this 3.2 percentage point difference, but it is consistent with GPT-4o's known strengths in instruction-following and structured evaluation tasks. The gap is not enormous — both backbones produce usable evaluation — but it establishes GPT-4o as the recommended configuration for Ego2WebJudge.

Ablation Studies and Robustness Checks

Impact of visual perception modality (Table 5). The most important ablation evaluates BU-Gemini-3-Flash under three input conditions: (1) no visual input — the agent receives only the text instruction, (2) detailed caption only — the agent receives structured text captions generated by Gemini-3.1-Pro describing the video, and (3) raw video input — the agent receives the actual egocentric video. The overall success rates (evaluated by Ego2WebJudge with Gemini-2.5 Pro) are:

• No visual input: 4.4% — effectively chance, confirming that the tasks are impossible without visual information.
• Detailed captions: 23.6% — a substantial improvement, demonstrating that text descriptions capture useful semantic information, but far below video-level performance.
• Raw video: 48.2% — more than double the caption-based performance.

The per-domain breakdown reveals where visual information loss is most damaging:

• Knowledge Lookup: Captions 39.1% → Video 75.0% (35.9 point gap). This domain, despite being the "easiest" overall, shows the largest absolute benefit from raw video. The likely explanation is that Knowledge Lookup tasks often require identifying specific entities (a particular university from a logo on a backpack, a specific car brand) where precise visual details (logo design, text on signage) are critical and get lost in caption compression.
• Local/Maps: Captions 38.7% → Video 48.3% (9.6 point gap). A smaller but still meaningful gap.
• Media Retrieval: Captions 29.5% → Video 50.7% (21.2 point gap). Media Retrieval requires matching specific actions or events — temporal dynamics that captions may describe ("a person exercises") but cannot convey with sufficient specificity to distinguish one exercise video from thousands of similar ones.
• E-Commerce: Captions 13.0% → Video 38.2% (25.2 point gap). Product identification requires fine-grained visual attributes (exact packaging color, brand logo design, text on labels) that captions frequently omit or describe imprecisely.
• Others: Captions 6.6% → Video 13.3% (6.7 point gap). Small sample size makes this estimate less reliable, but the direction is consistent.

Error analysis of failure trajectories (Section 5.3). A manual inspection of 50 randomly sampled unsuccessful trajectories from BU-Gemini-3-Flash (the strongest agent) yields the following failure distribution:

• 36% Object Misidentification: The agent incorrectly identifies the target object from the video, leading to retrieval of irrelevant items. Example: confusing "French's Classic Yellow Mustard" for the actually-referenced sauce (Figure 6 illustrates a related case where the agent misidentifies the temporal order — confusing which item is the "second" one picked up).
• 18% Temporal and Action Misunderstanding: The agent fails to correctly interpret temporal order or actions in the video, such as confusing the second and third interaction. This is a specifically video-understanding failure — recognizing which object is relevant based on when it appears — that static image recognition would not even attempt.
• 16% Failure in Cross-Modal Retrieval: The agent correctly identifies the target object but fails to locate the required information on the web. This is a web-execution failure that occurs despite correct visual perception — the agent knows what it is looking for but cannot find it online.
• 12% Coarse-Grained Matching Errors: The agent retrieves semantically similar but incorrect results, such as a related video that does not match the exact required tutorial or event. This is a precision failure — the agent gets the category right but the specific instance wrong.
• 18% Others: Failures due to instruction misinterpretation, planning inefficiency (exceeding the 40-step action limit), or external constraints such as CAPTCHA challenges and authentication barriers.

The dominance of perceptual errors (object misidentification + temporal misunderstanding = 54%) over web-execution errors (cross-modal retrieval failure = 16%) confirms that visual grounding, not web navigation, is the primary bottleneck. The 12% coarse-grained matching errors straddle both categories — they involve both perceptual imprecision (failing to notice the specific distinguishing features) and web-selection imprecision (choosing a nearby but incorrect search result).

Robustness of Ego2WebJudge to judge backbone choice (Tables 2, 4). While the paper does not frame this as a formal ablation, the instantiation of Ego2WebJudge with three different MLLM backbones (Qwen3-VL-Flash, Gemini-2.5 Pro, GPT-4o) serves as a robustness check on the evaluation framework itself. Key observations:

• All three judges produce consistent relative rankings of agents (Table 2). BU-Gemini-3-Flash is ranked first by all three judges. GPT-5.4 is ranked third by Qwen3-VL-Flash, fifth by Gemini-2.5 Pro, and fifth by GPT-4o — substantial disagreement, but the Claude-series and BU-Gemini-3-Flash maintain consistent ordinal positions.
• The absolute scores vary systematically: Qwen3-VL-Flash consistently gives higher scores than Gemini-2.5 Pro and GPT-4o (e.g., 57.2% for BU-Gemini-3-Flash vs. 48.2% and 51.4%, respectively). This suggests Qwen3-VL-Flash is a "lenient" judge, which is problematic for a benchmark — inflated scores would overestimate agent capability. The paper implicitly endorses GPT-4o and Gemini-2.5 Pro as preferred backbones given their closer alignment with human evaluation.
• The agreement rates with human judgment (Table 4) are highest for GPT-4o (84.0%) and lower for Gemini-2.5 Pro (80.8%) when used in Ego2WebJudge. The paper does not report Qwen3-VL-Flash's agreement rate, likely because its score inflation would produce low agreement despite high absolute scores.

Implicit ablation: agent capability vs. input modality. The performance gap between BU-GPT-4.1 (44.4% human-evaluated) and BU-Gemini-3-Flash (58.6%) is an implicit ablation on the value of native video understanding, since both agents use the Browser-Use framework but different MLLM backends with different visual processing capabilities (GPT-4.1 uses keyframes, Gemini-3-Flash uses raw video). The 14.2 percentage point gap, while not a controlled experiment (the MLLMs differ in many ways besides input modality), provides suggestive evidence that raw video input provides a substantial advantage over keyframe-based perception for egocentric grounding tasks.

Critical Assessment

Does the benchmark genuinely test visual grounding, or does it test web navigation with a visual "hint"?

The ablation in Table 5 provides strong evidence for genuine visual grounding dependence: performance drops from 48.2% to 4.4% when visual input is removed, confirming that the tasks cannot be solved from text instructions alone. However, the 23.6% success rate achieved with detailed captions (which provide rich textual descriptions of the video content) raises a subtle concern: a substantial fraction of tasks are solvable from high-quality text descriptions alone. This means the benchmark measures a gradient of visual grounding difficulty rather than a binary "requires vision" property. Some tasks genuinely require fine-grained visual perception unavailable in text (the 48.2% - 23.6% = 24.6 percentage point gap), while others are solvable when the video content is adequately described in words. This is not a weakness per se — real-world tasks span this spectrum — but it means that agent performance on Ego2Web is sensitive to the quality of video-to-text conversion when agents use caption-based input. An agent with a better captioning model could close some of the gap to video-native agents without actually improving its visual perception capabilities. The paper partially controls for this by using Gemini-3.1-Pro, a strong MLLM, for caption generation, but the 23.6% caption-based performance establishes a non-trivial "text-solvable" fraction of the benchmark.

Is the 40% "headroom" claim meaningful, or is 100% an unrealistic ceiling?

The paper consistently frames the gap between BU-Gemini-3-Flash's 58.6% and 100% as "room for improvement." However, 100% may not be a realistic ceiling for several reasons that the paper does not discuss. First, live websites change — a product that was available and findable during benchmark construction may become unavailable, change its listing page, or have its nutritional information updated. Second, some tasks may have inherent ambiguity (e.g., identifying "the same snack" when the video shows a partially obscured package). Third, CAPTCHA challenges, login requirements, and geo-restrictions can prevent task completion regardless of agent capability. The paper does not report what fraction of benchmark tasks are currently solvable by a perfectly capable agent operating against live websites. A human "oracle" success rate on the benchmark would have been informative — how often can a human, given the egocentric video and task instruction, successfully complete the web task? Without this, the 58.6% number is difficult to contextualize as a percentage of possible performance.

Are the agent comparisons fair given differing input modalities?

The paper's capability-adaptive design — giving each agent its best available input modality — is pragmatically justified but creates an apples-to-oranges comparison that conflates two variables: the agent's underlying reasoning capability and its ability to consume visual information. BU-Gemini-3-Flash's 58.6% vs. GPT-5.4's 30.6% under human evaluation reflects both the quality of the MLLM and the fidelity of its visual input. The paper does not provide an ablation where all agents receive the same captions, which would isolate reasoning capability from visual perception capability. Such an ablation would answer: "If we give all agents the same textual description of the video, how do their web-execution capabilities compare?" Without it, the performance hierarchy in Table 2 is a compound measure that cannot be decomposed into perception vs. action contributions.

The paper implicitly acknowledges this by discussing input modality as a finding (Section 5.2: "Impact of Visual Input Modality"), but this discussion treats modality as an explanation for performance differences rather than as a confound that limits the interpretability of agent rankings. A more rigorous design would report both "native modality" results (as currently done) and "common caption" results (all agents receive the same Gemini-3.1-Pro captions), enabling separate assessment of visual perception capability and web-execution capability.

Small sample sizes in domain-specific and error analyses.

The fine-grained domain analysis (Table 3) partitions 500 examples into five categories, with the "Others" category containing only 15 tasks. The reported success rates for this category (ranging from 0.0% for Claude 3.7 to 40.0% for BU-GPT-4.1) are based on very small denominators. A single success or failure changes the rate by approximately 6.7 percentage points, making these estimates highly unreliable. The paper acknowledges this implicitly by omitting low-count websites from Figure 4b ("we omit websites whose count<4"), but does not apply similar caution to the per-domain success rate reporting.

The error analysis (Section 5.3) is based on manual inspection of 50 trajectories — 10% of the benchmark. The reported failure-category percentages (36%, 18%, 16%, 12%, 18%) have standard errors of approximately 6–7 percentage points given this sample size. The qualitative claims about error distributions are directionally informative but should not be treated as precise estimates. A larger error analysis sample (or better, a systematic error taxonomy applied to all failure cases) would substantially strengthen the diagnostic claims.

Human evaluation protocol transparency.

Three annotators evaluate each agent's output with majority voting, but the paper does not report inter-annotator agreement metrics (Fleiss' kappa, pairwise agreement percentages, or any measure of annotator consistency). This is a significant omission for a benchmark that uses human evaluation as its gold standard. The 84% agreement between Ego2WebJudge and human evaluation is the headline validation number, but without knowing human inter-annotator agreement, we cannot determine whether the remaining 16% disagreement reflects judge error, human annotator noise, or both. If human annotators disagree with each other 10% of the time, then Ego2WebJudge's 84% agreement would represent near-perfect alignment with the achievable ceiling; if humans agree 98% of the time, the 84% represents a more substantial gap. The three-annotator design is appropriate, but the missing inter-annotator statistics prevent the reader from contextualizing the agreement rates.

No evaluation of task difficulty calibration or annotator bias.

The paper does not report whether tasks vary systematically in difficulty, or whether certain task types are consistently harder across all agents. While the domain-based breakdown (Table 3) provides a coarse difficulty signal, a per-task difficulty estimate (e.g., what fraction of agents succeed on each task) would reveal whether the benchmark has a healthy distribution of easy, medium, and hard tasks, or whether it is skewed toward uniformly difficult examples. If most tasks are failed by most agents, the benchmark provides limited discrimination between moderately capable and highly capable agents — it only distinguishes the very best from the rest. The 58.6% ceiling for the best agent suggests this is not a severe problem, but the distribution shape is unknown.

Additionally, the human verification process (Section 3.2) relies on annotators applying three criteria, but no measures of annotator calibration or consistency during the data generation stage are reported. Were the same annotators who verified the tasks also involved in the human evaluation of agent performance? If so, there is a potential for subtle bias — annotators who know the "correct" visual grounding for a task may apply stricter standards when evaluating agents than an annotator encountering the task fresh.

Missing baselines that would strengthen the paper.

Several baselines would have improved the diagnostic power of the experiments:

• A human oracle baseline: What success rate do humans achieve on Ego2Web given the same video and instruction? This would establish a realistic ceiling and contextualize the 58.6% agent performance.
• A text-only LLM baseline with oracle visual information: Provide the ground-truth visual annotation (e.g., "the product is French's Classic Yellow Mustard, 20 oz") directly to a text-only web agent. This would isolate web-execution capability from visual perception capability, answering: "If the agent could perceive perfectly, how well could it execute the web task?"
• A caption-uniformed comparison: Evaluate all agents using the same Gemini-3.1-Pro captions, removing input modality as a confound and enabling cleaner comparison of web-execution and reasoning capabilities.
• An oracle visual grounding + agent web execution pipeline: Pair a strong video understanding model (extracting the correct object identity from the video) with each agent's web-execution module, assessing whether the bottleneck is in perception or action.
• A static-website version of the benchmark: Evaluate agents on archived, frozen versions of the target web pages, eliminating the confound of live-website variability and enabling more reproducible comparisons across time.

Does the evaluation methodology support the claim that "true visual perception is essential"?

The Table 5 ablation (no visual input → 4.4%, captions → 23.6%, video → 48.2%) strongly supports the claim that visual information is necessary and that richer visual input improves performance. However, the "true visual perception" framing overstates what is demonstrated. The 48.2% video-based performance, while substantially better than captions, still leaves over half of tasks unsolved — the best agent with native video input fails on a majority of benchmark tasks. This suggests that current agents have not yet demonstrated that they can reliably use true visual perception for web grounding; they can use it partially and inconsistently. The ablation establishes that raw video provides information beyond captions that improves performance, but it does not establish that current agents are extracting that information effectively — the 48.2% ceiling may reflect poor utilization of available visual information rather than an upper bound on what video input can provide.

Generalizability concerns: single video source, single annotation pipeline, frozen in time.

All videos are sourced from Ego4D, which has known demographic and geographic biases (predominantly Western, English-speaking camera wearers performing everyday activities). The paper does not claim geographic or demographic diversity, but the single-source provenance means that Ego2Web inherently inherits Ego4D's biases — objects, brands, and activities that appear in the videos reflect the Ego4D collection distribution, and agents may perform differently on egocentric footage from different cultural contexts.

The task generation pipeline uses GPT-5 with a fixed website pool. As websites change (redesigns, shutdowns, geo-restriction changes), the "Web Feasibility" verification performed during construction becomes stale. The paper does not describe a mechanism for maintaining or updating the benchmark over time, which is a practical concern for a benchmark that uses live websites. A task that was feasible and verified in 2025 may become impossible in 2026 due to website changes unrelated to agent capability.

The benchmark's 500 examples, while substantial for a human-verified dataset, are modest by modern ML benchmark standards. The paper does not report the cost or time required to produce these 500 examples, which limits the community's ability to assess whether the semi-automatic pipeline can scale to thousands or tens of thousands of examples needed for training (as opposed to evaluation-only) purposes.

6. Limitations and Trade-offs

1. Cost of Difficulty/Visual-Grounding Assessment Is Unaccounted for in the Benchmark's Operational Model

The assumption or constraint. The Ego2Web benchmark assumes that the agent receives both the egocentric video and the task instruction as input, and that the agent must extract all necessary visual information from the video itself. However, the benchmark construction and evaluation pipeline relies on annotated visual evidence clips v — curated subsets of frames identified by human annotators that pinpoint exactly where the relevant objects or actions appear in the video. These annotations are used in two critical places: (1) during the human verification stage of data generation (Section 3.2), where annotators confirm that the task depends on information genuinely visible in the video, and (2) as input to Ego2WebJudge (Section 4), which uses the annotated keyframes as ground-truth visual evidence to determine whether the agent's web result matches the video content.

Neither the agent nor the automatic evaluator is required to produce these annotations — the agent must locate the relevant visual cues without being told where they are, and Ego2WebJudge receives them as external oracle information. This means that the cost of producing the visual evidence annotations is not amortized into any reported metric. The benchmark assumes these annotations exist, but they require human annotators to watch each three-minute video, identify the precise temporal segments containing task-relevant visual information, extract keyframes, and verify that those keyframes are sufficient for evaluation — a process that the paper does not quantify in terms of time, cost, or annotator effort.

The consequence. In any realistic deployment of an egocentric-video-grounded web agent, the system would need to either (a) autonomously identify which portions of a potentially long, unstructured video are relevant to a given task, or (b) rely on pre-annotated visual evidence segments. Option (a) is an unsolved perception problem — the agent must perform spatio-temporal grounding without knowing in advance what visual cues matter — and the benchmark does not evaluate this capability in isolation (agent failures could stem from either failing to find the relevant visual evidence or failing to interpret it correctly, and these are conflated in the success rate metric). Option (b) requires human annotation effort proportional to the number of tasks, making large-scale deployment infeasible. The paper's benchmark construction effectively assumes option (b) for evaluation purposes but option (a) for agent execution, creating an asymmetry: the benchmark is only evaluable because humans have done visual grounding work that agents are expected to perform autonomously at test time.

What evidence exists in the paper. The paper does not quantify the cost of producing the annotated visual evidence clips. Section 3.2 describes the human verification process but does not report annotator hours, per-example annotation time, inter-annotator agreement, or cost per verified example. Section 4 describes Ego2WebJudge's use of the annotated egocentric video evidence clip v as a core input, but does not discuss how this clip is produced or at what cost. Appendix A mentions that videos are selected from EgoSchema, which provides dense human narrations, but does not specify whether these narrations were used to produce the visual evidence annotations or whether separate annotation passes were required. The paper's acknowledgement of this limitation is implicit at best — there is no explicit statement that the annotation cost is a practical barrier to scaling or deploying the benchmark approach.

Mitigation status. Not addressed. The paper does not propose any method for automating the visual evidence annotation process, nor does it discuss whether the cost is amortizable across many agent evaluations (since the benchmark is static once built, the annotation is a one-time cost for benchmark construction rather than a recurring cost for evaluation). This is partially mitigated by the fact that Ego2Web is an evaluation-only benchmark — the annotations are produced once during construction and reused for all subsequent agent evaluations, making the per-evaluation cost zero. However, this does not address the deployment gap: a real system would need to perform visual grounding without oracle annotations, and the benchmark provides no signal about agents' ability to do so independently of their ability to interpret whatever visual evidence they happen to find.

2. Single Video Source (Ego4D) Constrains the Diversity of Visual Scenes, Activities, and Cultural Context

The assumption or constraint. All 500 egocentric videos in Ego2Web are sourced from Ego4D (Grauman et al., 2022), a large-scale dataset of first-person video collected primarily in North America and Europe, with camera wearers engaged in everyday activities like cooking, shopping, and home maintenance. The paper adopts EgoSchema's (Mangalam et al., 2023) selection criteria — three-minute clips with at least 30 human narrations — to further filter the Ego4D pool. This means the visual content of Ego2Web is constrained by both the geographic/demographic composition of Ego4D's participants and the activity types that EgoSchema deemed suitable for video reasoning tasks. The paper does not claim demographic or geographic diversity (Section 2.1 cites Ego4D's scale but not its representativeness), and the benchmark's domain distribution (Figure 4) reflects the prevalence of shopping and media-consumption activities in the source videos, not necessarily the distribution of tasks that users of egocentric AI assistants would perform in practice.

The consequence. An agent that performs well on Ego2Web may have learned (or been optimized for) visual recognition patterns specific to Western consumer environments — recognizing brands, products, and store layouts common in North American retail, identifying English-language text on signs and labels, or interpreting social interactions typical of Western cultural contexts. When deployed in different cultural or geographic settings — a market in Southeast Asia with unfamiliar products, a kitchen with different cooking tools and ingredients, a street scene with non-Latin signage — the agent's visual grounding capabilities may degrade in ways that Ego2Web cannot measure. This is not a hypothetical concern: egocentric AI assistants embedded in AR glasses or wearable cameras would be deployed globally, and their utility depends on recognizing objects, text, and activities across diverse cultural contexts. A benchmark that only tests recognition of objects and scenes from one cultural context provides an overly optimistic estimate of real-world readiness.

Additionally, the activity types captured in Ego4D (and thus Ego2Web) are primarily domestic, consumer, and leisure activities. The benchmark does not include professional or industrial contexts — a mechanic identifying a car part and ordering a replacement, a construction worker looking up specifications for a building material, a healthcare worker verifying medication information — that represent high-stakes, high-value deployment scenarios for egocentric agents. The absence of these contexts means the benchmark cannot assess whether current agents' visual grounding capabilities transfer to domains where errors have serious consequences.

What evidence exists in the paper. The paper is transparent about its data source: Section 3.2 states that the video pool is "sourced from public ego-centric video datasets (Grauman et al., 2022) that include both in-house and out-of-house recordings capturing first-person scenes across diverse contexts (e.g., household, shopping, travel, office)." Appendix A further explains the EgoSchema filtering criteria. However, the paper does not analyze the demographic, geographic, or activity-type distribution of the selected clips, nor does it discuss whether the 500-example sample is representative of any target deployment distribution. The domain-specific breakdown in Table 3 and Figure 4 describes web-task categories (E-Commerce, Media Retrieval, etc.), not the types of activities shown in the videos themselves. A reader cannot determine from the paper alone whether the egocentric videos show a narrow range of cultural contexts or a broad sample — the paper asserts diversity ("diverse contexts") without quantifying it.

Mitigation status. Not addressed. The paper does not propose extending the benchmark to include videos from more diverse geographic, cultural, or professional sources, nor does it discuss this as a limitation. The semi-automatic data generation pipeline (Section 3.2) is in principle compatible with videos from any source, so extending Ego2Web to additional egocentric video datasets (EPIC-KITCHENS, EgoExo4D, or purpose-collected footage from underrepresented contexts) would be technically feasible but is not attempted. The paper frames Ego2Web as a "first benchmark" for the egocentric-video-to-web-task category, implying that diversity limitations are acceptable for an initial effort, but does not explicitly bound the generalizability claims.

3. The Benchmark Is Static and Tied to Live Websites That Will Change, Degrading Reproducibility Over Time

The assumption or constraint. Ego2Web adopts the online evaluation setting (Section 3.1, Section 4), meaning that agents execute tasks against live, real-world websites (Amazon, YouTube, Wikipedia, Walmart, Google Maps, etc.) rather than against sandboxed, archived, or simulated versions. The paper argues that this is necessary for ecological validity — real websites have dynamic content, A/B testing, evolving interfaces, CAPTCHA challenges, and other properties that sandboxed environments cannot replicate. However, this design choice creates a fundamental tension with reproducibility and longevity: the specific web pages, product listings, search results, and interface layouts that existed when the benchmark was constructed (and when the human verification confirmed "Web Feasibility") will change over time. Products go out of stock, are delisted, or have their product pages redesigned; YouTube videos are removed or have their metadata updated; Wikipedia articles are edited; Google Maps listings are updated; CAPTCHA and authentication requirements change.

The consequence. Two consequences follow. First, cross-temporal comparisons degrade: an agent evaluated on Ego2Web in 2026 may face a different effective task distribution than an agent evaluated in 2027, not because the agents differ in capability but because the websites have changed. A task that was solvable in 2026 (the correct product page exists and is reachable) may become unsolvable in 2027 (the product is discontinued), making the 2027 agent's lower success rate a reflection of environmental drift rather than capability regression. Second, exact reproduction becomes impossible: a researcher attempting to reproduce the paper's reported success rates in a later year will evaluate against different website states, and the resulting numbers will not be directly comparable to the published results. This is not a hypothetical concern — web-agent benchmarks that rely on live websites (WebVoyager, Online-Mind2Web) have documented this issue, and it is an inherent property of any evaluation against non-stationary environments.

The paper's reporting of success rates as point estimates (e.g., "BU-Gemini-3-Flash achieves 58.6% SR under human evaluation," Table 2) implicitly assumes that the task distribution is fixed, which is only true at the moment of evaluation. As the gap between evaluation and reproduction widens, these numbers become increasingly historical artifacts rather than reproducible benchmarks.

What evidence exists in the paper. The paper acknowledges this issue only indirectly. Section 2.2 discusses the evolution of web-agent benchmarks from "static, pre-defined sandbox" to "live, real-world websites" and characterizes the online setting as more realistic. Section 3.2's Web Feasibility criterion — one of the three human verification checks — confirms that tasks are executable on the specified websites at the time of verification, but does not establish that they remain executable indefinitely. The paper does not report what fraction of tasks, if re-evaluated today, would still be feasible, nor does it propose a mechanism for detecting or handling website drift. No "snapshot date" for the evaluation is reported — we know when the paper was written (March 2026) but not exactly when the agent evaluations were conducted, making it impossible to contextualize the results relative to website states.

Mitigation status. Not addressed. The paper does not propose archiving website states (which is technically challenging and legally uncertain), periodically re-verifying tasks, providing a "feasibility score" per task that degrades over time, or maintaining a frozen mirror of the benchmark websites. The choice of online evaluation is defended on ecological validity grounds, but the trade-off with reproducibility is not discussed. Future work could address this by (a) providing timestamped snapshots of expected web states alongside the benchmark, (b) implementing a continuous monitoring system that flags tasks whose websites have changed, or (c) supplementing the online evaluation with a parallel offline evaluation on archived pages — but none of these are attempted.

4. Input Modality Asymmetry Confounds Agent Capability Comparisons, and the Paper Does Not Isolate Perception from Action

The assumption or constraint. The paper evaluates six agents with fundamentally different visual input capabilities: Gemini-based agents receive raw video, GPT-4o-based agents receive sampled keyframes, and Claude-series and GPT-5.4 agents receive text captions (generated by Gemini-3.1-Pro) because they cannot process video or keyframes natively (Section 5.1, Implementation Details). This capability-adaptive design — giving each agent its best available input format — is pragmatically motivated: forcing all agents to use captions would obscure the benefit of native video understanding, and forcing all agents to use video would exclude important baseline agents. However, this creates a fundamental confound in the performance comparisons: the observed differences between agents reflect both underlying reasoning capability and visual input fidelity, and these two factors cannot be separated in the reported numbers.

The consequence. When the paper reports that BU-Gemini-3-Flash achieves 58.6% human-evaluated success rate vs. GPT-5.4's 30.6% (Table 2), the reader cannot determine how much of this ~28 percentage point gap is attributable to (a) Gemini-3-Flash being better at reasoning about and acting on visual information vs. (b) Gemini-3-Flash receiving raw video while GPT-5.4 receives lossy text captions. The Table 5 ablation (raw video → 48.2% vs. captions → 23.6% for BU-Gemini-3-Flash) demonstrates that input modality alone can account for a ~25 percentage point gap within the same agent, which is comparable in magnitude to the entire gap between the strongest and weakest agents in Table 2. This means the agent rankings in Table 2 could be largely or entirely explained by input modality rather than by differences in web-execution or reasoning capability.

This confound also undermines the paper's diagnostic claim that "visual grounding" is the primary bottleneck. If GPT-5.4 were given raw video input (which its architecture may handle in the future), its performance might rise substantially, and the bottleneck diagnosis might shift toward web-execution capability instead. The current experimental design cannot distinguish "agent X is bad at visual grounding" from "agent X receives impoverished visual input" — both produce the same symptom (low success rates on visually-grounded tasks) but require different remedies (better visual encoders vs. better reasoning).

What evidence exists in the paper. The paper is aware of this confound and discusses it in Section 5.2 (under "Impact of Visual Input Modality"), but frames it as a finding rather than a limitation of the experimental design. The paper states:

"Agents built on GPT-5.4 and the Claude series consistently underperform across both human and automatic evaluations. A key reason is that these agents cannot directly access raw video inputs in computer-use mode. Instead, egocentric videos are first converted into textual captions, which inevitably leads to information loss, especially for fine-grained spatial-temporal cues and subtle visual interactions."

This paragraph acknowledges the modality gap but does not propose an experimental design to control for it. The paper does not report a common-caption baseline where all agents receive the same Gemini-3.1-Pro captions, which would equalize input modality and isolate reasoning/execution capability. Similarly, the paper does not report a common-keyframe baseline for the subset of agents that can process images. The Table 5 ablation provides modality-effect estimates for one agent (BU-Gemini-3-Flash) but does not provide comparable estimates for other agents, making it impossible to calibrate cross-agent comparisons.

Mitigation status. Partially addressed through discussion but not through experimental design. The paper's explicit acknowledgment of the modality effect is more transparent than ignoring it, but the experimental design leaves the confound unresolved. A reader who wants to know "which agent is actually best at the reasoning and web-execution aspects of this task, independent of visual input quality" cannot answer that question from the reported data. The paper could have included a caption-uniformed evaluation as a secondary metric, which would have provided a cleaner (if less ecologically valid) comparison of agent reasoning capabilities — but this is absent.

5. The Benchmark's 500-Example Size and Single-Task-Family Focus Limit Statistical Power and Domain Coverage

The assumption or constraint. Ego2Web consists of 500 video–task pairs, all drawn from a single task paradigm: egocentric video → web action. Within this paradigm, the tasks span five domain categories (E-Commerce, Media Retrieval, Knowledge Lookup, Local/Maps, Others), but these categories are not balanced — E-Commerce dominates at 230 examples (50.3%) while Local/Maps and Others together account for only 46 examples (9.2%) (Figure 4a, Table 3). The paper treats success rate on the full 500-example set as the primary evaluation metric and reports per-domain success rates as secondary analysis.

The consequence. The small and imbalanced dataset creates two problems. First, statistical power is limited for domain-specific claims. The "Others" category contains only 15 tasks, meaning that reported success rates in this category (ranging from 0.0% for Claude 3.7 to 40.0% for BU-GPT-4.1, Table 3) have standard errors of approximately 7–13 percentage points. A single additional success or failure changes the rate by ~6.7 percentage points, making these estimates too noisy to support reliable comparisons between agents or between domains. The Local/Maps category (31 tasks) is only marginally better. Even the largest category (E-Commerce, 230 tasks) has a standard error of ~3–4 percentage points on the per-agent success rate, meaning that differences between agents of less than ~8 percentage points may not be statistically reliable — yet the paper reports all success rates to one decimal place without confidence intervals, implying a precision the data does not support.

Second, the single-task-paradigm design limits the scope of capability assessment. Ego2Web tests one specific integration pattern: perceive objects/actions in egocentric video, then perform information retrieval or e-commerce actions on the web. It does not test other plausible integration patterns that real-world egocentric agents would need: perceive a physical object and manipulate a digital representation (e.g., 3D model alignment for augmented reality), perceive a scene and create web content (e.g., write a review based on a physical experience), perceive a document and translate or summarize it with web-augmented context, or perceive a multi-step physical process and schedule or coordinate corresponding web actions. The benchmark's task diversity (Figure 4a) captures surface-level variation in web domains but not deep variation in the type of perception–action integration required. All tasks follow the same pattern: identify something → find it on the web → report information. An agent that excels at this pattern but fails at other integration patterns would receive an inflated capability assessment from Ego2Web.

What evidence exists in the paper. The paper reports the task distribution explicitly in Figure 4a and Table 3, so the imbalance is transparent. Appendix A provides a more detailed website-level breakdown (Table 6) showing the number of tasks per website. The paper does not report confidence intervals, standard errors, or significance tests for any of the reported success rates or agreement rates. The per-domain analysis in Table 3 reports averages across agents but does not discuss the statistical reliability of these estimates. The error analysis (Section 5.3) is based on 50 manually inspected examples, a sample size the paper acknowledges is limited (50/500 = 10%) but does not discuss the resulting uncertainty in the reported error-category percentages.

Mitigation status. The paper acknowledges the benchmark's scope implicitly by framing it as "the first benchmark" for this task category (Section 1, contributions), implying that size and diversity limitations are acceptable for an initial effort that is expected to be extended by the community. The release of the data generation pipeline (Section 3.2) and the provision of code and prompts (Appendix B) partially mitigates the scalability concern by enabling other researchers to generate additional video–task pairs, though this does not address the domain-coverage limitation — extending to new integration patterns would require pipeline modifications, not just more data from the same distribution. The paper does not discuss the statistical power implications of the 500-example size or the per-category imbalances, nor does it recommend minimum sample sizes for reliable domain-specific evaluation.

6. The Ego2WebJudge Evaluator's 84% Agreement Ceiling Leaves a 16% Residual Gap That Is Not Characterized or Bounded

The assumption or constraint. The paper validates Ego2WebJudge by comparing its binary success/failure verdicts against human majority-vote judgments, reporting agreement rates of 84.0% (GPT-4o backbone) and 80.8% (Gemini-2.5 Pro backbone) averaged across six agents (Table 4). The paper treats this as validation that Ego2WebJudge "provides a reliable and scalable evaluation protocol" (Section 4) and uses it as the primary evaluation mechanism for the domain-specific breakdown (Table 3) and the ablation study (Table 5). However, the paper does not characterize the nature or distribution of the remaining 16% of cases where Ego2WebJudge disagrees with human annotators. The reader does not know whether the disagreements are concentrated in particular task categories, particular agents, or particular types of errors — and whether the disagreements systematically bias the reported success rates upward or downward.

The consequence. Two distinct risks arise from the unexplained residual disagreement. First, systematic bias could inflate or deflate reported agent performance. If Ego2WebJudge is systematically more lenient than human annotators — marking ambiguous or partially complete tasks as successes that humans would judge as failures — then the reported success rates in Table 3 (which use Ego2WebJudge with Gemini-2.5 Pro) may overestimate true agent capability. The observation in Table 2 that Qwen3-VL-Flash gives consistently higher scores than the other judge backbones (e.g., 57.2% for BU-Gemini-3-Flash vs. 48.2% for Gemini-2.5 Pro and 51.4% for GPT-4o) demonstrates that different judge instantiations can produce meaningfully different absolute scores — if the recommended GPT-4o backbone inherits a similar leniency bias relative to humans, the 84% agreement masks a consistent over-estimation of performance. Conversely, if Ego2WebJudge is systematically stricter, agent performance is under-estimated. The paper does not report whether the 16% of disagreements are balanced (equal numbers of false positives and false negatives) or skewed (predominantly one direction), making it impossible to determine the direction or magnitude of bias.

Second, unknown failure modes of the automatic evaluator could lead to misdiagnosis of agent failures. If Ego2WebJudge disproportionately disagrees with humans on tasks requiring fine-grained visual discrimination (e.g., distinguishing similar product variants), then agent performance on the most visually-demanding tasks may be mis-estimated, and the domain-specific analysis in Table 3 may misrepresent which domains are hardest for agents. The error analysis in Section 5.3 is based on manual inspection of agent failures, not evaluator failures — it tells us why agents fail but not why the automatic judge sometimes disagrees with humans about whether a failure occurred.

What evidence exists in the paper. The agreement rates in Table 4 are reported per agent and averaged across agents, but the paper does not report per-domain agreement rates, confusion matrices (how often the judge says "success" when humans say "failure" and vice versa), or qualitative examples of disagreement cases. The paper notes that agreement is higher for weak agents (Claude 3.7: 86.0% with GPT-4o) than for strong agents (BU-Gemini-3-Flash: 80.4% with GPT-4o), attributing this to the difficulty of evaluating nuanced, partially successful trajectories. However, this observation does not explain how the evaluator errs on these harder cases — does it miss subtle visual mismatches (false positives) or does it over-penalize minor web-execution errors that humans would overlook (false negatives)? The paper does not report human inter-annotator agreement (e.g., Fleiss' kappa), so the reader cannot determine what fraction of the 16% disagreement ceiling is attributable to evaluator error vs. inherent human annotator noise. If human annotators disagree with each other 8% of the time, then Ego2WebJudge's 84% agreement represents near-perfect alignment with the achievable ceiling; if humans agree 98% of the time, the 84% represents a 14-percentage-point gap from achievable performance.

Mitigation status. Not addressed. The paper validates Ego2WebJudge at the aggregate level (average agreement rate) but does not characterize the residual error. The "Strict Evaluation Principle" in the judge's prompt (Appendix B) — biasing toward failure in ambiguous cases — suggests a deliberate asymmetry (false negatives preferred over false positives), but the paper does not verify empirically that the judge actually implements this principle or measure its false-positive vs. false-negative rates. The paper's recommendation to use GPT-4o as the preferred judge backbone implicitly acknowledges that judge quality matters, but does not bound the remaining evaluation error or provide guidance on what level of agreement is sufficient for reliable benchmark use. Future work could address this by (a) reporting full confusion matrices, (b) analyzing disagreement cases to identify systematic evaluator failure modes, (c) reporting human inter-annotator agreement to establish an achievable ceiling, and (d) calibrating reported agent success rates to account for evaluator bias — none of which are done in the current paper.

7. Implications and Future Directions

How This Work Changes the Landscape

Ego2Web does not introduce a new model architecture, training algorithm, or agent design. Its contribution is infrastructural and diagnostic: it defines a previously nonexistent task category, builds the tooling to evaluate it, and uses that tooling to reveal concrete, measurable limitations in state-of-the-art systems. This kind of contribution — benchmark construction as primary research output — reshapes the field not by providing a new capability but by making a capability gap visible and measurable, which redirects research attention and resources toward the gap. The paper's impact, if it materializes, will be in changing what the community considers a "capable multimodal agent" to mean.

The paper creates a new evaluation category that bridges two disconnected research programs. Prior to Ego2Web, egocentric video understanding and web-agent execution were separate fields with separate benchmarks, separate leaderboards, and separate researcher communities. Ego2Web demonstrates that the integration between perception and action — the ability to see something in the physical world and then do something about it on the web — is a distinct capability that neither field's existing benchmarks can measure. An agent that achieves state-of-the-art on EgoSchema (demonstrating strong video reasoning) and WebVoyager (demonstrating strong web navigation) could still fail catastrophically on Ego2Web because EgoSchema does not require translating perception into action and WebVoyager does not require grounding actions in real-world visual evidence. The paper's Table 1 makes this gap explicit through its four-column taxonomy, and by filling the ✓, ✓, ✓, ✓ row with a concrete, executable benchmark, the paper defines a new capability axis that future agents will be evaluated against.

The magnitude of this shift depends on whether the community adopts Ego2Web as a standard evaluation criterion. The paper's release of code, prompts, and the data generation pipeline lowers adoption barriers, but the benchmark's reliance on live websites and human-verified annotations creates maintenance overhead that may slow uptake compared to static, self-contained benchmarks. If adopted, Ego2Web would complement rather than replace existing benchmarks — an agent would still be evaluated on WebVoyager for web-navigation competence and on EgoSchema for video-reasoning competence, with Ego2Web testing the integration layer that connects them. This is more of a reframing than a paradigm shift: it adds a new dimension to the evaluation landscape without rendering existing benchmarks obsolete.

The paper's primary diagnostic finding — that visual grounding, not web execution, is the dominant bottleneck — redirects research prioritization for multimodal agents performing grounded tasks. The evidence comes from three converging sources: (1) the error analysis showing that 54% of failures from the strongest agent involve perceptual errors (object misidentification, temporal misunderstanding) compared to 16% involving web-retrieval failures (Section 5.3), (2) the input modality ablation showing that raw video provides a 2× performance improvement over detailed captions for the same agent (Table 5: 48.2% vs. 23.6%), demonstrating that fine-grained visual information unavailable in text descriptions is the rate-limiting factor, and (3) the domain difficulty hierarchy in which Knowledge Lookup tasks requiring coarse-grained visual recognition are substantially easier than E-Commerce tasks requiring fine-grained product identification (75.0% vs. 38.2% for the best agent, Table 3), even though the web-navigation complexity is comparable across these domains.

Before Ego2Web, the web-agent research community had focused heavily on improving action execution — better planning algorithms, more robust error recovery, more efficient search strategies. The paper's findings suggest that for physically-grounded agents, improving visual perception and fine-grained grounding may yield larger marginal gains than further optimizing web-navigation capabilities. This is not an obvious conclusion from existing benchmarks: WebVoyager and VisualWebArena evaluate web-navigation competence with visual inputs that are themselves web content (screenshots, DOM trees), where the primary challenge is understanding UI layouts and page structures, not recognizing real-world objects. An agent optimized for WebVoyager might invest heavily in click-prediction accuracy and form-filling robustness while neglecting video-understanding capabilities, and Ego2Web reveals that this optimization path would leave a critical capability gap for physically-grounded deployment scenarios.

If this diagnosis is correct and generalizable, it implies that research investments should shift toward better video-native multimodal encoders (architectures that can process dense temporal visual streams without information loss), fine-grained visual grounding (distinguishing subtle product variants, reading small text on labels, tracking objects through occlusions), and temporal reasoning (correctly identifying the "fourth item picked up" or the "second exercise demonstrated"), rather than toward more sophisticated search algorithms, better DOM parsing, or more elaborate action-planning hierarchies. The paper does not make this argument explicitly, but it is the natural implication of the bottleneck analysis.

The paper reconciles no prior contradictions because the task category it defines did not previously exist. This is important to state clearly: Ego2Web does not resolve a debate between competing approaches, explain conflicting results in the literature, or unify disparate findings under a common framework. The two research programs it connects — egocentric video understanding and web-agent execution — were not in conflict; they were simply disconnected. The paper's contribution is to define the connection point and demonstrate that neither program's existing methods perform well at it, establishing a new research target rather than resolving an existing tension. This distinguishes Ego2Web from papers whose primary contribution is conceptual synthesis (e.g., showing that two seemingly contradictory findings are both true under different conditions, or that a common mechanism explains diverse empirical phenomena).

The Ego2WebJudge methodology establishes a template for multimodal, grounded evaluation that generalizes beyond this benchmark. The core design principle — that automatic evaluation of grounded agents must have access to the same grounding evidence that defines task success, and that the judge must perform cross-modal alignment between physical-world evidence and digital-world output — is not specific to egocentric video and web tasks. Any future benchmark that connects real-world perception to digital action (an agent that watches a cooking video and orders ingredients, observes a mechanical repair and purchases replacement parts, listens to a conversation and schedules a related meeting) will face the same evaluation challenge: how to verify that the agent's output is grounded in the original perceptual evidence without requiring human annotators for every evaluation. Ego2WebJudge provides a reusable architecture — key-point extraction, relevance-based evidence filtering, multimodal alignment judgment with explicit grounding rules — that can be adapted to other perception–action domains by replacing the specific perceptual modality and action environment while preserving the evaluation structure.

The finding that Ego2WebJudge achieves 84% agreement with human judgment using GPT-4o (Table 4), substantially outperforming prior web-agent evaluators like WebJudge (78.4%) and WebVoyager (74.7%), is a practical validation of this approach. The 5.6–9.3 percentage point improvement is concentrated in the cases where evaluation is hardest — strong agents producing nuanced, partially successful trajectories — demonstrating that the visual grounding component matters precisely when evaluation is most challenging. This establishes LLM-as-a-Judge with explicit grounding evidence as the preferred paradigm for evaluating grounded agents, and makes WebJudge-style trajectory-only evaluation look insufficient for tasks with external perceptual dependencies.

The paper implicitly argues that input modality is a first-class design variable for multimodal agents, not a fixed property of a benchmark. By evaluating agents with their best available input format — raw video for Gemini-native agents, keyframes for GPT-4o-based agents, text captions for Claude and GPT-5.4 — the paper treats input modality as a legitimate axis of agent comparison rather than forcing a common denominator. This is a methodological choice with implications for benchmark design: it acknowledges that different agents have fundamentally different perceptual capabilities, and that measuring an agent's performance in its optimal configuration is more informative than measuring all agents in a degraded common configuration. The finding that the input modality effect (raw video vs. captions, ~25 percentage point gap within the same agent, Table 5) is comparable in magnitude to the entire performance gap between the strongest and weakest agents in Table 2 (~32 percentage points) validates this choice — forcing all agents to use captions would compress the performance range and obscure the genuine advantage that video-native architectures provide.

However, this design choice also means that Ego2Web conflates two variables — underlying reasoning capability and visual input fidelity — in its agent comparisons. The paper does not provide a common-caption baseline that would isolate reasoning capability from perception capability across agents. Future benchmark designs might adopt a hybrid approach: report both "native modality" performance (reflecting real-world deployment configurations) and "common modality" performance (enabling cleaner capability comparisons), making both the ecological validity argument and the controlled experiment argument simultaneously.

Follow-Up Research This Work Enables

Human oracle baseline to establish a realistic performance ceiling. The paper reports that the strongest agent (BU-Gemini-3-Flash) achieves 58.6% success rate under human evaluation, framing the remaining ~41% as "room for improvement" toward an implicit 100% ceiling. However, 100% may not be achievable even for perfect agents because live websites change, products become unavailable, CAPTCHA challenges block automated access, and some tasks may have inherent ambiguity in the video evidence. A human oracle study — where human participants are given the same egocentric videos and task instructions and asked to complete the web tasks manually — would establish the true achievable ceiling and contextualize the 58.6% number. If humans achieve 85%, the ~27 percentage point gap represents the headroom for algorithmic improvement; if humans achieve 65%, the gap is smaller and may partly reflect inherent task difficulty rather than agent limitations. This experiment would also reveal whether human failures cluster in the same domains as agent failures (E-Commerce, Local/Maps) or follow a different pattern, providing diagnostic information about whether the task difficulty is perceptual, web-navigational, or inherent to the video–task pairing. The paper's existing human evaluation infrastructure — three annotators per agent output with majority voting — could be adapted for this study by having annotators perform the tasks themselves rather than evaluate agent outputs.

Caption-uniformed agent comparison to isolate perception from reasoning. The paper's Table 2 rankings conflate agent reasoning capability with visual input fidelity because different agents receive different input modalities (raw video, keyframes, captions). A controlled experiment where all six agents receive the same Gemini-3.1-Pro captions — the format currently used for Claude-series and GPT-5.4 agents — would equalize input modality and isolate differences in web-execution planning, search strategy, error recovery, and instruction-following. This would answer the question: "If we give all agents the same textual description of the video content, how do their web-execution capabilities compare?" The result would decompose the current performance gaps into a "perception gap" (the difference between caption-based and native-modality performance for each agent) and a "reasoning gap" (the difference between agents when receiving identical captions). If BU-Gemini-3-Flash's large advantage in Table 2 largely disappears under common captions, the paper's diagnostic emphasis on visual perception as the primary bottleneck would be strengthened; if the advantage persists, it would suggest that the Gemini-3-Flash backend also provides superior web-execution or reasoning capabilities independent of visual input quality. The paper already has the caption-generation infrastructure (Gemini-3.1-Pro captions are used for Claude and GPT-5.4) and evaluation infrastructure (Ego2WebJudge), making this experiment straightforward to conduct.

Combining a state-of-the-art video understanding model with existing web agents to isolate and quantify the perception bottleneck. The paper's error analysis (Section 5.3) and modality ablation (Table 5) suggest that visual perception errors, not web-execution errors, dominate the failure profile. A direct test of this hypothesis would pair a strong, standalone video understanding model with each web agent's execution module: the video model extracts the task-relevant object identity, attributes, and temporal grounding from the egocentric video and passes this structured information (e.g., "the fourth object picked up is a bag of Harvest Cheddar Sun Chips, 1 oz size") to the web agent, which then performs the web task using this oracle-quality visual information. If agent performance increases substantially with oracle visual grounding — especially for agents that originally received captions or keyframes — this would confirm that perception is the dominant bottleneck and that investment in better video understanding would yield the largest marginal gains. If performance improves only modestly, the bottleneck may lie in cross-modal retrieval (the agent knows what to look for but cannot find it on the web) or web-execution competence. The paper already provides annotated visual evidence clips v for each task (used by Ego2WebJudge), which contain the ground-truth visual information needed for task solving. These annotations could serve as the oracle visual grounding input, making this a low-cost experiment using existing benchmark infrastructure.

Extending Ego2Web to diverse egocentric video sources to test cultural and geographic generalization. The current benchmark uses videos exclusively from Ego4D, which has known Western, English-language biases in its participant demographics, activity types, and visual environments. An agent that performs well on Ego2Web may have learned visual recognition patterns specific to North American consumer contexts — recognizing brands, products, and store layouts common in Western retail — and may fail when deployed in different cultural settings. A systematic extension study would add videos from geographically and culturally diverse egocentric datasets: EPIC-KITCHENS (UK-based cooking activities with different ingredients and kitchen layouts than Ego4D's predominantly US cooking scenes), EgoExo4D (which includes participants from multiple countries performing skilled activities), and purpose-collected footage from underrepresented contexts (markets in Southeast Asia, street scenes in South America, home environments in Africa). The paper's semi-automatic data generation pipeline is designed to be video-source-agnostic — it takes any egocentric video pool, generates captions with Qwen3-VL, and produces tasks with GPT-5 — so extending Ego2Web to new video sources would require data collection and human verification but no pipeline modification. The key measurement would be the performance drop of current agents on non-Western videos compared to Ego4D-sourced videos, quantifying the generalization gap and motivating investment in geographically diverse training data for egocentric perception models.

Static-website snapshot evaluation to disentangle agent capability from website drift. The paper's online evaluation design means that website changes — product delistings, interface redesigns, CAPTCHA policy changes — will degrade the reproducibility of reported success rates over time. An agent evaluated in 2027 might achieve lower success rates than the same agent evaluated in 2026, not because the agent is worse but because the websites have changed. A complementary evaluation approach would archive or simulate the web pages that existed at the time of benchmark construction — capturing the specific product pages, search results, and information panels that human annotators verified as "Web Feasible" — and evaluate agents against these frozen snapshots. This would provide a reproducibility anchor: future agents could be evaluated on both the live web (measuring real-world deployment capability) and the frozen snapshot (measuring capability net of website drift, enabling clean comparison against published baselines). The technical challenge is non-trivial — archiving interactive, dynamic web pages requires more than static HTML snapshots — but web archive projects and headless browser recording tools make this feasible. The paper's provision of the exact task instructions, target websites, and verified visual evidence for each task means that the expected web state is partially specified, providing a starting point for snapshot construction. This experiment would also reveal what fraction of the "room for improvement" gap between 58.6% and the oracle ceiling is attributable to website drift versus genuine agent limitations.

Failure-contingent adaptive strategy allocation across domains. The paper identifies a clear domain-difficulty hierarchy (Knowledge Lookup > Media Retrieval > Local/Maps > E-Commerce, Table 3) but treats all tasks within a domain uniformly. A more fine-grained analysis would investigate whether within-domain task characteristics — the number of required visual cues, the specificity of object identification, the number of web-navigation steps, the presence of temporal ordering requirements — predict agent success rates. This would enable a difficulty-calibrated evaluation where agent performance is reported not only by domain but by task difficulty level, providing more nuanced discrimination between agents. If, for example, BU-Gemini-3-Flash's advantage over other agents is largest on tasks requiring temporal reasoning (identifying the "fourth" item picked up) but smallest on tasks requiring static object recognition (identifying a brand from a logo), this would provide actionable guidance about which visual reasoning capabilities to invest in. The paper already has the necessary data: the task generation pipeline records "must match" visual cues, timestamps, and difficulty estimates for each task (Section 3.2, Stage 2 of the LLM generation prompt). Correlating these task-level features with agent success/failure patterns across all 500 examples would reveal which task characteristics drive the performance gaps, going beyond the coarse domain-level analysis in Table 3.

Practical Applications and Downstream Use Cases

Wearable AI assistants (AR glasses, smart glasses) for shopping and information retrieval. The most direct application of Ego2Web's task paradigm is in augmented reality glasses that can see what the wearer sees and act on that visual information. A user walking through a store, seeing a product they like but want to price-compare, could ask their AR assistant: "Find this product on Amazon and tell me if it's cheaper online." The assistant would need to identify the product from the egocentric video feed, navigate to Amazon, locate the matching listing, and report the price comparison — exactly the capability Ego2Web evaluates. The paper's results suggest that current state-of-the-art agents (58.6% success rate for the best system, Table 2) are not yet reliable enough for this use case, where a wrong product match could lead to purchasing the wrong item. However, the domain-specific analysis (Table 3) shows that the best agent achieves 75.0% on Knowledge Lookup tasks — identifying a university from a logo on a backpack and finding its Wikipedia article — which suggests that information retrieval from coarse visual cues is approaching usable reliability. A phased deployment could start with Knowledge Lookup-style tasks (identifying landmarks, looking up brands, finding information about visible objects) while E-Commerce tasks requiring fine-grained product matching await further improvement.

Accessibility applications for visually impaired users. Egocentric video-grounded web agents could serve as an assistive technology layer for users who can capture first-person video (via smartphone camera or wearable device) but have difficulty reading text, recognizing products, or navigating web interfaces. A user could point their camera at a medication bottle and ask: "Order a refill of this prescription from my pharmacy's website," or point at a food package and ask: "Look up whether this contains allergens." The agent would need to read the label text from the egocentric video, identify the product and relevant details (dosage, active ingredients), and execute the appropriate web action. Ego2Web's task design — particularly the Knowledge Lookup and E-Commerce categories — directly mirrors this use case. The 75.0% Knowledge Lookup success rate for the best agent is encouraging for medication identification and information retrieval, but the 38.2% E-Commerce rate indicates that purchase-related tasks remain unreliable. The primary barrier is the fine-grained visual grounding requirement: distinguishing between similar product variants (different dosages, different formulations) where an error has health consequences, not just inconvenience. This application would benefit from a confidence-calibrated agent design where the system explicitly signals when visual evidence is insufficient for reliable identification, deferring to human judgment rather than guessing.

Self-improvement data generation pipelines for grounded agent training. The paper's semi-automatic data generation pipeline (Section 3.2) is a contribution in its own right that can be used to generate training data for fine-tuning multimodal agents on grounded web tasks. The pipeline — video captioning with an MLLM, task generation with an LLM, human verification for quality — can be run at scale on any egocentric video corpus to produce thousands or tens of thousands of video–task pairs. These pairs could be used to fine-tune a base MLLM to perform the full perception-to-action pipeline end-to-end: given an egocentric video and a task instruction, directly predict web actions or structured queries. The paper demonstrates that current agents struggle with this task (58.6% ceiling), but supervised fine-tuning on a large corpus of successful trajectories — generated by combining oracle visual grounding with strong web agents, or by human demonstration — could substantially improve performance compared to zero-shot prompting. The pipeline's modularity (separate captioning and generation stages) means that as better MLLMs and LLMs become available, the quality of generated training data improves automatically without re-annotation. The primary cost is the human verification stage, which the paper does not quantify but which would need to be scaled (perhaps through crowd-sourcing or automated quality filters) for training-scale data production. The 84% agreement between Ego2WebJudge and humans (Table 4) suggests that the judge itself could serve as an automatic quality filter, flagging low-quality generated tasks for human review while accepting high-confidence examples, reducing human annotation cost.

When to Prefer This Method

The paper does not propose a method in the conventional sense — it is a benchmark, not an algorithm or system design. There is no "Ego2Web method" to prefer over alternatives. The paper does not position itself against competing approaches to egocentric web-agent evaluation because no such approaches existed before this work. The choice facing a practitioner or researcher is not "should I use the Ego2Web method or the X method?" but rather "should I evaluate my agent on Ego2Web, and if so, what does the evaluation tell me that other benchmarks do not?"

The decision rule is straightforward and follows from the paper's task definition:

• Include Ego2Web in your agent's evaluation suite if the agent is intended for deployment scenarios where users will provide egocentric visual input (via AR glasses, wearable cameras, smartphone video) and expect the agent to perform web tasks based on what it sees. The benchmark evaluates the integration of visual perception and web execution — a capability that no existing benchmark measures — and the 40% headroom between current state-of-the-art (58.6%) and oracle performance (100%) makes it a discriminating evaluation for near-term progress. The domain-specific breakdown (Table 3) provides diagnostic information about which task types (E-Commerce, Media Retrieval, Knowledge Lookup, Local/Maps) your agent handles well or poorly.

• Use Ego2WebJudge (preferably with GPT-4o as the judge backbone) for scalable evaluation once you have established baseline human-evaluated performance on a subset of the benchmark. The 84% agreement with human judgment (Table 4) is high enough for reliable relative comparisons between agent versions, though the remaining 16% disagreement — whose direction (false positives vs. false negatives) and domain distribution is not characterized — means that small absolute differences between agents should be interpreted cautiously. The paper's strict evaluation principle (bias toward marking uncertain cases as failures) makes Ego2WebJudge a conservative evaluator, so success rates should be treated as lower bounds on true capability.

• Do not rely solely on Ego2Web if your agent is intended for text-only or screenshot-only deployment (where the user provides no egocentric visual context), or if your agent's primary value proposition is web-navigation efficiency and robustness rather than visual grounding. Ego2Web evaluates a specific capability — perception–action coupling grounded in real-world visual evidence — that many current web agents do not claim to possess and do not need for their target use cases. WebVoyager, WebArena, or OSWorld are more appropriate for evaluating pure web-navigation competence, and EgoSchema or EgoThink are more appropriate for evaluating video understanding in isolation. Ego2Web is complementary to, not a replacement for, these existing benchmarks.