Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification

ArXiv: 2603.26648

🎯 Pitch

Even the best coding agents, like Claude-Opus-4.5, achieve a Visual Score of only 38.4 when tasked with building full-stack websites from visual prototypes, with performance degrading sharply on smaller device form factors. This benchmark reveals that current multimodal agents systematically break down in cross-page coordination and persistent state reasoning.

1. Executive Summary

This paper introduces Vision2Web, a hierarchical benchmark for evaluating multimodal coding agents on end-to-end visual website development, spanning three progressively harder task levels—static webpage generation, interactive multi-page frontend reproduction, and long-horizon full-stack website construction—curated from real-world websites into 193 tasks with 918 prototype images and 1,255 test cases. To enable reproducible, implementation-agnostic evaluation, the paper proposes a workflow-based agent verification paradigm combining a GUI agent verifier for functional correctness (executing expert-designed test workflows with explicit guided actions and validation criteria) and a VLM-based judge for visual fidelity (performing component-level comparisons against UI prototypes), achieving 87.2% node-level agreement with human annotations for the GUI verifier and a median Spearman correlation of 0.80 for the VLM judge. Evaluating eight state-of-the-art multimodal models under two coding-agent frameworks (OpenHands and Claude Code), the strongest agent—Claude-Opus-4.5—reaches a Visual Score of only 38.4 and a Functional Score of 57.6 on full-stack tasks, while weaker models like Seed-1.8-VL and Qwen3-VL variants fail entirely, establishing that current agents exhibit systematic deficiencies in cross-page coordination, persistent state reasoning, and long-horizon planning, with performance degrading sharply as task complexity increases and as prototype images become larger or target smaller device form factors.

2. Context and Motivation

The Core Gap: No Systematic Benchmark for Visual, End-to-End Website Development

The central problem Vision2Web addresses is deceptively simple to state but has resisted systematic evaluation: how do we rigorously measure whether a multimodal coding agent can take a set of visual prototypes and textual requirements and build a complete, functional, interactive website from scratch? The paper identifies three interlocking deficiencies in existing evaluation infrastructure that make this question unanswerable with current benchmarks.

First, the task formulations are too narrow. The most prominent software engineering benchmarks, such as SWE-Bench and its multimodal variants (Jimenez et al., 2023; Yang et al., 2024), evaluate agents on incremental, issue-driven code patches—given an existing codebase and a bug report, can the agent fix it? This captures a specific, valuable skill (localized debugging within an established architecture) but fundamentally fails to assess the holistic skill of constructing a system from a blank slate. Real-world software development often begins not with a patch to an existing repository, but with a set of requirements and visual designs that must be translated into a working application spanning frontend, backend, and database layers. SWE-Bench-style benchmarks simply cannot measure this capability.

Second, the multimodal coverage is misaligned with the problem's multimodal nature. Website development is inherently multimodal: developers work from both visual prototypes (screenshots, mockups) and textual specifications (requirements documents, descriptions of interactive behavior). Yet existing benchmarks bifurcate along modality lines in ways that prevent holistic evaluation. On one side, text-only benchmarks like VIBE Bench (MiniMax, 2025) and WebGen Bench (Lu et al., 2025) have begun exploring end-to-end development from scratch, but they provide no visual prototypes—agents receive only textual descriptions of the desired website. This is not representative of real-world frontend development, where mockups and design files are the primary specification artifact and textual descriptions serve as supplementary context. On the other side, multimodal benchmarks like Design2Code (Si et al., 2025) evaluate UI-to-code generation from screenshots, but they are restricted to static, single-page reproduction tasks. Design2Code asks: given a single screenshot, can you reproduce its visual appearance in HTML/CSS? It does not ask about multi-page navigation, interactive state changes, form validation, authentication flows, or database-backed functionality. The consequence is that we have benchmarks for "can the agent see a design and code it" and separate benchmarks for "can the agent read a spec and build a system," but no benchmark that integrates both modalities in a task that spans the full development lifecycle.

Third, the verification mechanisms are insufficient for complex, end-to-end system outputs. This is a deeper, more subtle problem that the paper diagnoses carefully. Even when benchmarks attempt end-to-end evaluation (e.g., WebGen Bench), they struggle to reliably and reproducibly assess whether an agent-produced website actually works. The difficulty arises from a fundamental tension in software testing: implementation diversity. Two different agents given the same website specification will produce differently structured code—different frameworks, different CSS organization, different HTML hierarchies, different API endpoints, different state management patterns. Traditional unit tests (assert that the login button has id="login-btn") are brittle to this diversity and break on perfectly functional implementations that happen to use different identifiers or structures. Pixel-level comparisons work for static pages but fail for interactive applications because they can only evaluate a single render state, not multi-step interaction flows. Human evaluation is reliable but doesn't scale to the 1,255 test cases in Vision2Web, and it introduces subjectivity and irreproducibility.

The paper's diagnosis is precise: existing verification approaches are either too rigid (brittle to implementation diversity), too narrow (limited to static snapshots), or too unconstrained (autonomous LLM agents given vague objectives like "test the login function" explore erratically and produce unreproducible results). What's needed is an approach that preserves the flexibility to accept diverse correct implementations while constraining execution enough to produce reproducible, objective assessments.

Why This Problem Matters

The motivation is not purely academic—it responds to a rapidly shifting landscape in software engineering practice and AI evaluation.

The rise of autonomous coding agents creates an urgent need for evaluation. As the paper notes in its introduction, contemporary coding agents (powered by models like Claude, GPT-5, and Gemini) are increasingly capable of performing end-to-end software development tasks encompassing "system design, data processing, code generation, and project debugging." Tools like Claude Code (Anthropic), Cursor, and GitHub Copilot have crossed the threshold from "autocomplete for individual lines" to "multi-file refactoring and project scaffolding." Yet our evaluation methodology has not kept pace. We are deploying agents into real development workflows while still measuring their capabilities on benchmarks designed for an earlier era of single-shot code generation—HumanEval (Chen, 2021), MBPP (Austin et al., 2021), and even SWE-Bench evaluate skills that are several orders of magnitude narrower than what contemporary agents are being asked to do in practice. This creates a dangerous situation where agents may appear competent on available benchmarks while possessing hidden, catastrophic failure modes on realistic tasks—the very failure modes the paper documents in its results (Section 4.3), where full-stack projects "fail to launch, crash during execution, or exhibit pervasive functional errors."

Visual website development is an ideal testbed for multimodal agent intelligence. The paper argues—correctly—that website development naturally satisfies the requirements for evaluating long-horizon multimodal reasoning because it:

• Spans the full software lifecycle (requirements → design → implementation → debugging → deployment)
• Requires coordinated understanding across modalities (visual prototypes, textual requirements, code)
• Demands multi-step reasoning about application state, user interactions, and cross-page coordination
• Produces a verifiable artifact (a running web application) that can be systematically tested

This makes Vision2Web not just a benchmark for "can agents build websites," but a proxy for a broader capability: can an agent hold a complex, multi-modal specification in its "mind" and translate it through hours of sequential decision-making into a functioning system? This skill—long-horizon multimodal planning and execution—is arguably the central challenge for the next generation of AI coding assistants, regardless of the specific domain they're applied to.

Systematic capability diagnosis requires hierarchical task design. The paper's hierarchical structure (Level 1: static pages, Level 2: interactive frontends, Level 3: full-stack websites) is motivated by more than just increasing difficulty. It enables capability disentanglement—the ability to isolate which specific skills fail as complexity increases. If an agent scores 60 on static pages but 20 on full-stack tasks, the drop doesn't just tell you "it's harder"—it tells you approximately where in the development pipeline the breakdown occurs. Without hierarchical design, a low overall score on an end-to-end benchmark is uninterpretable: does the agent fail because it can't reproduce visual layouts? Because it can't implement interaction logic? Because it can't integrate frontend with backend? Because it can't debug deployment issues? Vision2Web's three levels systematically address each of these, enabling fine-grained failure attribution.

Where Existing Approaches Fall Short

The paper's survey of related work (Section 5) identifies specific limitations across three relevant research threads.

UI-to-Code benchmarks are limited to static, single-page reproduction. The UI2Code literature (Section 5.1) has produced valuable datasets and metrics—Design2Code (Si et al., 2025) with its Block-Match and CLIP similarity metrics, WebCode2M (Gui et al., 2025) with its large-scale real-world webpage corpus, Vision2UI (Gui et al., 2024) with layout annotations—but these benchmarks share a fundamental constraint: they evaluate the ability to reproduce a single static screenshot as HTML/CSS. There is no notion of interaction state, no multi-page navigation, no backend integration, no form validation, no authentication. The paper's comparison to Design2Code-Hard (Figure 6, Appendix A.1.2) illustrates how much simpler existing benchmarks are: Design2Code-Hard pages average 251 HTML tags, 10 levels of DOM depth, and 22 unique tag types, while Vision2Web static webpage tasks average 1,385 tags, 22 levels of DOM depth, and 40 unique tag types—roughly 5× more structural complexity even at the simplest benchmark level. This matters because the skills required to reproduce a simple landing page (centering a hero section, matching a color palette, arranging a grid of cards) are qualitatively different from those required to reproduce a complex, information-dense real-world webpage with nested layouts, responsive breakpoints, and precise component-level alignment.

Autonomous coding agent evaluations span too narrow a range. The SWE-Bench ecosystem (Jimenez et al., 2023; Yang et al., 2024) evaluates agents on repository-level issue resolution—given an existing large codebase and a GitHub issue describing a bug or feature request, the agent must locate the relevant files, produce a patch, and pass the associated tests. This is an important capability, but it evaluates a fundamentally different workflow than greenfield development. Issue resolution benefits from the surrounding codebase as context: function signatures, existing test patterns, established architectural conventions, and import structures all provide scaffolds that constrain the solution space. Greenfield development from scratch lacks these scaffolds—the agent must make all architectural decisions itself, from technology stack selection to directory structure to database schema design, with no existing code to imitate. The paper does not claim that SWE-Bench is "bad"—it claims it's insufficient for evaluating the full range of software engineering capabilities that contemporary agents need.

Emerging end-to-end benchmarks lack multimodality or verification rigor. The two most relevant recent benchmarks are VIBE Bench (MiniMax, 2025) and WebGen Bench (Lu et al., 2025). Both evaluate end-to-end website generation from scratch, which is closer to Vision2Web's scope. However:

• VIBE Bench is text-only—agents receive textual descriptions of the desired application but no visual prototypes. This eliminates the multimodal reasoning challenge that is central to real-world frontend development. A textual description says "there should be a blue button with rounded corners in the top right"; a prototype shows that button in context with all its spatial relationships to surrounding elements. The cognitive skill of translating visual layouts to code (which involves understanding spatial hierarchies, relative sizing, color relationships, and component boundaries) is fundamentally different from translating textual descriptions, and a text-only benchmark cannot evaluate it.

• WebGen Bench provides 101 tasks with 647 test cases but lacks structured multimodal inputs and, critically, lacks the principled verification framework that Vision2Web introduces. The paper explicitly contrasts WebGen Bench's approach—which relies on underspecified task definitions and insufficiently constrained verification procedures—with Vision2Web's workflow-based agent verification paradigm (Section 1, third bullet of the gap statement). While WebGen Bench demonstrates that end-to-end website evaluation is possible, it does not solve the reproducibility and rigor problems that arise when evaluating diverse agent implementations against ambiguous specifications.

Verification for interactive systems is particularly underdeveloped. This is perhaps the most technically interesting gap the paper identifies. Testing static pages is relatively straightforward: you can compare screenshots pixel-by-pixel (as in traditional UI testing tools) or use embedding similarity metrics like CLIP. But testing interactive, multi-page, full-stack applications requires verifying behaviors across time and state, not just static appearances. Does the Login button actually authenticate users? Does form validation block invalid submissions? Does navigation between pages preserve application state? Does the search bar filter results correctly? These are functional correctness questions that cannot be answered by looking at a single rendered page. Traditional approaches—handwritten Selenium scripts, unit tests with hardcoded selectors, manual QA—are all either too brittle to implementation diversity or too labor-intensive to scale. The paper's observation that "autonomous LLM- or VLM-based agents... frequently exhibit unconstrained execution when required to handle diverse website realizations and provided with loosely specified objectives" (Section 3) captures the failure mode of naive AI-driven testing: tell a GUI agent to "test the login," and it might do so in 50 different ways across 50 different implementations, producing results that are neither comparable nor reproducible.

How Vision2Web Positions Itself

Vision2Web positions itself not as an incremental extension of existing work but as a synthesis that addresses the identified gaps holistically. The paper's contributions form a coherent design philosophy:

Hierarchical task design for capability disentanglement. Rather than a single monolithic benchmark, Vision2Web's three levels create a diagnostic ladder. An agent that performs well on Level 1 (static pages) but poorly on Level 2 (interactive frontends) reveals a specific deficiency in interaction logic implementation. An agent that performs well on Level 2 but poorly on Level 3 (full-stack) reveals a deficiency in backend integration and system-level planning. The results (Table 3) validate this design: Gemini-3-Pro-Preview achieves a strong Visual Score of 63.3 on desktop static pages (Level 1) but drops to 11.7 on full-stack tasks (Level 3), providing a clear signal that visual reproduction capability does not transfer to system integration capability.

Realistic multimodal data grounded in real-world websites. Vision2Web is not a synthetic benchmark. The multi-stage construction pipeline described in Section 2.2—structural assessment of 63,515 websites from the C4 validation set, VLM-based content screening of 7,391 candidates, and manual review by human annotators—ensures that the benchmark reflects real-world website diversity while avoiding contamination from popular websites that may have appeared in training data. The use of the C4 validation set as the source corpus is a deliberate contamination-prevention choice, since agents trained on web-crawled data are likely to have memorized popular websites but not C4-validation-set pages. The resulting distribution (Figure 2) spans four major categories (Content, Transaction, SaaS Platforms, Public Services) and 16 subcategories, with explicit coverage of diverse website types from news portals to CRM systems to government platforms.

Workflow-based agent verification as the core evaluation innovation. This is where Vision2Web makes its most distinctive technical contribution. The paper frames evaluation not as a single "does it work?" judgment but as a directed acyclic graph of verification nodes, each with explicit guided actions and validation criteria. The key insight—formalized in Algorithm 1—is that by constraining how the agent interacts (Ai: guided actions) while preserving flexibility in what is verified (Vi: validation criteria that can be implementation-agnostic), the framework achieves both reproducibility and adaptability. The GUI agent verifier is given a structured context $C_i = \{H_{<i}, O_i, A_i, V_i\}$ that includes the historical execution trace and node-specific guidance, preventing the "unconstrained execution" problem while allowing the agent to handle implementation diversity in how elements are located and interacted with. The VLM judge performs component-level visual comparison using a structured scoring rubric (1.0 perfect match down to 0.0 no match) that decomposes the page into functional blocks, providing more nuanced and interpretable visual fidelity scores than pixel-level or embedding-based metrics.

The paper validates this verification framework through two studies (Section 4.4): the GUI agent achieves 87.2% node-level agreement with human annotations (218 of 250 test nodes correctly judged), and the VLM judge achieves a median Spearman rank correlation of 0.80 with human preferences, compared to a human inter-annotator correlation of 0.78. This is a strong result: the automated judge is nearly as consistent with human judgment as a second human is. The paper is transparent about remaining inaccuracies, attributing them to "model-intrinsic reasoning hallucinations" and noting the plan to update evaluator models quarterly.

A clear position in the benchmark landscape. In the taxonomy established by Table 1, Vision2Web is the only benchmark that combines: (1) a large number of diverse, real-world tasks (193 vs. 101 for WebGen Bench and 484 for Design2Code), (2) multimodal inputs (visual prototypes + textual requirements, which WebGen Bench and VIBE Bench lack), (3) a hierarchical task structure that covers the full spectrum from UI-to-code to full-stack development (which Design2Code's static-page focus misses), and (4) a principled verification framework with both functional and visual evaluation components. This positions Vision2Web as filling a gap that no existing benchmark addresses individually, and doing so with methodological rigor that raises the bar for evaluation in this area.

Reconciling a Tension in the Paper's Framing

There is a subtle but important tension in how the paper frames its contribution relative to existing benchmarks, and understanding it clarifies what Vision2Web is—and is not—trying to do. On one hand, the paper critiques existing benchmarks for being too narrow (SWE-Bench's incremental patches), too single-modality (WebGen Bench's text-only specs), or too single-level (Design2Code's static pages). On the other hand, Vision2Web's three levels include a level (Level 1) that is essentially "Design2Code but harder"—reproducing static webpages from screenshots at three device resolutions. This might seem redundant: why include Level 1 at all if it overlaps with existing benchmarks?

The answer lies in the paper's capability disentanglement principle. Level 1 isn't redundant because it serves as the diagnostic baseline within Vision2Web's unified evaluation framework. If you evaluate an agent on Design2Code and then separately on WebGen Bench, you can't cleanly attribute performance differences because the evaluation protocols, metrics, and task distributions differ. By keeping all three levels within the same benchmark—same evaluation framework, same agent configuration, same metrics, same category distribution—Vision2Web enables apples-to-apples comparison across complexity levels. When Gemini-3-Pro-Preview drops from 63.3 on static pages to 11.7 on full-stack (Table 3), you can be confident this is a genuine capability degradation, not an artifact of different evaluation methodologies. This is the paper's response to "but Design2Code already exists": existing benchmarks exist in isolation; Vision2Web integrates them into a diagnostic ladder.

Furthermore, the paper's Figure 6 comparison shows that Vision2Web's static webpage tasks are substantially more complex than Design2Code-Hard's, with 5× more DOM tags and 2× more unique tag types on average. So Level 1 is not just "Design2Code replicated"—it's a harder, more realistic version that better represents the complexity of real-world websites even at the static reproduction level.

3. Technical Approach

3.1 Reader Orientation

Vision2Web is not a single model or algorithm but rather a benchmark and evaluation framework—a carefully constructed collection of website development tasks paired with an automated testing system that uses AI agents to verify whether a coding agent's output is correct. The core problem it solves is reproducible, implementation-agnostic evaluation of multimodal coding agents on end-to-end website development, and the shape of the solution is a hierarchical task design (three levels of increasing complexity) combined with a workflow-based verification paradigm where a GUI agent executes structured test procedures and a VLM judge scores visual fidelity, together producing functional correctness scores and visual similarity scores that are comparable across diverse agent implementations.

3.2 Big-Picture Architecture (Diagram in Words)

The Vision2Web system has four major components, arranged in a pipeline from construction to evaluation:

1. Benchmark Construction Pipeline — takes raw web pages from the C4 validation set through structural assessment, VLM-based content screening, and manual review to produce a curated set of 193 tasks across three hierarchical levels, each task containing prototype images, textual requirements, and a multimedia resource library.

2. Test Workflow Annotator — a human-AI collaborative process (PhD researchers + Claude Code) that produces structured test workflows for each task, where each workflow is a directed sequence of verification nodes specifying testing objectives, guided actions, and validation criteria.

3. Coding Agent Under Test — the system being evaluated (any multimodal model integrated into a coding agent framework like OpenHands or Claude Code), which receives the task inputs (prototypes, requirements, resources) and produces a complete deployable web application in a containerized environment.

4. Workflow-Based Agent Verification System — two specialized AI verifiers that execute against the agent's deployed output: a GUI Agent Verifier (instantiated with GLM-4.6V) that follows test workflows, performs guided interactions with the rendered application, and determines whether each functional verification node passes or fails, and a VLM-Based Judge (instantiated with Gemini-3-Pro-Preview) that performs component-level visual comparisons between rendered pages and reference prototypes, assigning fidelity scores on a 0–1 scale per component.

The information flow is: benchmark construction produces tasks → annotation produces test workflows → coding agents produce deployed applications → GUI agent verifier executes workflows and produces Functional Scores (FS) → VLM judge compares rendered pages to prototypes and produces Visual Scores (VS). These scores are aggregated per task level to produce the benchmark results shown in Table 3.

3.3 Roadmap for the Deep Dive

• First, the hierarchical task formulation—how tasks are structured across the three levels, what inputs each level provides, and how the levels build on each other to enable capability disentanglement. This is the foundation: everything else depends on understanding what agents are being asked to do.

• Second, the benchmark construction pipeline—the multi-stage filtering process that transforms 63,515 raw websites into 193 curated tasks, including the specific criteria applied at each stage and the contamination-prevention strategy. This explains where the tasks come from and why they represent realistic, diverse evaluation instances.

• Third, the workflow-based agent verification framework—the formal abstraction of testing as a dependency graph, the design philosophy behind decoupling and integrating test nodes, and the structure of functional and visual verification nodes. This is the core evaluation innovation and the most technically dense part of the paper.

• Fourth, the GUI Agent Verifier—how it is configured, what context it receives, how it makes decisions, and how it produces pass/fail judgments for functional verification nodes. This includes the node tuple structure, the prompt design, and the action space.

• Fifth, the VLM-Based Judge—how it performs component-level visual comparison, the scoring rubric, the component segmentation approach, and how visual scores are aggregated across components and prototypes. This includes the design rationale for structured rubrics over pixel-level comparisons.

• Sixth, the agent-assisted annotation process—how test workflows are created through collaboration between human experts and Claude Code, with different strategies for different task levels (lightweight for static pages, automated for frontends, expert-in-the-loop for full-stack). This explains how the verification infrastructure is itself constructed.

• Seventh, the evaluation protocol—how coding agents are configured, the containerized environment, the deployment requirements, and the metrics (Visual Score, Functional Score, Deployment Success Rate). This is where the framework actually runs.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark and evaluation infrastructure paper whose core technical contribution is the workflow-based agent verification paradigm—a method for reproducibly evaluating end-to-end software systems using constrained AI agents rather than brittle unit tests or unconstrained autonomous evaluation.

Hierarchical Task Formulation

The task formulation is the foundational design decision that enables Vision2Web's diagnostic capability. The paper organizes all tasks into three levels of increasing complexity, where each level adds new requirements while preserving the demands of previous levels. This is not merely increasing difficulty—it is a deliberate capability disentanglement strategy where performance drops between levels can be attributed to specific skill gaps.

Level 1: Static Webpage. At this level, the agent receives three prototype images showing the same webpage rendered at desktop, tablet, and mobile resolutions, along with explicit resolution specifications. The agent must produce a single static webpage (HTML/CSS) that faithfully reproduces the layout, visual content, and styling at each target resolution. There are no interactive elements, no multi-page navigation, no backend—pure visual reproduction across responsive breakpoints.

The key design choice here is the cross-device responsive requirement. Most UI-to-code benchmarks (like Design2Code) evaluate reproduction at a single resolution, typically desktop. By requiring reproduction at three resolutions, Vision2Web tests whether the agent understands responsive design principles—whether it uses CSS media queries, flexible grid layouts, or other techniques to adapt the same content to different viewport widths, versus simply hardcoding pixel positions that break on smaller screens. The paper does not prescribe how responsive behavior should be achieved (media queries vs. fluid layouts vs. separate stylesheets), only that the rendered output at each resolution matches the prototype.

Level 1 tasks include a multimedia resource library containing images, icons, videos, and fonts that the agent must correctly reference and incorporate. This simulates the realistic scenario where a developer receives design assets alongside mockups and must correctly map asset files to the visual elements in the prototype. The paper notes that asset handling is a specific failure mode even at this level (Section 4.3): agents "over-rely on file names and lack robust multimodal grounding," causing inconsistencies when assets are unnamed or ambiguously referenced.

Level 2: Interactive Frontend. Building on Level 1's visual reproduction requirements, Level 2 adds multi-page interactivity. Inputs include multiple prototype images (one per page or view) and textual descriptions of inter-page logical relationships—which pages link to which, what navigation structures exist, what interactive behaviors are expected. The agent must produce a fully interactive multi-page frontend that preserves structural consistency and coherent navigation flows.

The critical new capability being tested is cross-page reasoning in a multimodal context. The agent must look at multiple prototype images (showing different pages of the same website), understand that they belong to the same application (shared header, footer, navigation bar, color scheme), extract the navigation structure from both the visual prototypes (which show links and buttons) and the textual descriptions (which specify relationships), and implement a consistent frontend where clicking a "Services" link in the navigation bar actually navigates to the Services page whose prototype was provided.

This level is where interaction state enters the benchmark. While Level 1 only requires visual fidelity (does the page look right?), Level 2 requires functional fidelity as well (do the interactive elements work?). The GUI agent verifier now tests behaviors like: clicking a navigation link navigates to the correct page; dropdown menus expand and collapse; form elements accept input; interactive cards respond to hover or click events.

The paper's data (Table 2) shows the increased complexity quantitatively: Level 2 tasks average 5.9 prototype images (vs. 3 for Level 1) and 7.5 test cases (vs. none for Level 1, since static pages only require visual verification).

Level 3: Full-Stack Website. This is the most complex level and the one that most closely simulates real-world engineering scenarios. Inputs include structured requirement documents alongside prototype images. The requirement document (exemplified in Appendix A.1.3's "Simplified Requirement Document Example") includes sections on product background, business workflows (page navigation flow, core workflows like authentication and posting), and detailed requirements specifications organized by functional module (e.g., user authentication, home portal, topic category, post detail, post creation, user profile).

The agent is expected to:

• Interpret multi-page requirement documents and prototype images
• Design database schemas and generate seed data
• Choose a technology stack (frontend framework, backend server, database)
• Implement both frontend (matching prototypes) and backend (implementing specified business logic)
• Integrate frontend and backend (API calls, state management, authentication persistence)
• Perform debugging and verification
• Produce a deployable system accessible at http://localhost:3000

This level evaluates comprehensive end-to-end software engineering capabilities: requirements interpretation, system architecture, database design, frontend-backend integration, state management across components, authentication flows, CRUD operations, search/filter/sorting functionality, file and media operations, and deployment scripting.

Table 2 quantifies the complexity increase: Level 3 tasks average 8.5 prototype images, 28.2 test cases (nearly 4× more than Level 2), and 4,300 text tokens in the requirement document (4× more than Level 2's 1,000 tokens). The task distribution (Table 7, Appendix A.1.2) shows that full-stack tasks exist across all four major website categories and all 16 subcategories, ensuring that the benchmark tests full-stack development across diverse domain contexts (news portals, e-commerce platforms, CRM systems, government portals, etc.).

The capability disentanglement logic. The three levels form a diagnostic ladder: if an agent scores well on Level 1 but poorly on Level 2, the deficit is specifically in interaction logic implementation and cross-page coordination (since visual reproduction is independently verified to work). If an agent scores well on Level 2 but poorly on Level 3, the deficit is specifically in backend integration, system architecture, and long-horizon planning (since frontend implementation and interaction logic are independently verified to work). The paper's results validate this: Gemini-3-Pro-Preview achieves VS = 63.3 on Level 1 desktop pages but drops to VS = 11.7 on Level 3, while Claude-Opus-4.5 maintains relatively higher Level 3 performance (VS = 38.4, FS = 57.6), demonstrating that visual reproduction capability does not automatically translate to system integration capability.

Benchmark Construction Pipeline

The construction pipeline transforms a large corpus of raw web pages into a curated, contamination-free benchmark through three sequential filtering stages. All source websites come exclusively from the C4 validation set (Raffel et al., 2020), a deliberate choice to avoid potential data leakage from popular websites that may have been memorized by models trained on web-crawled data.

Stage 1: Structural Assessment. The first filter analyzes DOM-level properties of candidate web pages, following principles adapted from Design2Code (Si et al., 2025). Three specific properties are examined:

• HTML tag distribution: the frequency and variety of HTML elements used on the page. Pages dominated by a single tag type (e.g., only <div> containers with no semantic structure) are excluded because they lack the structural richness needed to evaluate layout reproduction.
• DOM tree depth: the nesting level of HTML elements. Pages with very shallow DOM trees (flat, unstructured layouts) are too simple; pages with malformed structures (e.g., unclosed tags, invalid nesting) are technically problematic.
• Token length: the total size of the page in tokens. Pages with insufficient content are too simple for evaluation; pages with excessive token length may indicate auto-generated or spam content.

Pages that pass these criteria are those with "non-trivial structural complexity"—they have meaningful semantic markup, appropriately deep DOM hierarchies, and sufficient content to constitute a realistic development task. This stage reduces the candidate set to 63,515 websites from an unspecified initial corpus size.

Stage 2: Content Screening. The second filter uses a VLM (vision-language model) to score candidate websites for content and design quality. The VLM evaluates three dimensions:

• Functional richness: whether the page contains meaningful interactive components (forms, navigation menus, search bars, content filters, authentication elements) rather than being purely static or purely decorative.
• Modular clarity: whether the page's visual components are organized into clearly distinguishable functional blocks (header, sidebar, main content, footer, card grids) with coherent spatial relationships.
• Visual coherence: whether the page exhibits consistent styling (color scheme, typography, spacing), professional-quality layout, and appropriate information hierarchy.

Pages lacking meaningful interactive components, exhibiting poor layout organization, or offering limited functional coverage are excluded. This stage is critical because it filters for pages that will produce meaningful evaluation tasks—a page that is visually beautiful but functionally inert makes a poor benchmark instance because it cannot test interaction logic. After this stage, 7,391 pages remain.

The paper does not specify which VLM is used for this screening stage, nor the exact scoring thresholds. This is a minor transparency gap, though the subsequent manual review stage provides a quality backstop.

Stage 3: Manual Review. The final and most rigorous filter involves human annotators reviewing remaining websites across all task levels. Reviewers evaluate each website based on multiple criteria:

• Page consistency and quality across device resolutions: does the page maintain visual coherence and functionality when rendered at desktop, tablet, and mobile viewport widths? Pages that break badly at smaller sizes are excluded or flagged for revision.
• Implementation difficulty: is the page complex enough to constitute a meaningful test of agent capabilities, but not so complex (e.g., requiring specialized WebGL rendering, real-time video processing, or proprietary API integrations) that it becomes an unreasonable task?
• Overall page dimensions: are the prototype images of manageable size? The paper's Figure 5 (Appendix A.1.2) shows that static webpage prototype sizes range from roughly 500×300 pixels to over 3000×2000 pixels, with the distribution concentrated in the 1000–2000 pixel range.
• Clarity and richness of interactive functionality: for Level 2 and Level 3 tasks, do the interactive elements have clear, testable behaviors? Vague interactions ("the page should be dynamic") are excluded in favor of concrete, verifiable behaviors ("clicking the Login button navigates to the authentication page").
• Category balance: reviewers actively select websites to ensure balanced coverage across the 16 subcategories and 4 major categories. This is not purely a quality filter—it also shapes the benchmark's distribution to reflect real-world website diversity.

The manual review stage is where the final task assignment to levels occurs. A website that passes structural assessment and content screening might become a Level 1 task if it is a high-quality static page, a Level 2 task if it has clean multi-page navigation, or a Level 3 task if it has backend-dependent functionality. The paper does not detail how many annotators were involved, what inter-annotator agreement was, or how long the manual review took—these are practical details that would strengthen reproducibility but are not essential for understanding the methodology.

Final benchmark composition. The pipeline produces 193 tasks: 100 static webpage tasks (Level 1), 66 interactive frontend tasks (Level 2), and 27 full-stack website tasks (Level 3). The decreasing number of tasks as complexity increases reflects the reality that full-stack websites are harder to find in the wild (many websites are largely static or have shallow interactivity) and more expensive to annotate (requiring expert-in-the-loop test case design, as described in Section 3.3). Across all levels, the benchmark contains 21,516 total input files, 918 prototype images, and 1,255 test cases.

The category distribution (Figure 2) shows tasks spanning four major categories: Content (news, community, multimedia, knowledge, publishing), Transaction (retail, vertical markets, services, procurement), SaaS Platforms (CRM, HCM, ERP, TMS, AI platform), and Public Services (government portal, public utility websites). Each subcategory has at least one task at each level (Table 7), ensuring that no domain is evaluated only at easy levels.

Comparison to existing benchmarks. The paper provides a quantitative comparison to Design2Code-Hard in Figure 6 (Appendix A.1.2):

• Average tag count: Design2Code-Hard = 251 ± 232, Vision2Web (Webpage) = 1,385 ± 985 (5.5× more)
• Average DOM depth: Design2Code-Hard = 10 ± 4, Vision2Web = 22 ± 7 (2.2× deeper)
• Average unique tags: Design2Code-Hard = 22 ± 5, Vision2Web = 40 ± 14 (1.8× more diverse)

This establishes that even Vision2Web's simplest level is substantially harder than the leading existing static UI-to-code benchmark. The standard deviations on Vision2Web metrics are also larger (985 vs. 232 for tag count), indicating greater diversity in task difficulty—the benchmark includes both moderately complex and extremely complex static pages, rather than clustering around a narrow difficulty band.

The Workflow-Based Agent Verification Framework

This is the paper's core evaluation innovation—the mechanism by which complex, end-to-end website outputs are assessed reproducibly and objectively despite diverse agent implementations. The framework rests on two key abstractions: (1) a formalization of testing as a directed dependency graph of verification nodes, and (2) an agent-in-the-loop execution model that constrains how testing proceeds while allowing flexibility in how the agent handles implementation-specific details.

The dependency graph abstraction. The paper observes that end-to-end website testing involves a sequence of interdependent steps where later verifications rely on the successful execution of earlier interactions over shared application states. For example, testing a "post creation" workflow first requires authentication (logging in), then navigation to the post creation page, then form interaction, then submission, then verification that the post appears on the detail page. Each step depends on the previous one—you cannot test post submission if login failed.

Rather than treating this as a monolithic test script, Vision2Web formalizes testing as:

"A directed dependency graph, where each node represents a self-contained verification sub-procedure (e.g., functional or visual) and edges encode sequential dependencies and shared states. Each node comprises a sequence of interactions that brings the application into a target state, followed by verification."

Formal representation of verification nodes. Each node $n_i$ in the graph corresponds to a verification sub-procedure targeting a specific aspect of website correctness. The paper explicitly categorizes nodes into two types:

Functional verification node $n_i = \langle O_i, A_i, V_i \rangle$:

where:

• $O_i$ is the testing objective—a natural language description of what functionality is being tested (e.g., "Verify that post creation succeeds when all required fields are filled with valid data, and the user is navigated to the newly created post detail page with matching content")
• $A_i$ is the guided actions—a sequence of constrained interaction steps the verifier agent should perform (e.g., "1. Click the Log in button, 2. Type '12345678' into the Phone number field, 3. Click the Continue button...")
• $V_i$ is the validation criteria—logical assertions or state-based checks that determine whether the test passes (e.g., "The post title displays 'Test topic', The page navigates to the newly created post detail page, The post content displays 'It's a test content'")

What this form computes: a functional verification node takes a testing objective, constrains the agent's interactions to a predefined sequence, and provides explicit pass/fail criteria. The node is executed by the GUI agent verifier (described in the next subsection), which follows $A_i$ to bring the application into a target state and then checks whether $V_i$ holds.

Why this form: the explicit modeling of $A_i$ is the key design choice that prevents the "unconstrained execution" problem the paper identifies in Section 3. Traditional approaches would provide only the objective $O_i$ ("test the login function") and let the agent decide how to test—this leads to erratic behavior across implementations because the agent might test the login in different ways (different credentials, different navigation paths, different verification checks), producing non-comparable results. The $A_i$ constrains how the agent interacts while $V_i$ preserves what is being verified, decoupling the two concerns. The validation criteria are implementation-agnostic: they specify what should be true (the title field displays "Test topic") without specifying how the agent should verify this (by checking DOM properties, by reading visible text, by checking a specific element's state).

Visual verification node $n_i = \langle P_i \rangle$:

where $P_i$ is the target prototype image that the rendered page should visually match.

What this form computes: a visual verification node takes a reference prototype and triggers the VLM judge to perform component-level visual comparison. Unlike functional nodes which specify actions and validation criteria, visual nodes are purely a target image reference—the testing workflow's navigation steps bring the application to the correct page, and then the VLM judge compares the rendered state to the prototype.

Why this form: visual verification is fundamentally different from functional verification in that there are no interactive steps to perform (the page is already rendered in the target state) and the evaluation criteria are perceptual rather than logical. The simplicity of $\langle P_i \rangle$ reflects this—all the complexity lives in the VLM judge's comparison logic (described in the VLM judge subsection below), not in the node structure.

The shared context $C_i$. At each verification node $n_i$, the verifier agent is provided with an explicitly constructed context:

$C_i = \{H_{<i}, O_i, A_i, V_i\}$

where:

• $H_{<i}$ records the objectives and actions from all preceding verification nodes—the historical execution trace showing what has been tested and what interactions have been performed
• $O_i, A_i, V_i$ are the current node's objective, actions, and validation criteria as defined above

What this form computes: the context bundles the current node's specification with the history of what came before. This gives the agent awareness of temporal dependencies and state transitions—it knows that authentication was already performed, that navigation to a specific page already occurred, and that the current application state is the result of all prior actions.

Why this form: including $H_{<i}$ is essential for two reasons. First, it prevents redundant actions—the agent doesn't re-login when it's already authenticated because it can see login was already performed. Second, it enables the agent to reason about state transitions—if a prior node's action navigated to /posts/new, the current node can assume the browser is on that page and begin interactions from there. Without this historical context, the agent would need to either re-establish state (wasteful and potentially error-prone if state changes between executions) or infer state from the current page (which may be ambiguous).

From graph to workflows. The paper instantiates the dependency graph into test workflows—agent-executable subgraphs corresponding to coherent interaction trajectories under a shared application context. The paper provides two principled guidelines for constructing these workflows (Section 3.1):

1. Decoupling dependent test nodes. Test cases that span multiple functional modules (e.g., product browsing, shopping cart, checkout) are separated into distinct workflows. The rationale is to mitigate error accumulation and propagation along excessively long interaction chains: if a failure occurs early in a 15-step workflow testing both browsing and checkout, the later checkout tests are never reached and their results are lost. By decoupling, failures in one module don't obscure evaluation of other modules.

2. Integrating related test nodes. Test cases that operate within the same application context (e.g., multiple UI interactions within a single page or functional module) are grouped into a single workflow. The rationale is to reduce redundant setup and navigation—testing five different form validation scenarios on the same page doesn't require navigating to that page five times from scratch. Each test starts from the same shared state, saving execution time and reducing opportunities for navigation errors.

The tradeoff here is between coverage reliability (decoupling prevents cascading failures) and execution efficiency (integrating avoids redundant navigation). The paper's guidelines are heuristic-based but principled—they prioritize coverage for cross-module dependencies (where failures are likely and costly) and efficiency for within-module tests (where the shared context is stable).

Algorithm 1: Workflow-Based Agent Verification. The paper formalizes the execution procedure in pseudocode:

Algorithm 1 Workflow-Based Agent Verification
Input: Workflow W = (n1 → ··· → nt), initial state S0
Output: Aggregate functional and visual scores (F, V)

H, F, V ← ∅
for ni ∈ W do
    if ni is Functional verification then
        (Fi, Si+1) ← GUIAgentVerifier(H, Oi, Ai, Vi, Si)
        F ← F ∪ {Fi}; H ← H ∪ {(Oi, Ai)}
    else if ni is Visual verification then
        (Vi, Si+1) ← VLMBasedJudge(Pi, Si)
        V ← V ∪ {Vi}
    end if
end for
return (F, V)

What this algorithm computes: for a given test workflow (an ordered sequence of verification nodes) and an initial browser state $S_0$, it sequentially processes each node. For functional nodes, the GUI agent verifier takes the current history, objective, guided actions, validation criteria, and browser state, executes the guided actions, checks the validation criteria, and returns a pass/fail judgment $F_i$ plus the resulting browser state $S_{i+1}$. The history $H$ is updated with the executed objective and actions. For visual nodes, the VLM judge compares the current page state to the prototype $P_i$ and returns a visual score $V_i$ plus the (unchanged) browser state. The algorithm returns the full set of functional judgments $F$ and visual scores $V$.

Why this algorithm: the key property is sequential state propagation—each node receives the browser state $S_i$ produced by the previous node's execution and returns the state $S_{i+1}$ after its own execution. This directly encodes the dependency structure of the verification graph: Node 3's verification depends on Node 2's actions having been performed because Node 2's actions produced the browser state that Node 3 starts from. The algorithm is deterministic in its control flow (always executing nodes in order) but flexible in its execution (the agent can handle implementation-specific details like locating elements).

The overall Functional Score and Visual Score computation. For a given task level, the Functional Score (FS) is:

$\text{FS} = \frac{\text{Number of passed functional verification nodes}}{\text{Total number of functional verification nodes}} \times 100$

What this computes: the proportion of functional test cases that the agent's output passes, scaled to a 0–100 range. A score of 57.6 (Claude-Opus-4.5 on full-stack tasks under OpenHands) means that approximately 57.6% of the functional verification nodes across all full-stack task workflows were passed.

Why this form: a simple pass-rate is the most interpretable and comparable metric for functional correctness. It doesn't weight nodes by importance (though the annotation process could, in principle, do so by including more nodes for critical functionality) and is robust to different numbers of nodes across tasks. The paper explicitly notes that this is reported as "the proportion of passed functional verification nodes" (Section 3.2).

For the Visual Score (VS), the computation is:

$\text{VS}_{\text{task}} = \frac{1}{|P|} \sum_{p \in P} \text{VS}(p)$

where $P$ is the set of all prototype images for the task, and $\text{VS}(p)$ is the VLM judge's score for prototype $p$, computed as:

$\text{VS}(p) = \frac{1}{|C_p|} \sum_{c \in C_p} \text{score}(c)$

where $C_p$ is the set of components segmented by the VLM judge in the rendered page corresponding to prototype $p$, and $\text{score}(c) \in \{0, 0.25, 0.5, 0.75, 1.0\}$ is the per-component fidelity score.

What this computes: the visual score for a task is the average of per-prototype visual scores, where each per-prototype score is the average of per-component scores across all components that the VLM judge identifies in that page. Per-component scores are discrete values from the rubric.

Why this form: averaging across components and prototypes provides a granular, interpretable measure. The component-level averaging means that a page with 20 components where 19 are perfect (score 1.0) and 1 is missing (score 0.0) gets a score of 0.95—high but not perfect, accurately reflecting that the page is mostly correct but has a notable flaw. The prototype-level averaging means that a task with 8 prototype images where the agent perfectly reproduces 6 but fails on 2 gets a score around 0.75—reflecting the partial success. Alternative aggregation methods (e.g., minimum score across components) would be overly harsh (one small error would make the page score 0), while taking only the overall page judgment (without decomposing into components) would lose diagnostic information about where the failures occur.

GUI Agent Verifier Design

The GUI Agent Verifier is the component that executes functional verification nodes. It is instantiated using GLM-4.6V (Hong et al., 2025) following the task execution protocol of WebVoyager (He et al., 2024), though the paper notes that "the framework itself is agnostic to the specific agent architecture."

Agent configuration and context. At each functional verification node $n_i = \langle O_i, A_i, V_i \rangle$, the agent receives the structured context:

$C_i = \{H_{<i}, O_i, A_i, V_i\}$

This context is presented to the agent through a carefully designed prompt (shown in Appendix A.2.2). The prompt includes:

• A role specification: "You are a GUI Testing Agent. Your primary task is to execute software test cases on a Web application..."
• The current time (for logging/timestamping)
• The full context $C_i$ structured as "Objective" ($O_i$), "Actions" ($A_i$), and "Validations" ($V_i$)
• The test platform specification ("Web")
• The action space definition (Click, Type, Scroll, Wait, GoBack, Refresh, Key, Answer)
• The history of already-performed actions with their thoughts
• Current observations in the form of two screenshots: a full-page screenshot and a viewport screenshot

Action space. The paper specifies a predefined GUI action space: Click (click on a UI element at specified coordinates or by describing the element), Type (enter text into an input field), Scroll (scroll the page vertically or horizontally), Wait (pause execution for a specified duration), GoBack (navigate to the previous page), Refresh (reload the current page), Key (press a keyboard key), and Answer (provide a pass/fail judgment with reasoning). These actions follow "standard web interaction semantics"—the exact mechanism by which the agent translates action descriptions to browser interactions is not detailed but presumably uses accessibility tree parsing, screenshot coordinate mapping, or DOM querying (as is standard in web agent frameworks).

The execution loop. For each functional verification node, the agent follows a sequential process:

1. Observe: the agent receives the full-page screenshot (showing the entire rendered page, potentially requiring scrolling to view all content) and the viewport screenshot (showing only what is currently visible in the browser window). Having both is important because some UI elements may be below the fold in the viewport screenshot but visible in the full-page screenshot.

2. Think: the agent produces a "Thought" describing its reasoning about what it needs to do next and why, based on the current observation, the guided actions $A_i$, and the history $H_{<i}$.

3. Act: the agent selects and executes one action from the action space (Click, Type, Scroll, etc.).

4. Repeat: steps 1–3 until all guided actions in $A_i$ have been executed (the agent reaches the target state specified by the objective).

5. Verify: the agent evaluates whether the validation criteria $V_i$ are satisfied in the current state, then produces a final judgment (pass or fail) with reasoning.

What the pass/fail judgment $F_i$ means. A functional node passes if and only if all validation criteria in $V_i$ are satisfied after executing all guided actions in $A_i$. Partial satisfaction (some criteria met, some not) constitutes a failure. The judgment is binary—there is no partial credit for functional verification nodes.

Why this design works for implementation diversity. The guided actions $A_i$ specify what to do in semantic terms ("Click the Log in button," "Type '12345678' into the Phone number field") without specifying where those elements are in the DOM or how to locate them. The agent uses its visual and contextual understanding to locate the relevant elements in the rendered page—whether the Login button is an <a> tag with class nav-login, a <button> with id login-btn, or a <div> with an onclick handler. This decoupling of intent (specified in $A_i$) from implementation (handled by the agent's perception) is what makes the verification implementation-agnostic.

The validation criteria $V_i$ are similarly semantic: "The page navigates to the newly created post detail page" can be satisfied regardless of whether the URL changes to /posts/123, whether a modal appears, or whether the page content updates in place. The agent must determine whether the semantic condition holds in the current state, interpreting page transitions, content changes, and UI feedback through the lens of human-understandable outcomes.

Validation of the GUI agent verifier (Section 4.4). The paper evaluates the verifier's reliability by measuring agreement with human annotations. Approximately 100 test workflows are randomly sampled from 64 tasks. For each workflow, all constituent test nodes are independently examined by human annotators who verify whether the nodes satisfy the intended test requirements.

At the node level: 218 of 250 nodes (87.2%) are correctly judged by the verifier relative to human annotations. This means that for a given verification node, the GUI agent's pass/fail judgment matches the human's judgment 87.2% of the time. The residual 12.8% inaccuracies are attributed to "model-intrinsic reasoning hallucinations" (the agent incorrectly perceives a UI element, misinterprets a validation criterion, or fails to execute an action correctly). The paper notes that these inaccuracies "are expected to diminish as the proficiency of the GUI Agent continues to improve."

What constitutes a "correct judgment" in this validation study is not explicitly defined—it presumably means that the agent's binary pass/fail decision matches the human annotator's binary pass/fail decision for the same test node. The paper does not report whether the errors are biased (e.g., the agent is more lenient than humans, passing nodes that humans would fail) or symmetric.

VLM-Based Judge Design

The VLM-Based Judge is the component that evaluates visual fidelity by comparing rendered pages against reference prototypes. It is instantiated with Gemini-3-Pro-Preview (DeepMind). Unlike the GUI agent verifier which performs interactive testing, the VLM judge performs a purely perceptual comparison task—it receives two static images (a prototype and a rendered page) and produces a structured fidelity judgment.

Component segmentation. The judge's first task is to decompose both the prototype image and the actual rendered page into meaningful UI components based on visual and functional layout. The prompt specifies:

"You need to automatically segment the page into meaningful UI components based on visual and functional layout. Divide the page into logical functional blocks, not too granular or too coarse. For each block, treat it as a single component for scoring purposes."

What this does: rather than comparing the entire page as a single monolithic image (which would produce a single coarse score), the judge identifies component boundaries—header, navigation bar, hero section, card grid, sidebar, footer, etc.—and scores each independently. This decomposition is performed by the judge itself based on its visual understanding, not predefined by annotators. The granularity guideline ("not too granular or too coarse") gives the judge flexibility to adapt component boundaries to the specific page structure—a simple landing page might have 5 components (header, hero, features, testimonials, footer), while a complex dashboard might have 20+ components (navigation, sidebar filters, multiple chart widgets, data tables, toolbar, etc.).

Why automatic segmentation: manually annotating component boundaries for 918 prototype images across 193 tasks would be prohibitively expensive and would introduce annotator subjectivity in boundary placement. By delegating segmentation to the VLM judge, the process is scalable and consistent (the same model produces both segmentation and scoring, so the two are aligned). The downside is that segmentation errors (e.g., grouping two logically distinct components into one, or splitting one component into two) can propagate into the scoring. The paper does not explicitly validate segmentation quality, but the high Spearman correlation with human preferences (0.80 median) suggests that segmentation errors, if present, do not substantially distort the aggregate rankings.

The scoring rubric. Each component receives a discrete score from the set $\{0, 0.25, 0.5, 0.75, 1.0\}$ according to a five-level rubric:

What this rubric computes: for each component identified in the page, the judge assigns a score based on alignment with the corresponding component in the prototype. The descriptions encode both spatial properties (position, alignment, sizing, spacing) and content properties (text, fonts, colors, icons, images, multimedia). The levels are ordered by severity: 1.0 means essentially indistinguishable; 0.75 means recognizable but with subtle issues; 0.5 means noticeably wrong but still functionally similar; 0.25 means severe degradation; 0.0 means missing or completely wrong.

Why this rubric form: the five discrete levels balance granularity (sufficient to distinguish minor from major errors) with reliability (human-like categorical judgments are easier for VLMs to produce consistently than continuous 0–100 scores). The < 2px threshold in the 0.75 level provides a concrete, measurable boundary for "slight misalignment" versus "noticeable misalignment." The inclusion of both spatial and content dimensions in each level ensures that the judge evaluates the full visual experience—a component could be perfectly positioned but have wrong text (score 0.75 or lower depending on severity), or have perfect content but be misplaced (similar deduction).

The rubric intentionally uses human-perceptual language ("no visually discernible differences") rather than pixel-level metrics. This aligns the scoring with what matters to human users—whether the page looks right—rather than what's easy to compute automatically (pixel-wise MSE). The paper justifies this in Section 3: "Pixel-level comparisons, while effective for static renderings, are limited to snapshots and rely on low-level appearance similarity that often diverges from human perceptual judgments."

The structured output format. The judge outputs a JSON array of component objects:

[
  {
    "name": "<component name>",
    "score": <0 | 0.25 | 0.5 | 0.75 | 1>,
    "reason": "<brief explanation of why this score was given>"
  }
]

The name field provides human-readable component identification (e.g., "header navigation bar," "hero section with CTA buttons," "footer with social links"). The reason field provides interpretability—not just what score was assigned, but why, enabling human reviewers to audit judgments and researchers to understand common failure patterns.

Validation of the VLM judge (Section 4.4). The paper evaluates the judge's alignment with human preferences using Spearman rank correlation coefficient ($\rho$), a standard metric for evaluating preference alignment and ranking consistency in subjective judgment tasks. The paper notes that $\rho > 0.5$ indicates "substantial rank consistency."

The evaluation procedure: across 100 randomly sampled prototypes, the VLM judge's ranking of agent outputs is compared to human annotators' rankings of the same outputs. The judge achieves an average Spearman correlation of 0.66 and a median of 0.80. For comparison, human inter-annotator agreement on the same set yields a Spearman correlation of 0.78.

Interpreting these numbers: the median of 0.80 is very close to the human inter-annotator correlation of 0.78, meaning that on the majority of prototypes, the VLM judge is essentially as aligned with human judgment as a second human would be. The average being lower (0.66 vs. 0.78) indicates that there are some "challenging cases" where the judge diverges significantly from human preferences, pulling down the mean while the median remains high. The paper explicitly acknowledges this: "Given the intrinsic subjectivity of visual preference judgments and the non-trivial disagreement among human annotators, the observed correlation represents a strong and practical level of alignment in most instances, while leaving room for improvement in challenging cases."

Why Spearman correlation rather than raw agreement: visual fidelity is inherently subjective, and absolute scores (e.g., "this page deserves a 0.73") are less meaningful than relative rankings ("this page is better than that page"). Spearman correlation measures whether the judge ranks agent outputs in the same order as human annotators, which is the relevant signal for benchmark evaluation (we want to know which agent performs best, not the exact absolute visual quality). The paper's choice of $\rho$ over agreement metrics like Cohen's kappa reflects this focus on ranking consistency.

Quarterly evaluator updates. The paper states a plan to "update both the VLM judge and GUI agent on a quarterly basis using the latest backbone models" to "maintain evaluator reliability over time." This is an acknowledgment that evaluator quality is not static—as better VLMs become available, the verification framework should be upgraded to incorporate them, improving accuracy and reducing hallucination. This is a pragmatic approach: rather than claiming the current evaluators are perfect, the paper treats them as components that can and should be improved over the benchmark's lifetime.

Agent-Assisted Test Case Annotation

The test workflows that drive the verification framework must themselves be created—they are not automatically derived from the tasks. The paper describes an annotation process that combines human expertise with AI assistance, with the strategy adapted to task complexity across levels.

For Level 1 (Static Webpage): annotation is "lightweight and reduces to resolution-specific visual verification." This means the test workflows for static pages consist only of visual verification nodes—there are no functional verification nodes because there is no interactivity to test. Each prototype image (desktop, tablet, mobile) becomes a separate visual verification node, and the VLM judge compares the agent's rendered page at that resolution to the prototype. No GUI agent interaction is needed because static pages have no state to manipulate.

For Level 2 (Interactive Frontend): annotation is "largely automated, with Claude Code inferring navigation structures from prototypes to generate navigation-oriented test cases without involving backend logic." The key phrase is "inferring navigation structures from prototypes"—Claude Code examines the multiple prototype images for a frontend task, identifies navigation elements (links, buttons, dropdown menus) that connect pages, and constructs test workflows that verify these navigation paths work correctly. For example, if the prototypes show a Home page, Services page, and Contact page with a shared navigation bar, Claude Code produces test workflows that verify clicking each navigation link loads the correct page.

The annotation is "without involving backend logic" because Level 2 tasks are frontend-only—there is no database, no authentication, no server-side processing. The test cases focus on client-side behavior: navigation correctness, form input handling, UI state changes (e.g., dropdown expansion, tab switching), and visual consistency across pages.

For Level 3 (Full-Stack Website): annotation requires "dedicated annotation due to long-horizon dependencies, cross-module interactions, and frequent boundary or exceptional cases." The paper notes that "purely agent-driven annotation is often unstable under such settings"—Claude Code alone cannot reliably construct comprehensive test workflows for full-stack applications because the complexity exceeds what can be inferred from prototypes and requirement documents without human judgment about what constitutes meaningful, coverage-complete testing.

The process follows an expert-in-the-loop strategy:

1. Domain experts (PhD researchers) first draft high-level testing workflows from the requirement documents and design prototypes. These drafts "explicitly encode task dependencies, execution order, and shared application states while avoiding redundant validation across workflows." The experts determine the overall testing strategy: which functional modules need testing, how tests should be grouped into workflows (applying the decoupling/integrating guidelines), what the dependency structure is, and what the coverage priorities are.

2. Claude Code refines and expands the workflows by translating the high-level specifications into executable interaction sequences and verifiable criteria. The experts specify what should be tested ("verify that post submission fails when the content field is empty"); Claude Code translates this into concrete guided actions ("1. Click the Submit button" after ensuring the content field is empty) and validation criteria ("Post submission fails and does not proceed" and "An error message is displayed indicating that the content field is required").

The collaborative approach leverages complementary strengths: humans provide strategic judgment about what constitutes meaningful testing (coverage decisions, dependency understanding, boundary case identification), while Claude Code handles the tactical task of translating strategies into executable sequences (generating specific interaction steps, phrasing validation criteria precisely, ensuring format consistency).

Test case type distribution (Figure 3). The paper provides a breakdown of test case categories across full-stack tasks to illustrate the resulting diversity and coverage. The categories (Table 5, Section 4.2) include:

• Navigation & Routing: verifying page transitions, link correctness, breadcrumb trails
• State Management: verifying that application state persists across interactions (e.g., filter selections preserved when navigating back)
• Form Interaction: verifying input acceptance, validation, error messages, submission behavior
• UI Feedback: verifying visual feedback for interactions (e.g., loading spinners, success toasts, error highlighting)
• Authentication & Authorization: verifying login/logout flows, session persistence, permission-based access control
• CRUD Operations: verifying Create, Read, Update, Delete operations on data entities
• Search / Filter / Sorting: verifying query functionality, filter application, sort order
• File & Media Operations: verifying file uploads, image display, media playback

This categorization demonstrates that the full-stack test cases cover a comprehensive range of real-world web application functionality rather than focusing narrowly on a few easy-to-test features.

Illustrative workflow example (Appendix A.2.1). The paper provides a concrete example of a test workflow for a full-stack task (the Airbnb Community Center forum application). The workflow tests post creation with four sequential nodes:

• Node 1 (Functional): Navigate to the post creation page as a logged-in user. Guided actions: click Log in, type phone number, click Continue, click Post button. No validation criteria (this node only establishes state).
• Node 2 (Visual): A visual verification node pointing to the "post" prototype—verifying the post creation page renders correctly.
• Node 3 (Functional): Verify form accepts valid input. Guided actions: type title, select topic from dropdown, click sub-topic card. Validation: title field displays correct text, topic dropdown shows selection, sub-topic card is visually selected.
• Node 4 (Functional): Verify post submission fails when content is empty (negative test). Guided actions: click Submit. Validation: submission fails, error message displayed.
• Node 5 (Functional): Verify post creation succeeds with all fields filled. Guided actions: type content, click Submit. Validation: submission succeeds, navigates to post detail page, title and content match.

This example demonstrates the workflow design principles in action: dependent nodes are integrated (Nodes 1→3→4 build on shared authentication state), test cases within the same module (post creation) are grouped into a single workflow, and the testing covers both positive (success) and negative (validation failure) scenarios.

Evaluation Protocol and Experimental Configuration

The paper evaluates coding agents under standardized conditions to ensure fair comparison. The evaluation protocol specifies the agent configuration, the execution environment, the deployment requirements, and the metrics.

Coding agent configuration. The paper evaluates eight multimodal models integrated into two coding agent frameworks:

• Frameworks: OpenHands (Wang et al., 2024b) and Claude Code (Anthropic, b). OpenHands is an open platform for AI software developers as generalist agents; Claude Code is Anthropic's AI-powered coding assistant.
• Models: Claude-Opus-4.5, Claude-Sonnet-4.5 (Anthropic), Gemini-3-Pro-Preview, Gemini-3-Flash-Preview (DeepMind), GPT-5 (OpenAI), Seed-1.8-VL (ByteDance), Qwen3-VL-32B-Instruct, Qwen3-VL-8B-Instruct (Bai et al., 2025).

The paper uses carefully designed prompts that guide the agent through the required workflow (shown in Appendix A.3.1). The prompt structure for full-stack tasks includes:

1. Input materials enumeration: explicitly listing all available inputs—the Product Requirement Document (/workspace/prd.md), prototype images (provided inline in the prompt, not as file paths), and resource files (/workspace/resources/**/* including images, videos, audio, icons).

2. Mandatory workflow specification: a five-phase process—Planning & Design (define database schema, choose tech stack, design directory structure), Seed Data Generation (generate realistic seed data reflecting prototypes), Full-Stack Implementation (backend with APIs/validation/authentication, frontend replicating visuals with live API integration), Deployment & Verification (build, verify at localhost:3000, generate startup script), and Documentation (design document, README).

3. Required deliverables: complete source code, design document, seed data, deployment script (start.sh), README.

4. Hard constraints: implement all PRD features without skipping or inventing functionality, all visuals must match prototypes, system must be fully reproducible via bash /workspace/start.sh, do not terminate until all steps and startup verification are complete.

The hard constraint about not terminating prematurely is particularly important. The paper notes in its failure mode analysis (Section 4.3) that agents "lack reliable mechanisms for autonomous verification"—they may incorrectly believe they have completed a task when they haven't, leading to premature termination. The prompt explicitly instructs against this.

Containerized execution environment. All agent evaluations run in a Docker container based on Ubuntu 22.04, preconfigured with necessary runtimes and development tools (Dockerfile shown in Appendix A.3.2). The environment includes:

• System tools: curl, wget, git, vim, unzip, sudo, build-essential, gcc, g++, make, cmake, net-tools, iputils-ping
• Database clients: postgresql-client, default-mysql-client, redis-tools, sqlite3 (supporting multiple backend database choices)
• Node.js 20.x (for JavaScript/TypeScript frontend and backend frameworks)
• Python 3.12 with venv and pip (for Python-based backends)
• Playwright with Chromium (for browser automation in testing)

The container runs as a non-root agent user with a /workspace directory. The paper notes (Appendix A.3.2) that "no Model Context Protocol (MCP) or additional orchestration/configuration layers are installed, ensuring that the agent relies solely on its own reasoning and coding capabilities to complete the assigned tasks." This is an important constraint: agents cannot offload complex reasoning to external tools or pre-configured project templates—they must make architectural decisions themselves.

Deployment requirements. Each agent generates a startup script (/workspace/start.sh) that deploys the project on a fixed port (3000). The paper allows "up to three iterations to collect more analyzable evaluation results"—if deployment fails, the agent gets two more attempts to fix issues and redeploy. Deployments exceeding 10 minutes or producing errors after three attempts are treated as failures.

Deployment Success Rate (DSR). The DSR is the fraction of tasks for which the agent produces a deployable application that starts successfully on port 3000 within the time and iteration limits. The paper presents DSR "for reference rather than an official metric" (Table 3 note), meaning it is tracked but not considered the primary evaluation signal. This is appropriate because DSR measures a necessary precondition for evaluation (the application must run to be tested) rather than a capability signal per se—an application that deploys successfully but is completely non-functional in terms of the specification would have a high DSR but low FS and VS.

The metrics reported. For each agent-framework combination and each task level, the paper reports:

• Level 1 (Static Webpage): Visual Score broken out by device type (Desktop, Tablet, Mobile), plus the average across devices ("Avg") and the Deployment Success Rate. There is no Functional Score because static pages have no interactivity to test.
• Level 2 (Interactive Frontend): Visual Score (average across all prototypes), Functional Score (average pass rate across all functional verification nodes), their average ("Avg"), and DSR.
• Level 3 (Full-Stack Website): Visual Score, Functional Score, their average, and DSR.

All scores are reported on a 0–100 scale.

Why separate device-specific scores for static pages: the paper's Finding 2 (Section 4.2) is that "agent performance systematically degrades when adapting to smaller device form factors." Reporting separate Desktop, Tablet, and Mobile scores makes this degradation visible and quantifiable. For Gemini-3-Pro-Preview under OpenHands, the drop from Desktop (63.3) to Mobile (48.3) is a 23.7% relative decline, quantifying the responsive design challenge that a single average score would obscure.

Why average VS and FS for interactive/stack tasks: the paper reports both scores but does not combine them into a single composite metric. This preserves the diagnostic value—a task with high VS but low FS indicates good visual reproduction but poor functional implementation, while the reverse indicates functional correctness but poor visual fidelity. Combining them would lose this signal. The "Avg" column in Table 3 is a simple arithmetic mean provided for high-level comparison, not a weighted or normalized composite.

4. Key Insights and Innovations

Innovation 1: Capability Disentanglement via Hierarchical Task Design as a Diagnostic Instrument

The most conceptually distinctive contribution of Vision2Web is not that it has three difficulty levels—many benchmarks do—but that the levels are architectured as a diagnostic ladder where performance drops between levels attribute failure to specific skill gaps. This transforms the benchmark from a simple "harder than before" test into an instrument for localizing which capabilities are missing.

Before Vision2Web, the evaluation landscape for coding agents was fragmented by scope: SWE-Bench tested localized debugging (Jimenez et al., 2023), Design2Code tested static visual reproduction (Si et al., 2025), and emerging end-to-end benchmarks like WebGen Bench (Lu et al., 2025) tested holistic website generation. An agent's performance on these benchmarks was incommensurable—a score of 60 on Design2Code and 20 on WebGen Bench could not be interpreted as "the agent can reproduce layouts but fails at system integration" because the benchmarks used different task distributions, evaluation protocols, and metrics. The field had no way to answer the question where does this agent break?

Vision2Web's hierarchical design answers this by making the levels cumulative and diagnostic. Level 1 isolates visual reproduction; Level 2 adds interactivity on top of the visual requirement; Level 3 adds backend integration on top of both. Because all levels share the same evaluation framework, the same category distribution, and the same metrics, a performance drop between levels is attributable to the added capability demand, not to methodological differences. When Gemini-3-Pro-Preview drops from Visual Score 63.3 on Level 1 desktop pages to 11.7 on Level 3 (Table 3), the inference is clean: visual reproduction capability does not transfer to full-stack system integration, and the bottleneck is specifically in backend architecture, state management, or cross-layer debugging. When Claude-Opus-4.5 maintains 38.4 VS and 57.6 FS on Level 3, the inference is that it possesses some of these integration skills that Gemini lacks but still falls short of robust full-stack competence.

This is a fundamental reframing of benchmark design philosophy—from "measure aggregate performance on increasingly hard tasks" to "construct a controlled experiment where each level adds one capability dimension while holding others constant." It belongs in the same conceptual category as the Chinchilla scaling laws' separation of model size and data quantity as independent axes (Hoffmann et al., 2022): just as Chinchilla showed that you cannot reason about scaling by varying "compute" monolithically, Vision2Web shows that you cannot reason about coding agent capability by varying "task difficulty" monolithically. You must vary specific capability axes independently to understand what drives performance.

The evidence supporting this as a genuine diagnostic instrument rather than merely three difficulty tiers lies in the counterintuitive pattern in the results. If the levels simply measured "harder tasks," all agents would degrade uniformly across levels. But Table 3 shows differential degradation: Gemini-3-Flash-Preview achieves 47.8 average VS on static pages (competitive with Claude-Sonnet-4.5's 47.1) but drops to 12.5 average on full-stack (vs. Claude-Sonnet-4.5's 20.4). Gemini's static-page capability is strong but its system integration capability is disproportionately weak—a finding invisible to a monolithic benchmark. Similarly, Seed-1.8-VL achieves some frontend capability (VS/FS of 7.5/33.9 on Level 2 under OpenHands) but collapses to 0.0/0.0 on full-stack tasks (Table 3), indicating that its failure mode is specifically backend integration rather than interaction logic. These differential patterns are the diagnostic signal the hierarchical design was built to produce.

Innovation 2: Workflow-Based Agent Verification as a Middle Path Between Brittle Scripts and Unconstrained Exploration

The paper's second major conceptual move is its reframing of automated software testing for AI-generated outputs. The field faced a seemingly intractable tension: traditional test scripts (Selenium with hardcoded CSS selectors) are too rigid for implementation diversity—two correct agents may produce different HTML structures, different class names, different element hierarchies, and a test that passes one will break on the other. At the opposite extreme, fully autonomous LLM-based evaluators given vague objectives ("test the login") are too unconstrained—they explore erratically, produce non-reproducible results, and cannot guarantee coverage because their execution path depends on model-specific whims at each decision point.

Vision2Web's workflow-based paradigm resolves this tension through a separation of concerns that was not present in prior work. The core insight, encoded in the functional verification node $\langle O_i, A_i, V_i \rangle$ , is to constrain how the agent interacts (through guided actions $A_i$) while preserving flexibility in what is verified (through implementation-agnostic validation criteria $V_i$). The guided actions specify sequential steps in semantic terms—"Click the Log in button," "Type '12345678' into the Phone number field"—without anchoring to specific DOM selectors, class names, or element IDs. The agent must use its visual and contextual understanding to locate the Login button in the specific implementation it is testing, whether that button is rendered as <a class="nav-login">, <button id="login-btn">, or a React component with a generated CSS module class name. The validation criteria specify semantic outcomes—"The page navigates to the newly created post detail page"—without prescribing URL patterns, DOM mutations, or state management mechanisms.

This is a fundamental shift from both dominant paradigms. Traditional GUI testing research (dating back to Selenium and its predecessors) viewed the test as a script to be executed deterministically; implementation diversity was a bug to be fixed by standardizing selectors. Autonomous agent testing viewed the test as a goal to be achieved through open-ended exploration; reproducibility was a bug to be fixed through better prompting or agent alignment. Vision2Web's approach views the test as a constrained interaction trajectory where the path is fixed (ensuring reproducibility) but the means of traversal is delegated to an agent (ensuring implementation tolerance). The analogy is to path-following in robotics: a navigation system gives waypoints (guided actions) but the robot's controller handles the specifics of terrain and obstacle avoidance (element location and interaction).

The verification framework's formalization as a directed dependency graph with shared context $C_i = \{H_{<i}, O_i, A_i, V_i\}$ adds another layer of sophistication. By propagating execution history $H_{<i}$ through sequential nodes, the framework explicitly models state dependencies in the testing process—Node 4's verification of post submission depends on Node 1's authentication having succeeded. This prevents the agent from redundantly re-establishing state (which wastes time and introduces new failure modes) and gives it the context needed to reason about temporal dependencies ("since we already logged in, the current page should have an authenticated user's avatar visible"). Prior testing frameworks handled state dependencies implicitly (through test ordering conventions) or not at all (each test started from scratch), neither of which is satisfactory for long-horizon full-stack testing where authenticating once and then performing 20+ dependent interactions is the norm.

The validation results (Section 4.4) support the viability of this middle path. The GUI agent verifier achieves 87.2% node-level agreement with human annotations (218/250 nodes correctly judged), indicating that the constrained-execution approach produces reliable pass/fail decisions despite implementation diversity. The 12.8% residual error is attributed to model-intrinsic reasoning hallucinations—the agent's visual perception or reasoning occasionally fails—not to the framework design. This is an important distinction: the framework is sound; the current instantiation's accuracy is bounded by the underlying VLM's capabilities, which improve over time. The quarterly evaluator update plan (Section 4.4) operationalizes this as a pragmatic design choice rather than a fundamental weakness.

Innovation 3: Structural Complexity Analysis Revealing Existing Benchmarks as Fundamentally Unrepresentative

Vision2Web makes a sharp empirical argument that goes beyond "we built a harder benchmark." Through its structural complexity analysis (Figure 6, Appendix A.1.2), the paper demonstrates that existing UI-to-code benchmarks—specifically Design2Code-Hard, the leading benchmark in this space—evaluate a qualitatively different task than real-world website reproduction. The numbers are stark: Design2Code-Hard pages average 251 HTML tags with 22 unique tag types and 10 levels of DOM depth, while Vision2Web's static webpage tasks average 1,385 tags, 40 unique tag types, and 22 levels of DOM depth—roughly 5.5× more elements, 1.8× more tag diversity, and 2.2× deeper nesting.

These are not marginal difficulty increases; they represent a phase transition in task complexity. A 251-tag page is typically a simple landing page with a hero section, a few feature cards, and a footer—something a competent human developer could reproduce in under an hour. A 1,385-tag page with 22 levels of DOM depth is a densely structured information portal with nested navigation, multi-column layouts, deeply hierarchical content sections, and complex widget structures—the kind of page that might take a professional developer a full day to reproduce faithfully. The cognitive demands are fundamentally different: at 251 tags, an agent can plausibly succeed through pattern-matching ("add a centered hero, then a 3-column card grid, then a footer"), while at 1,385 tags, the agent must understand and reproduce a complex, non-obvious information architecture that requires reasoning about spatial hierarchies, content prioritization, and visual balance across multiple nested levels of organization.

This finding is significant because it challenges the validity of existing benchmark results as measures of real-world capability. When Design2Code reports strong model performance on "hard" UI-to-code tasks, those results reflect competence on a task class that is systematically simpler than the real-world web pages that constitute actual frontend development work. The paper's analysis implies that the field has been measuring performance on a restricted, unrepresentative sample of the target distribution, and that reported scores may not transfer to realistic settings. This is a construct validity critique—it argues that existing benchmarks do not measure what they claim to measure—and it provides quantitative evidence for the claim rather than relying on qualitative intuition.

The larger standard deviations on Vision2Web's metrics (985 vs. 232 for tag count) further indicate that the benchmark captures a wider diversity of task difficulties than Design2Code, which clusters around a narrow complexity band. This is important for evaluation sensitivity: a benchmark where all tasks are similarly hard cannot discriminate between agents with small but meaningful capability differences. Vision2Web's diversity means it can detect both the agent that handles simple pages flawlessly but fails on complex layouts, and the agent that handles moderate complexity but collapses on information-dense pages.

This is not merely a scaling-up of existing work—it is a redefinition of the task's difficulty range. By anchoring the benchmark to real-world website complexity rather than the output of simplified data collection pipelines (Design2Code's tasks were generated by screenshotting simplified web pages), Vision2Web establishes a new baseline for what "visual website development" means in an evaluation context. The implication for future work is clear: benchmarks that do not reflect real-world structural complexity are measuring a fundamentally different skill than the one they claim to evaluate.

Innovation 4: The Verification Framework as a Self-Validating Measurement Instrument

Vision2Web's approach to validating its own evaluation framework represents a methodological innovation that extends beyond this specific benchmark. The paper treats the verifiers not as ground-truth oracles but as measurement instruments whose accuracy must be empirically characterized—and it provides the validation methodology and quantitative results that enable users to interpret scores with appropriate uncertainty.

Specifically, Section 4.4 reports two validation studies: (1) the GUI agent verifier's node-level agreement with human annotations (87.2% on 250 nodes from ~100 workflows across 64 tasks), and (2) the VLM judge's Spearman rank correlation with human preferences (median 0.80, average 0.66, compared to human inter-annotator correlation of 0.78). These are not afterthoughts—they are presented as core results that establish the reliability bounds of the entire benchmark's evaluation signal.

This is methodologically significant because benchmarks in AI have historically treated their evaluation metrics as given. When SWE-Bench reports that an agent resolves 25% of issues, the implicit assumption is that the test cases perfectly capture resolution correctness. When HumanEval reports pass@1, the assumption is that the unit tests are complete and correct. Rarely do benchmarks validate whether their evaluation procedures themselves agree with human judgment, and when they do, it is typically a small-scale sanity check rather than a systematic study. Vision2Web's validation is systematic—250 nodes, 100 prototypes, quantitative metrics, comparison to human inter-annotator baselines.

The most striking result is that the VLM judge's median Spearman correlation of 0.80 is essentially indistinguishable from the human inter-annotator correlation of 0.78. This means that for a typical prototype comparison, asking the VLM judge to rank agent outputs produces rankings that are as aligned with a given human's preferences as another human's rankings would be. This is a strong result that supports the judge as a practical substitute for human evaluation in ranking tasks—the primary use case for benchmark leaderboards. The gap between median (0.80) and average (0.66) reveals that there exist "challenging cases" where the judge diverges significantly, pulling down the mean. The paper's transparency about this gap (rather than reporting only the more favorable median) is itself a methodological virtue—it tells users that the judge is very reliable most of the time but can be unreliable on certain prototypes, and that these challenging cases are an area for improvement.

The quarterly evaluator update plan—"update both the VLM judge and GUI agent on a quarterly basis using the latest backbone models"—reflects a pragmatic recognition that evaluation quality is not static. Rather than claiming the current verifiers are definitive, the paper treats them as upgradable components whose accuracy will improve as the underlying VLMs improve. This is a departure from the traditional benchmark philosophy where evaluation procedures are fixed to ensure comparability over time. Vision2Web effectively argues that the benchmark's construct validity (measuring what it intends to measure) is more important than strict metric comparability across years, and that verifier upgrades represent improvements in measurement accuracy rather than changes in the measurement target.

This self-validating approach is a conceptual model for how AI-based evaluation should work in domains where ground-truth is expensive or impossible to obtain. Rather than claiming the AI evaluator is correct by definition, Vision2Web establishes its accuracy bounds empirically and plans for incremental improvement. This shifts the evaluation conversation from "is the AI judge good enough?" (a binary question) to "what is the AI judge's error rate, and how does it compare to human annotator disagreement?" (a quantitative, contextualized question).

Innovation 5: Systematic Characterization of Category-Specific Agent Weaknesses

Beyond aggregate performance scores, Vision2Web enables a fine-grained capability diagnosis that reveals which specific functional categories coding agents struggle with—a level of analysis that was not possible with previous benchmarks because they either lacked functional testing entirely (Design2Code) or lacked structured categorization of test cases (WebGen Bench).

The category-wise functional score breakdown in Table 5 (Section 4.2, Finding 6) reveals systematic patterns: Claude-Opus-4.5 achieves 66.3% pass rate on Navigation & Routing and 61.8% on Authentication & Authorization—relatively strong performance on tasks that involve discrete state transitions with clear success/failure signals. But it drops to 43.2% on State Management, 43.6% on CRUD Operations, and 33.3% on File & Media Operations—tasks that require persistent state tracking across components, correct data flow between frontend and backend, or coordination between application logic and system-level resources.

This pattern is not obvious a priori. One might expect authentication to be harder than CRUD (it involves session management, token persistence, and security constraints), but agents handle it relatively well because the workflow is standardized: a login form, a token exchange, a redirect or state change. CRUD operations are conceptually simpler (create, read, update, delete) but harder for agents to implement correctly because they require the agent to coordinate data models across database schema, API endpoints, frontend form handling, and display logic—four distinct layers that must remain consistent. The agent might correctly implement the database INSERT but fail to wire the frontend form's submit handler to the correct API endpoint, or might implement the API endpoint correctly but fail to display the newly created entity on the list page. These cross-layer consistency requirements appear to be a specific Achilles' heel.

The website category breakdown (Table 4) provides another diagnostic dimension. Public Service websites—characterized by simple structures, limited interactions, and primarily informational content—yield the highest performance across agents. SaaS Platforms—characterized by multi-page navigation, complex interaction patterns, role-based access, and data-heavy dashboards—consistently yield the weakest results. The gap between Claude-Opus-4.5's performance on Public Service (VS/FS of 56.9/60.0) and SaaS Platforms (VS/FS of 22.9/39.9) is striking: the same agent is 2.5× better at visual reproduction and 1.5× better at functional correctness when the target website has simpler interaction patterns. This reveals that current agents' capabilities are highly sensitive to structural complexity in ways that aggregate scores obscure.

This is more than an interesting observation—it provides a roadmap for capability improvement. The category-wise analysis tells researchers and developers exactly where to focus: improving state management, CRUD consistency, and file operations will yield larger gains than further optimizing navigation or authentication, which are already relatively strong. Without this granular breakdown, a developer seeing "FS = 57.6" for Claude-Opus-4.5 on full-stack tasks would not know whether to invest effort in improving authentication flows (already at 61.8%) or CRUD operations (at 43.6%, with substantial room for improvement). The category-level diagnostic transforms the benchmark from a summative assessment ("how good is this agent?") into a formative assessment ("what should we work on next?").

This innovation is enabled by the workflow-based annotation process described in Section 3.3, where test cases are explicitly categorized (Figure 3) during expert-in-the-loop annotation. It is the combination of hierarchical level design (diagnosing vertical capability drops) and functional category design (diagnosing horizontal capability patterns) that gives Vision2Web its diagnostic power. Neither dimension alone would be sufficient: levels without categories would tell you when agents fail but not on what, while categories without levels would tell you what agents struggle with but not at which development stage the struggle emerges. Together, they provide a two-dimensional diagnostic grid (complexity level × functional category) that enables precise attribution of failure modes.

5. Experimental Analysis

Evaluation Methodology

• Dataset. Vision2Web comprises 193 tasks across three hierarchical levels—100 static webpage tasks, 66 interactive frontend tasks, and 27 full-stack website tasks—curated from the C4 validation set (Raffel et al., 2020) through a multi-stage filtering pipeline. The benchmark contains 918 prototype images and 1,255 test cases spanning 16 subcategories organized into four major website categories (Content, Transaction, SaaS Platforms, Public Services). All source websites come exclusively from the C4 validation set to avoid contamination from popular websites that may have appeared in training data (Section 2.2).

• Base model(s). The paper evaluates eight state-of-the-art multimodal models: Claude-Opus-4.5, Claude-Sonnet-4.5 (Anthropic), Gemini-3-Pro-Preview, Gemini-3-Flash-Preview (DeepMind), GPT-5 (OpenAI), Seed-1.8-VL (ByteDance), Qwen3-VL-32B-Instruct, and Qwen3-VL-8B-Instruct (Bai et al., 2025). These span a range of scales and capabilities, from frontier closed-source models to open-weight models, enabling evaluation of how coding agent performance varies with model quality. The models are integrated into two coding agent frameworks: OpenHands (Wang et al., 2024b) and Claude Code (Anthropic, b).

• Metrics. The paper reports three primary metrics (Table 3): Visual Score (VS), the average component-level fidelity score (0–100 scale) across all prototypes for a task level, computed by the VLM-based judge comparing rendered pages against reference prototypes using a five-level rubric (0, 0.25, 0.5, 0.75, 1.0 per component); Functional Score (FS), the proportion of passed functional verification nodes across all test workflows for a task level (0–100 scale), where each node passes if and only if all validation criteria are satisfied after executing all guided actions; and Deployment Success Rate (DSR), the fraction of tasks for which the agent produces a deployable application within three iterations and 10 minutes per attempt, reported "for reference rather than an official metric." For Level 1 (static pages), only VS is reported per device type (Desktop, Tablet, Mobile) and as an average. For Levels 2 and 3, both VS and FS are reported along with their average.

• Baselines. The paper does not implement separate baseline methods in the traditional sense—the benchmark itself is the contribution, and all eight models evaluated serve as the reference points. However, implicit baselines include: (1) prior benchmark results on Design2Code (Si et al., 2025) for static UI-to-code tasks, which this paper's Figure 6 shows are substantially simpler than Vision2Web's static webpage tasks; (2) prior results on WebGen Bench (Lu et al., 2025) for end-to-end website generation, which lacks multimodal inputs and structured verification; and (3) the natural baseline of comparing weaker models against Claude-Opus-4.5 as the strongest performer on complex tasks. The paper also validates its verifiers against human annotations (Section 4.4), establishing human-level agreement as a baseline for evaluation quality.

• Generation budget / compute accounting. The paper does not standardize on a fixed generation budget across agents—each agent receives the same task inputs (prototype images, textual requirements, resource files) and is evaluated on the quality of its final output, not on the compute consumed to produce it. This differs from the scaling-law-style "generations" budget used in the reference example paper. The cost accounting here is through the deployment constraint: agents get up to three iterations to produce a working deployment, with each attempt capped at 10 minutes (Section 4.1). This means compute is controlled through a time-and-retry budget rather than a token generation budget. The evaluation framework uses GLM-4.6V for the GUI agent verifier and Gemini-3-Pro-Preview for the VLM judge, both operating in a containerized environment (Appendix A.2.2), but their compute cost is not factored into agent performance metrics.

• Cross-validation / statistical protocol. The paper validates the verifiers themselves through two studies (Section 4.4). For the GUI agent verifier: approximately 100 test workflows are randomly sampled from 64 tasks, and all constituent test nodes (250 total) are independently examined by human annotators to verify whether they satisfy the intended test requirements, yielding 87.2% node-level agreement (218 of 250 nodes correctly judged). For the VLM judge: across 100 randomly sampled prototypes, Spearman rank correlation is computed between the judge's rankings and human annotators' rankings, yielding a median of 0.80 and an average of 0.66, compared to human inter-annotator correlation of 0.78 on the same set. The paper does not use cross-validation for agent performance evaluation itself—all 193 tasks are used in the reported results (Table 3), with no held-out task splits for hyperparameter tuning (which is appropriate since the agents being evaluated are not tuned on Vision2Web).

Main Quantitative Results

Aggregate Performance Across Task Levels

The headline result is in Table 3, which presents end-to-end performance for all eight models across both frameworks and all three task levels. The overarching finding (Finding 1) is that agent performance degrades consistently and substantially as task complexity increases, with even the strongest model struggling on full-stack tasks.

Under the OpenHands framework, Claude-Opus-4.5 achieves:

• Level 1 (Static Webpage): VS = 58.9 (Desktop), 53.7 (Tablet), 47.7 (Mobile), DSR = 98%
• Level 2 (Interactive Frontend): VS = 46.5, FS = 66.7, DSR = 98.5%
• Level 3 (Full-Stack Website): VS = 38.4, FS = 57.6, DSR = 96.3%

The drop from Level 1 Desktop VS (58.9) to Level 3 VS (38.4) represents a 34.8% relative decline in visual fidelity, and the drop from Level 2 FS (66.7) to Level 3 FS (57.6) represents a 13.6% relative decline in functional correctness. Even the deployment success rate drops from 98% to 96.3%, indicating that full-stack tasks cause deployment failures even for the strongest agent.

The contrast with weaker models is stark. Gemini-3-Pro-Preview under OpenHands achieves:

• Level 1: VS = 63.3 (Desktop), 55.8 (Tablet), 48.3 (Mobile), DSR = 95%
• Level 2: VS = 29.7, FS = 40.7, DSR = 93.9%
• Level 3: VS = 11.7, FS = 22.6, DSR = 77.8%

The Gemini model actually outperforms Claude-Opus-4.5 on static pages (63.3 vs. 58.9 on Desktop) but dramatically underperforms on full-stack tasks (VS 11.7 vs. 38.4; FS 22.6 vs. 57.6). The Level 2 to Level 3 drop for Gemini is catastrophic: VS drops from 29.7 to 11.7 (a 60.6% relative decline), and FS drops from 40.7 to 22.6 (a 44.5% relative decline), compared to Claude-Opus-4.5's more measured declines. This differential degradation pattern is precisely the diagnostic signal the hierarchical design was built to produce: it reveals that Gemini's visual reproduction capability (strong on static pages) does not transfer to system integration capability, while Claude-Opus-4.5 possesses integration skills that Gemini lacks but still falls well short of robust full-stack competence.

The weakest models largely fail on complex tasks:

• Seed-1.8-VL under OpenHands: Level 1 Desktop VS = 1.1 (essentially no visual reproduction capability), Level 3 VS = 0.0, FS = 0.0, DSR = 14.8% (the 14.8% DSR indicates it can sometimes deploy a project but that project has zero functional or visual correctness)
• Qwen3-VL-32B-Instruct under both frameworks: Level 1 VS near 0 across all devices; Level 2 and 3 VS = 0.0, FS = 0.0, DSR = 0–4%
• Qwen3-VL-8B-Instruct under OpenHands: Level 1 Desktop VS = 0.2, DSR = 51% (it can deploy static pages roughly half the time but cannot reproduce visual content at all), Level 2 and 3 VS = 0.0, FS = 0.0

The Qwen models' inability to complete multimodal coding tasks despite having 51% DSR on static pages (for the 8B variant) is revealing: the models can sometimes produce a deployable application but cannot translate visual prototypes into code, producing pages that render but bear no visual resemblance to the target prototypes. This suggests a specific weakness in visual grounding rather than general coding inability.

Device-Dependent Performance Degradation

Within Level 1 (Static Webpage), Table 3 reveals systematic device-dependent degradation (Finding 2). Desktop interfaces achieve the highest fidelity across all models, while tablet and mobile layouts show lower scores. For specific models under OpenHands:

• Gemini-3-Pro-Preview: Desktop 63.3 → Tablet 55.8 (11.8% relative decline) → Mobile 48.3 (23.7% relative decline from Desktop)
• Claude-Opus-4.5: Desktop 58.9 → Tablet 53.7 (8.8% relative decline) → Mobile 47.7 (19.0% relative decline)
• GPT-5: Desktop 49.0 → Tablet 44.6 (9.0% relative decline) → Mobile 40.5 (17.3% relative decline)
• Claude-Sonnet-4.5: Desktop 51.9 → Tablet 44.9 (13.5% relative decline) → Mobile 44.4 (14.5% relative decline)

The degradation is roughly monotonic with device size for all models, but the magnitude varies: Claude-Sonnet-4.5 shows an unusual pattern where the Tablet-to-Mobile drop (44.9 to 44.4) is much smaller than the Desktop-to-Tablet drop (51.9 to 44.9), suggesting it handles very small viewports relatively better than medium ones. The paper attributes these device-dependent drops to "current agents' limited capacity to process and reason over complex visual inputs" (Finding 2). The responsive design challenge is a genuine capability test—an agent that hardcodes desktop pixel positions will see catastrophic degradation at smaller viewports, while one that uses proper responsive techniques (media queries, flexible grids) will see more moderate declines.

Figure 4 provides a complementary analysis: Visual Scores are correlated with prototype image height, with larger, denser prototype images inducing additional performance declines for representative models. The paper notes this reflects "current agents' limited capacity to process and reason over complex visual inputs"—longer pages with more content are harder to reproduce faithfully, even at the same device resolution.

Framework-Dependent Performance Variation

Table 3 also reveals that agent performance varies with the choice of coding agent framework (Finding 4). Across most non-Claude models, performance under OpenHands tends to be higher than under Claude Code:

• GPT-5 Level 3 under OpenHands: VS = 18.3, FS = 49.7, DSR = 100% vs. under Claude Code: VS = 9.6, FS = 23.4, DSR = 85.2%
• Gemini-3-Pro-Preview Level 3 under OpenHands: VS = 11.7, FS = 22.6 vs. under Claude Code: VS = 5.7, FS = 12.9
• Gemini-3-Flash-Preview Level 3 under OpenHands: VS = 7.7, FS = 17.2 vs. under Claude Code: VS = 2.3, FS = 4.6

The framework gap is substantial: GPT-5's Level 3 FS nearly doubles under OpenHands (49.7 vs. 23.4), and its VS roughly doubles (18.3 vs. 9.6). This suggests that the OpenHands framework provides better scaffolding or tool integration for non-Claude models, enabling more effective multi-step development workflows.

For Claude models, the pattern is mixed: Claude-Opus-4.5 achieves higher VS on Desktop static pages under OpenHands (58.9 vs. 54.2) but lower Level 2 VS under OpenHands (46.5 vs. 46.1)—the difference is negligible. Claude-Sonnet-4.5 consistently performs better under OpenHands (Level 1 Desktop VS: 51.9 vs. 44.7; Level 3 VS: 15.7 vs. 14.5). The paper notes this "reflects framework design differences" and suggests that "further research on joint optimization of models and agent frameworks could be beneficial for improving overall system performance"—the interaction between model capabilities and framework affordances is non-trivial and merits dedicated investigation.

Category-Specific Performance Patterns

Table 4 (Section 4.2, Finding 5) breaks down Level 3 (full-stack) performance by website category for three selected models under OpenHands. The results reveal systematic sensitivity to website type:

Claude-Opus-4.5:

• Public Service: VS = 56.9, FS = 60.0
• Transaction: VS = 43.2, FS = 64.9
• Content: VS = 37.1, FS = 61.2
• SaaS Platform: VS = 22.9, FS = 39.9

The pattern is consistent: Public Service websites (simple structures, limited interactions, primarily informational) yield the highest visual and functional scores; Transaction and Content websites show moderate performance; SaaS Platforms (multi-page navigation, complex interaction patterns, role-based access) consistently yield the weakest results. The gap between Public Service (VS 56.9) and SaaS Platform (VS 22.9) represents a 59.8% relative decline—the same agent is 2.5× better at visual reproduction when the target website has simpler interaction patterns.

GPT-5 shows a broadly similar pattern but with compressed variance: Public Service VS = 27.4, FS = 56.0 vs. SaaS Platform VS = 16.7, FS = 40.5. The gap is smaller in absolute terms but similar in relative terms.

Claude-Sonnet-4.5 shows a notably different pattern: its best category is SaaS Platform (VS = 21.7, FS = 42.8), not Public Service (VS = 41.2, FS = 52.0). This inversion suggests that Claude-Sonnet-4.5 handles complex interaction patterns relatively better than simple informational layouts—a counterintuitive finding that would be invisible in an aggregate score. The paper notes this anomaly but does not deeply analyze its causes.

Functional Category Performance Analysis

Table 5 (Section 4.2, Finding 6) provides the most granular diagnostic: per-functional-category pass rates for three models under OpenHands on Level 3 full-stack tasks. The categories and results for Claude-Opus-4.5:

• Navigation & Routing: 66.3% pass rate
• Authentication & Authorization: 61.8%
• UI Feedback: 56.4%
• Search / Filter / Sorting: 55.6%
• Form Interaction: 49.2%
• CRUD Operations: 43.6%
• State Management: 43.2%
• File & Media Operations: 33.3%

The ranking reveals a clear hierarchy of difficulty. Navigation and Authentication—tasks involving discrete state transitions with well-defined success/failure signals—are the most reliable. State Management and CRUD Operations—tasks requiring persistent state tracking across components, correct data flow between frontend and backend, and consistency across database schema, API endpoints, and frontend display logic—show markedly lower pass rates. File & Media Operations are the worst, with only one-third of test cases passing, reflecting the difficulty of coordinating frontend logic with system-level resource handling.

Comparing models: GPT-5 achieves 56.8% on Form Interaction (vs. Claude-Opus-4.5's 49.2%) but only 30.8% on UI Feedback (vs. 56.4%), suggesting different capability profiles—GPT-5 may be stronger at implementing form logic but weaker at implementing visual feedback mechanisms like loading spinners, success toasts, and error highlighting. Claude-Sonnet-4.5 underperforms both on almost every category, with File & Media Operations at 0.0% (complete failure on all file/media test cases). This per-category breakdown transforms the benchmark from a summative assessment ("Claude-Opus-4.5 achieves FS 57.6") into a formative diagnostic ("the bottleneck is specifically State Management and CRUD, where pass rates are below 45% even for the strongest agent").

Ablation Studies and Robustness Checks

Framework comparison (OpenHands vs. Claude Code): Table 3 serves as a de facto ablation of coding agent framework choice. The finding—that most non-Claude models perform substantially better under OpenHands than under Claude Code—establishes that agent performance is not purely a function of the underlying multimodal model. For GPT-5 on full-stack tasks, switching frameworks changes the Functional Score from 23.4 (Claude Code) to 49.7 (OpenHands), a 2.1× improvement. For Gemini-3-Pro-Preview, the Level 3 FS changes from 12.9 to 22.6. The paper does not isolate which framework features drive this difference (tool integration, prompt structure, error recovery mechanisms, workspace management), but the magnitude of the effect demonstrates that framework design is a first-class variable in coding agent evaluation, not a minor implementation detail.

GUI agent verifier validation against human annotations: Section 4.4 reports that across ~100 randomly sampled test workflows from 64 tasks, the GUI agent verifier correctly judges 218 of 250 nodes (87.2% agreement) relative to human annotations. This is the primary robustness check for the functional evaluation pipeline. The residual 12.8% error is attributed to "model-intrinsic reasoning hallucinations." The paper does not break down errors by whether they are false positives (agent passes a failing test) vs. false negatives (agent fails a passing test), which would be important for understanding bias direction. The random sampling across 64 tasks (33% of the 193 total tasks) provides reasonable coverage, though the sample of 250 nodes from an unspecified total number of verification nodes makes it difficult to assess whether the sampled nodes are representative of the full distribution of test difficulty.

VLM judge validation against human preferences: Section 4.4 reports Spearman rank correlation between the VLM judge and human annotators across 100 randomly sampled prototypes. The median of 0.80 vs. human inter-annotator correlation of 0.78 indicates that the judge is essentially as reliable as a second human for ranking tasks on typical prototypes. The gap between median (0.80) and average (0.66) reveals the existence of challenging cases where the judge diverges significantly. The paper does not characterize these challenging cases—are they prototypes with ambiguous component boundaries, unusual color schemes, or complex overlapping layouts? Identifying the failure modes would guide future judge improvements and help users interpret when to trust judge scores.

Device-specific static webpage evaluation: The Level 1 results (Table 3) provide an implicit ablation of device form factor. By requiring reproduction at Desktop, Tablet, and Mobile resolutions for the same webpage, the benchmark reveals that responsive design capability is a distinct skill from single-resolution reproduction. The consistent Desktop → Tablet → Mobile degradation pattern across all models (Finding 2) demonstrates that this is a systematic weakness, not a model-specific quirk. An agent that perfectly reproduces the Desktop layout but fails on Mobile has not truly mastered visual reproduction—it has merely memorized a specific layout at a specific viewport width.

Model scale comparison (within the Qwen family): The pair Qwen3-VL-32B-Instruct vs. Qwen3-VL-8B-Instruct provides a natural ablation of model scale within a single architecture family. The 8B variant achieves non-zero (though near-zero) performance on Level 1 under Claude Code (Desktop VS = 12.4) and a 46% DSR under OpenHands, while the 32B variant achieves near-zero performance under all configurations (Level 1 Desktop VS = 0.0–0.9 under both frameworks). This is a counterintuitive result: the larger model performs worse than the smaller model on static webpage reproduction. The paper does not analyze this inversion, but it suggests that scale alone does not guarantee visual grounding capability—the 32B model may have been trained with different data mixtures or objectives that deprioritized UI-to-code translation relative to the 8B model.

Qualitative failure mode analysis: Section 4.3 and Appendix A.4.1 provide qualitative ablation through representative failure cases. At Level 1, failures include "misaligned layouts, incorrect sizes, and color mismatches, especially for regularly arranged components" and fragile asset handling where agents "over-rely on file names and lack robust multimodal grounding." At Level 2, failures compound: "visual fidelity degrades on subsequent pages, with missing or misaligned components, malfunctioning interactive elements, and broken navigation links." At Level 3, "deficiencies in long-horizon planning and execution become dominant"—projects may "fail to launch, crash during execution, or exhibit pervasive functional errors." Figures 10–12 (Appendix A.4.1) provide visual examples of these failure modes. While not quantitative ablations, these case studies establish face validity for the benchmark's difficulty and illustrate the specific capability gaps the quantitative scores reflect.

C4 validation set as contamination control: The benchmark construction pipeline's exclusive use of the C4 validation set (Section 2.2) is not directly tested as an ablation—the paper does not compare performance on C4-sourced tasks vs. tasks from other sources. However, the choice is motivated by contamination prevention: agents trained on web-crawled data are likely to have memorized popular websites but not C4-validation-set pages. The near-zero performance of multiple models on many tasks (e.g., Qwen models scoring 0.0 on Level 2 and 3) indirectly supports the claim that the tasks are not solved by memorization—if these models had memorized the target websites during training, they would score higher.

Critical Assessment

Does the benchmark actually measure what it claims to measure?

Vision2Web claims to evaluate "visual website development" across three levels of increasing complexity. The experiments demonstrate that agents achieve varying scores across these levels, and that performance degrades as complexity increases. But the critical question is whether the measured degradation reflects genuine capability gaps in the dimensions the paper claims, or whether it reflects artifacts of the evaluation framework.

The strongest evidence that the benchmark measures genuine capability is the differential degradation patterns across models. If the levels simply measured "harder tasks" in a generic sense, all models would degrade uniformly. Instead, Table 3 shows that Gemini-3-Pro-Preview achieves the highest static webpage scores (Desktop VS 63.3, beating Claude-Opus-4.5's 58.9) but collapses on full-stack tasks (VS 11.7 vs. Claude-Opus-4.5's 38.4). This pattern cannot be explained by general task difficulty—it requires a specific explanation: Gemini has strong visual reproduction capability but weak system integration capability. This is precisely the kind of capability disentanglement the hierarchical design was built to produce, and the experiments validate it.

However, there are important caveats about what the benchmark is actually measuring. Level 1 (static webpage) measures visual reproduction capability plus basic responsive design, but the scores are generated by a VLM judge that has its own biases—a 0.80 median Spearman correlation with human preferences means the judge disagrees with humans on roughly 20% of rankings in the worst case. If the judge systematically overpenalizes certain visual styles or underpenalizes certain error types, the Level 1 scores may not accurately reflect human-perceived visual fidelity. The paper's validation study demonstrates rank consistency but does not characterize the judge's absolute scoring calibration—whether a VS of 58.9 for Claude-Opus-4.5 means the output is "moderately good" or "barely acceptable" by human standards. Rank consistency is sufficient for leaderboard purposes (we care whether A > B) but not for interpreting absolute scores.

Levels 2 and 3 mix visual and functional evaluation, with FS measuring whether guided interactions produce expected outcomes. The guided actions $A_i$ in verification nodes constrain the agent through semantic descriptions ("Click the Log in button"), and the agent must locate elements in the specific implementation. If the GUI agent verifier fails to locate an element (12.8% error rate against human annotations), a functional test may fail not because the implementation is wrong but because the verifier couldn't find the button. This means FS scores conflate agent coding quality with verifier perceptual capability. The paper partially addresses this by validating the verifier against humans, but the 87.2% agreement means roughly 1 in 8 nodes is misjudged. For a full-stack task with 28.2 test cases on average (Table 2), this implies 3–4 misjudged nodes per task on average, which could substantially distort per-task FS scores.

Are the performance differences between models statistically reliable?

The paper does not report any measures of statistical reliability—no confidence intervals, no standard errors, no significance tests. The test set is 193 tasks total, with only 27 at Level 3 (full-stack). A difference between two agents on Level 3 (e.g., Claude-Opus-4.5's FS of 57.6 vs. GPT-5's FS of 49.7 under OpenHands) is based on at most 27 × 28.2 = 761 verification nodes, and likely fewer since some agents have DSR < 100% (tasks that fail to deploy contribute zero to the FS numerator, but it's unclear how they're handled in the denominator). The absence of confidence intervals makes it impossible to determine whether the observed differences are reliable or could be reversed with a different sample of tasks.

This is a significant methodological gap, particularly for a benchmark that is likely to be used for model comparison and leaderboard construction. The paper should at minimum report bootstrap confidence intervals for the main metrics or discuss the effective sample size for comparison purposes. Given the small number of full-stack tasks (27), differences on the order of 5–10 FS points may not be statistically significant.

The framework-dependent performance variation is large and unexplained.

The finding that GPT-5's full-stack FS nearly doubles under OpenHands (49.7) compared to Claude Code (23.4) is a major result—but the paper provides no analysis of why this happens. Is OpenHands providing better error recovery? More effective tool use scaffolding? Different prompting strategies? Different timeout or retry policies? Without understanding the mechanism, the result is both striking and frustrating: it demonstrates that framework choice matters enormously, but provides no guidance on how to design better frameworks or select frameworks for specific models.

This also complicates the interpretation of model comparisons. Is Claude-Opus-4.5 genuinely the best model for website development, or is it simply the model that happens to work best with the frameworks tested? Could a different framework configuration make GPT-5 competitive or superior? The paper's claim that "Claude-Opus-4.5 consistently achieves the strongest performance" (Finding 3) must be qualified by the observation that framework choice interacts with model performance in ways that are not understood.

The benchmark validates verifiers but does not characterize evaluation noise holistically.

Section 4.4 validates the GUI agent verifier and VLM judge separately, but does not characterize the end-to-end evaluation noise when both verifiers operate together on the same task. The GUI agent's 87.2% node-level agreement and the VLM judge's 0.80 median Spearman correlation are measured independently; the compound error when both are used sequentially in a workflow is not analyzed. If a workflow has 5 functional nodes and 3 visual nodes, and each functional node has a 12.8% error probability, the probability that at least one node is misjudged is approximately 50% (1 - 0.872^5). This means that for longer workflows, the per-task FS and VS scores may be substantially affected by verifier noise in addition to genuine agent capability variation.

The quarterly evaluator update plan (Section 4.4) is a pragmatic acknowledgment of this limitation, but it also means that Vision2Web scores from different time periods may not be directly comparable—a model evaluated with the Q1 2026 verifier may receive different scores than the same model evaluated with the Q2 2026 verifier if the verifier's accuracy improves. This is a deliberate tradeoff (accuracy over temporal comparability), but it complicates the benchmark's use for tracking progress over time.

The category-level analysis is insightful but based on small per-category sample sizes.

Table 4 reports VS and FS by website category, but each category contains only a handful of tasks. From Table 7 (Appendix A.1.2), full-stack task counts per subcategory are exactly 2 for most subcategories (Content subcategories, Transaction subcategories) and 1 for SaaS subcategories. Aggregating to the four macro categories, Public Service has 4 full-stack tasks (2 Government Portal + 2 Public Utility), Transaction has 8, Content has 10, SaaS has 5. The per-category scores in Table 4 are averages over 4–10 tasks—extremely small samples. A single anomalously difficult task in the SaaS category could substantially depress the average. The category-level differences are interesting and hypothesis-generating, but they should not be interpreted as reliable quantitative estimates of category-specific capability.

The benchmark does not test the boundary between what models can and cannot do.

Vision2Web establishes that current models struggle on full-stack website development, but it does not characterize how close they are to competence. Would a 2× improvement in a specific capability (e.g., state management) bring Claude-Opus-4.5 to FS > 80? Or are multiple capability breakthroughs needed? The benchmark's difficulty is fixed—tasks are either passed or failed—without graduated difficulty levels within Level 3 that could reveal partial progress. If a future model improves FS from 57.6 to 62.0, is that meaningful progress or measurement noise? The paper provides no mechanism for distinguishing incremental improvements from evaluation variance, which limits the benchmark's utility for tracking research progress at the frontier.

The benchmark does not explore compute scaling.

Unlike the reference example paper which systematically studied how performance varies with test-time compute budget, Vision2Web gives each agent the same task specification and a fixed time/retry budget (three attempts, 10 minutes each). It does not test whether giving agents more time, more retries, or more compute would improve performance. This is a reasonable design choice for a benchmark focused on capability measurement rather than compute optimization, but it means the reported scores are specific to a particular compute budget. A model that could achieve high performance given 10× more inference time would appear incompetent under the current protocol. The paper's contribution is evaluating capability at a fixed budget, not characterizing how capability scales with budget—both are valuable, but they answer different questions.

Missing experiments that would strengthen the paper:

• Statistical reliability analysis: confidence intervals on the main metrics, bootstrap estimates of ranking stability, power analysis for model comparison.
• Verifier error characterization: breakdown of GUI agent errors by type (false positive vs. false negative), analysis of whether errors are concentrated in specific task categories or difficulty levels.
• Framework ablation: controlled experiments varying specific framework features (prompt structure, tool availability, retry policies) to isolate what drives the OpenHands vs. Claude Code performance gap.
• Time-budget scaling: performance as a function of allowed development time or retry budget—does performance saturate at 10 minutes, or would agents continue improving given more time?
• Human baseline: expert human developers attempting the same tasks under comparable conditions (same inputs, similar time constraints) to establish what level of performance is achievable and how far current agents are from human-level capability.
• Inter-annotator agreement for task construction: the manual review stage (Section 2.2) involves human annotators evaluating website quality and assigning tasks to levels—reporting inter-annotator agreement would establish the reliability of the benchmark's construction, not just its evaluation.
• Sensitivity to prototype image quality: Level 1 results are stratified by prototype height (Figure 4), but the analysis does not control for other image properties (compression artifacts, color depth, content density) that might affect agent performance independent of actual task difficulty.

6. Limitations and Trade-offs

The Difficulty Estimation Analogy Does Not Apply — But a Parallel Overhead Exists in Verification Infrastructure Cost

Vision2Web is a benchmark, not an inference-time method, so the "difficulty estimation cost" problem from the reference example does not directly apply. However, the paper does have a structurally analogous issue: the verification infrastructure has a significant construction and maintenance cost that is not trivialized by the benchmark's design. The workflow-based verification framework is the paper's core evaluation innovation, but the cost of producing the test workflows that drive it is substantial — and this cost is concentrated on the most valuable part of the benchmark (Level 3 full-stack tasks).

The assumption. The paper assumes that the expert-in-the-loop annotation process described in Section 3.3 is a one-time cost that produces reusable test workflows. For Level 3 full-stack tasks specifically, "domain experts first draft high-level testing workflows from requirement documents and design prototypes, explicitly encoding task dependencies, execution order, and shared application states," after which "Claude Code further refines and expands the workflows by translating high-level specifications into executable interaction sequences and verifiable criteria." The paper does not quantify the human effort involved — how many person-hours per task, how many annotators, or how many rounds of iteration between experts and Claude Code.

The consequence. The consequence is that extending or replicating Vision2Web is expensive in a way that scales with the most valuable part of the benchmark. The 27 full-stack tasks are the hardest to construct (requiring expert-in-the-loop annotation of long-horizon workflows) but also the most diagnostically valuable (they reveal system integration failures invisible at lower levels). If a research group wants to build a larger full-stack benchmark following Vision2Web's methodology, they face a non-trivial annotation cost that is not parameterized or estimated in the paper. This limits the benchmark's role as a template for future work — the methodology is sound, but the cost of applying it to new domains or larger scales is unknown. Additionally, the quarterly evaluator update plan (Section 4.4) means that the VLM judge and GUI agent verifier will be replaced periodically, requiring re-validation against human annotations each time — an ongoing maintenance cost that is acknowledged but not quantified.

What evidence exists. The paper provides no direct measurement of annotation cost. The only data point is the final benchmark size: 193 tasks with 1,255 test cases. The Level 3 tasks have 28.2 test cases on average (Table 2), and the test case type distribution (Figure 3) shows diversity across eight functional categories, implying substantial annotation effort per task. The expert-in-the-loop strategy is described qualitatively but not costed.

Mitigation status. The paper does not attempt to mitigate this — it acknowledges the annotation strategy but treats it as a fixed cost of benchmark construction rather than a limitation to be solved. The use of Claude Code to "refine and expand" expert drafts partially automates the process, but the expert drafting step remains manual. Future work could explore fully automated workflow generation from requirement documents and prototypes, but this is not addressed.

The Verification Framework's Accuracy Is Bounded by Evaluator Model Quality — And the Bounds Are Material

The workflow-based verification paradigm delegates pass/fail judgments and visual fidelity scoring to AI agents (GLM-4.6V for the GUI agent verifier, Gemini-3-Pro-Preview for the VLM judge). The paper validates these agents against human annotations, but the validation results reveal error rates that are non-trivial for a benchmark that will be used for model comparison at the frontier.

The assumption. Section 4.4 assumes that the verifiers are accurate enough to serve as evaluation oracles — that a functional node judged "pass" by the GUI agent genuinely indicates correct agent output, and that a Visual Score difference between agents reflects genuine visual quality differences. The paper validates this assumption by measuring agreement with humans: 87.2% node-level agreement for the GUI agent (218/250 nodes correctly judged) and a median Spearman correlation of 0.80 for the VLM judge (vs. human inter-annotator correlation of 0.78).

The consequence. The 87.2% node-level agreement means that roughly 1 in 8 functional verification nodes is misjudged. For a full-stack task with 28.2 test cases on average (Table 2), this implies approximately 3–4 misjudged nodes per task. The Functional Score is computed as the proportion of passed nodes, so these misjudgments directly affect the metric. The paper does not characterize whether errors are biased — does the verifier systematically overestimate agent correctness (false positives, inflating scores) or underestimate it (false negatives, deflating scores)? If the error is biased, comparisons between agents could be systematically distorted. If it is symmetric noise, comparisons are attenuated but not biased — but the small number of full-stack tasks (27) means that even symmetric noise could produce unreliable rankings when score differences are small (e.g., the 7.9-point FS gap between Claude-Opus-4.5 at 57.6 and GPT-5 at 49.7 under OpenHands, Table 3).

For the VLM judge, the gap between median Spearman correlation (0.80) and average Spearman correlation (0.66) reveals that there exist "challenging cases" where the judge diverges substantially from human preferences. On those cases, Visual Scores are unreliable as measures of visual fidelity. The paper does not characterize how many such cases exist, how severely they diverge, or whether they cluster in specific task categories or prototype types.

The quarterly evaluator update plan introduces an additional complication: scores from different time periods may not be comparable. If the Q1 2026 verifier judges a model and the Q2 2026 verifier judges the same model (or a competitor model) with different accuracy, the scores are not on the same scale. The paper explicitly plans for this ("update both the VLM judge and GUI agent on a quarterly basis") but does not propose a mechanism for score calibration across evaluator versions.

What evidence exists. Section 4.4 provides the validation numbers directly. The 87.2% figure comes from 250 nodes across ~100 workflows from 64 tasks. The Spearman correlations come from 100 randomly sampled prototypes. The paper acknowledges the residual error ("inaccuracies are largely attributable to model-intrinsic reasoning hallucinations") and the challenging cases ("leaving room for improvement in challenging cases"). The compound error when both verifiers operate in sequence is not characterized.

Mitigation status. Partial. The paper validates the verifiers systematically, which is methodologically stronger than most benchmarks (which typically assume their evaluation metrics are correct without validation). The quarterly update plan acknowledges the need for ongoing improvement. However, the paper does not propose specific mitigations for the error that currently exists — no ensemble verification across multiple agents, no confidence scores on pass/fail judgments, no flagging of low-confidence nodes for human review. The verifier error is treated as an acceptable cost rather than a problem to be solved within the current benchmark version.

The Benchmark Is Single-Domain (Website Development), Single-Source (C4 Validation Set), and Single-Output-Modality (Code)

Vision2Web evaluates one specific task — visual website development — using data from one specific source — the C4 validation set — producing one specific output type — deployable web application code. The paper does not claim otherwise, but the scope limitation has implications for how broadly the benchmark's findings should be interpreted.

The assumption. The paper's task formulation (Section 2.1) restricts evaluation to website development across three complexity levels. The dataset construction pipeline (Section 2.2) sources all tasks "exclusively from the C4 validation set to avoid potential leakage from popular websites." The output is always a web application accessible at localhost:3000. The paper positions website development as "an ideal testbed for evaluating long-horizon multimodal agent intelligence" (Section 1) because it spans the full software lifecycle and requires coordinated understanding across modalities.

The consequence. The findings about which capabilities agents lack — cross-page coordination, persistent state reasoning, long-horizon planning (Findings 1, 5, 6) — are established only for the specific intersection of visual website development, C4-sourced web designs, and web application technology stacks. It is plausible that these deficiencies generalize to other multimodal software engineering tasks (mobile app development from mockups, game UI implementation from designs, data dashboard construction from specifications), but the paper provides no evidence. Similarly, it is plausible that agents perform differently on websites sourced from different distributions — the C4 validation set's pages may systematically differ from the broader web in complexity, design style, or interactive patterns.

The single-source constraint is a deliberate contamination-prevention choice, but it means the benchmark cannot detect whether agents have memorized C4-style web design patterns specifically. If a model was trained on web data that included pages similar in distribution to C4 (even if not the exact pages), it might perform better on Vision2Web than on websites drawn from a truly out-of-distribution source. This is not a contamination problem in the traditional sense (the exact test pages are unlikely to be memorized), but a distribution shift problem — the benchmark measures performance on a specific web design distribution, and generalizing to "real-world website development" requires assuming that C4 validation pages are representative of the broader web.

What evidence exists. The paper's category distribution (Figure 2, Table 7) shows diversity across 16 subcategories and four major types, which partially addresses domain narrowness — the benchmark does not test only one kind of website. The structural complexity comparison to Design2Code (Figure 6) shows that Vision2Web's static pages are substantially more complex than existing benchmarks' pages, supporting the claim that the benchmark reflects real-world complexity. However, no experiment compares C4-sourced tasks to tasks from another source, so the representativeness assumption is untested.

Mitigation status. Not addressed. The paper does not claim generalizability beyond website development, and the title and abstract are explicit about the scope ("A Hierarchical Benchmark for Visual Website Development"). This is a limitation in the sense of scope rather than a flaw — the benchmark does exactly what it claims to do, but the claims are bounded to one domain. The paper does not suggest future work on extending the methodology to other software engineering domains, though the workflow-based verification framework is, in principle, domain-agnostic and could be applied to mobile development, desktop applications, or game UI implementation.

The Small Number of Full-Stack Tasks (27) Limits the Reliability of the Most Diagnostically Valuable Comparisons

The hierarchical design's diagnostic power — attributing performance degradation to specific skill gaps — is most valuable at Level 3 (full-stack), where system integration failures become visible. But Level 3 has only 27 tasks, and this small sample size has consequences for the reliability of model comparisons and category-level analyses.

The assumption. The paper treats the 27 full-stack tasks as a representative sample of the broader full-stack website development task distribution. The category-level analysis (Table 4) further subdivides these 27 tasks into four macro categories, each containing 4–10 tasks. The per-functional-category analysis (Table 5) aggregates across all full-stack tasks. The paper reports point estimates (VS, FS) without confidence intervals or standard errors.

The consequence. With 27 tasks at Level 3, the effective sample size for model comparison is small. The Functional Score difference between Claude-Opus-4.5 (57.6) and GPT-5 (49.7) under OpenHands — a 7.9-point gap — is based on at most 27 × 28.2 = 761 verification nodes (the average test case count from Table 2), and fewer if some tasks fail to deploy (DSR is 96.3% for Opus-4.5 and 100% for GPT-5, so deployment failures don't change the comparison meaningfully here). Without confidence intervals, we cannot determine whether this 7.9-point difference is statistically reliable or could be reversed with a different sample of 27 tasks.

The category-level analysis in Table 4 is even more fragile. The SaaS Platform category contains 5 full-stack tasks (Table 7: CRM, HCM, ERP, TMS, AI Platform each have 1 full-stack task). Claude-Opus-4.5's SaaS Platform FS of 39.9 is an average over at most 5 × 28.2 = 141 verification nodes, and quite possibly fewer if some tasks have fewer than the average number of test cases. A single anomalously easy or difficult task in this category could shift the average by several points. The observation that SaaS Platforms yield the weakest results is hypothesis-generating but should not be treated as a precise quantitative finding.

The per-functional-category pass rates in Table 5 (e.g., File & Media Operations at 33.3% for Claude-Opus-4.5) are aggregated across all full-stack tasks. The number of test cases per category is not reported, but Figure 3 shows the distribution visually — some categories (File & Media Operations) may have very few test cases. A 33.3% pass rate could represent 1 of 3 test cases passed, which is a point estimate with extremely high variance.

What evidence exists. The paper provides the task counts: Table 7 shows that each subcategory has either 1 or 2 full-stack tasks, for a total of 27. Table 2 shows the average test cases per full-stack task (28.2 ± 12.0). The standard deviation of 12.0 indicates substantial variation in the number of test cases per task, meaning some tasks may have substantially fewer than 28 test cases. No confidence intervals or statistical tests are reported.

Mitigation status. Not addressed. The paper does not discuss sample size as a limitation, does not report confidence intervals or standard errors, and does not perform any statistical tests on model comparisons. This is a significant methodological gap given the benchmark's likely use case — comparing models and tracking progress over time. As models improve and performance gaps narrow, the statistical reliability of small-sample comparisons becomes increasingly important, and Vision2Web provides no tools for assessing it.

The Benchmark Does Not Explore How Performance Scales with Compute, Time, or Framework Configuration — It Measures Capability at a Fixed Budget

The paper evaluates agents under a fixed protocol: three deployment attempts with a 10-minute timeout per attempt, using two specific coding agent frameworks (OpenHands and Claude Code). It does not systematically vary the compute budget, the allowed development time, or the framework configuration. This is a design choice, not a flaw, but it has implications for what the reported scores mean and how they should be used.

The assumption. Section 4.1 specifies the evaluation protocol: "each agent generates a startup script to run projects on a fixed port, with up to three iterations allowed to collect more analyzable evaluation results. Deployments exceeding 10 minutes or producing errors are treated as failures." The paper implicitly assumes that this protocol is sufficient to elicit each model's best achievable performance and that the results reflect genuine capability differences rather than sensitivity to the specific budget chosen.

The consequence. The reported scores are specific to a particular compute budget and framework configuration. We do not know whether:

• More development time would close the gap between weaker and stronger models. If Gemini-3-Pro-Preview could achieve Claude-Opus-4.5-level full-stack performance given 10× more inference time, the current results would reflect a speed difference rather than a capability difference.
• More deployment retries would recover failing tasks. The 10-minute timeout and three-attempt limit are arbitrary thresholds. An agent that deploys correctly on the fourth attempt (after 13 minutes) is treated identically to one that never deploys at all — both contribute to a lower DSR, and if non-deployment propagates to FS = 0, the functional score is zeroed out.
• Different framework configurations would change the relative ordering of models. The finding that GPT-5's full-stack FS nearly doubles under OpenHands (49.7) vs. Claude Code (23.4) demonstrates that framework choice is a first-class variable, but the paper provides no mechanism for determining whether OpenHands is "better" for GPT-5 or whether a different framework configuration would produce even higher scores.

The 10-minute timeout is particularly consequential because it bounds the agent's ability to engage in extended debugging, iterative refinement, or complex architectural planning. The paper's Finding 6 (Section 4.2) identifies "deficiencies in long-horizon planning and execution" as a key failure mode — but the experimental protocol itself imposes a horizon limit that may truncate successful long-horizon execution. An agent that would eventually succeed given 60 minutes is indistinguishable from one that would never succeed under the 10-minute cap.

What evidence exists. The framework comparison (Table 3) provides direct evidence that the choice of protocol matters enormously — the same model can score dramatically differently under different frameworks. The DSR column provides indirect evidence of budget sensitivity: Claude-Opus-4.5's DSR on full-stack is 96.3% under OpenHands, meaning 1 of 27 tasks failed to deploy within the time/retry budget. We don't know whether that task would have succeeded with more time or retries. No systematic time-scaling or retry-scaling experiments are reported.

Mitigation status. Not addressed as a limitation — the paper presents the fixed-budget protocol as the evaluation specification and does not discuss its implications for score interpretation. The framework comparison is presented as Finding 4 ("Agent performance varies across frameworks") rather than as evidence that the reported scores are protocol-dependent. This is the most significant open question for a practitioner interpreting the results: if I give Claude-Opus-4.5 4× more development time or use a different framework, will it score meaningfully higher on full-stack tasks, or has it reached a capability ceiling that additional compute cannot overcome? The paper provides no evidence either way.

The Absence of a Human Baseline Makes It Impossible to Interpret How Far Agents Are from Competence

The paper evaluates eight models against each other, establishing that Claude-Opus-4.5 is the strongest and that Qwen models largely fail. But without a human baseline — what scores do competent human developers achieve on the same tasks under comparable conditions? — the absolute performance levels are uninterpretable. Is a Visual Score of 38.4 on full-stack tasks (Claude-Opus-4.5 under OpenHands) "impressive given the difficulty" or "far from usable"? Is a Functional Score of 57.6 "approaching professional competence" or "still failing on nearly half of expected behaviors"?

The assumption. The paper implicitly assumes that model-to-model comparison is sufficient for benchmarking progress — that knowing Claude-Opus-4.5 is stronger than GPT-5 is the relevant signal, and absolute scores provide only ordinal information.

The consequence. Without a human baseline, the benchmark cannot answer the most practically important question: how close are coding agents to replacing or augmenting human developers on visual website development tasks? A practitioner deciding whether to deploy Claude-Opus-4.5 for frontend development needs to know whether its 38.4 VS / 57.6 FS represents "good enough for production with minor cleanup" or "requires complete reimplementation by a human." The category-level analysis (Table 4) shows that performance varies dramatically by website type — Public Service websites achieve 56.9 VS / 60.0 FS, which may be approaching usable quality, while SaaS Platforms achieve 22.9 VS / 39.9 FS, which almost certainly is not. But without a human reference point, even the Public Service numbers are ambiguous.

The human baseline would also calibrate the difficulty of the benchmark itself. If expert human developers score VS ~95 and FS ~100 on Level 1 static pages (indicating the tasks are straightforward for humans), the model scores of 50–60 represent a substantial gap. If humans score VS ~60 and FS ~80 on full-stack tasks under the same time constraints (indicating the tasks are genuinely hard even for experts), the model scores of 38.4 and 57.6 represent much more impressive performance relative to human capability.

What evidence exists. Zero. The paper does not report any human performance on Vision2Web tasks, does not compare model scores to any human reference, and does not discuss the interpretability of absolute scores. The verification validation studies (Section 4.4) measure human-verifier agreement but not human task-completion performance.

Mitigation status. Not addressed. This is a significant gap given that Vision2Web is positioned as a benchmark for "evaluating the visual website development capabilities of multimodal coding agents" (Section 2.3). Knowing that agents are getting better relative to each other is useful for tracking research progress; knowing how close they are to human-level performance is essential for understanding when (or whether) they can be deployed in practice. The paper provides only the former.

The Verifier and the Coding Agent May Share Failure Modes — Visual Grounding Weaknesses in the Evaluator Could Systematically Favor or Penalize Specific Agent Output Styles

The GUI agent verifier and VLM judge are themselves VLM-based systems that perform visual understanding tasks — locating elements on rendered pages and comparing visual fidelity. The coding agents being evaluated are also VLM-based systems performing visual understanding tasks — translating prototypes into code. This creates a potential for correlated errors where the verifier misjudges agent outputs in ways that are systematically related to the visual properties of those outputs, not randomly distributed noise.

The assumption. Section 4.4 validates the verifiers against human annotations and finds acceptable agreement (87.2% node-level, 0.80 median Spearman). The paper implicitly assumes that the verifier errors are uncorrelated with the coding agent's output characteristics — that the verifier would judge a human-written correct implementation with the same accuracy distribution as a model-written correct implementation.

The consequence. If the VLM judge has specific visual biases — for example, preferring certain color palettes, layout patterns, or typography styles — it may systematically favor coding agents whose outputs happen to align with those biases. Conversely, if a coding agent produces outputs that are correct but unusual (unconventional but valid HTML structures, non-standard CSS approaches, atypical component arrangements), the VLM judge may penalize them relative to more conventional outputs, even if both are equally correct by human standards. This would create systematic bias in model rankings that is invisible to the current validation studies, which measure agreement with humans on the same set of agent outputs without varying the output distribution.

A more specific concern: the paper notes (Section 4.3) that one failure mode is that "agents over-rely on file names and lack robust multimodal grounding, causing visible inconsistencies when assets are unnamed or ambiguously referenced." If a coding agent produces a page where the layout is correct but an image is missing (showing a broken image placeholder), the VLM judge should penalize the "multimedia missing or incorrect" component. But if the VLM judge itself struggles with visual grounding — misidentifying what a broken image placeholder represents — it may assign inconsistent scores to this failure mode. The residual error in the GUI agent verifier (12.8%) is attributed to "model-intrinsic reasoning hallucinations" — but if the coding agent also exhibits reasoning hallucinations that produce subtly malformed outputs, the verifier's hallucinations may interact with the coding agent's hallucinations in unpredictable ways.

What evidence exists. The paper provides no evidence that verifier errors are uncorrelated with coding agent output characteristics. The validation studies (Section 4.4) measure agreement with humans on a fixed set of agent outputs — they do not test whether the verifier's accuracy varies across different output distributions (e.g., human-written vs. Claude-written vs. GPT-5-written implementations). The paper does not report whether the 12.8% GUI agent error rate differs across coding agents or task categories, which would be a first-order check for correlated errors.

Mitigation status. Not addressed. This is a subtle methodological concern that is difficult to detect without controlled experiments where human-written correct implementations are inserted into the evaluation pipeline alongside model-written implementations to test for verifier bias. The quarterly evaluator update plan (using newer VLMs as evaluators) may incidentally reduce the risk if newer VLMs have different (and hopefully less correlated) failure modes, but this is speculative and not tested.

7. Implications and Future Directions

How This Work Changes the Landscape

Vision2Web makes three interconnected contributions to the evaluation landscape, each operating at a different level of methodological abstraction: (1) a hierarchical capability disentanglement framework that transforms benchmark design from "measure aggregate performance on harder tasks" to "construct a controlled experiment isolating specific skill dimensions," (2) a workflow-based agent verification paradigm that resolves the long-standing tension between brittle scripted testing and unconstrained autonomous evaluation, and (3) a construct validity critique of existing benchmarks, supported by quantitative structural complexity analysis, that redefines what "realistic" means for UI-to-code and web development evaluation. Together, these constitute a methodological shift rather than a paradigm shift—the paper does not introduce a new class of models or a fundamentally new evaluation theory, but it establishes patterns and standards that raise the bar for what counts as rigorous evaluation in this area.

The hierarchical design as a diagnostic instrument is the most transferable contribution. The insight that performance differences between benchmark levels can be attributed to specific capability gaps—because the levels are architectured to add one capability dimension while holding others constant—is not domain-specific. It applies to any area where agent capability is multi-dimensional and evaluation should support attribution of failure to specific sub-skills. Consider robotic manipulation: a hierarchical benchmark might have Level 1 (visual perception from a fixed camera), Level 2 (pick-and-place with known object geometry), Level 3 (multi-step assembly with tool use and occluded views). A performance drop between Level 2 and Level 3 isolates the specific deficit in occlusion reasoning and tool coordination, just as Vision2Web's Level 2 to Level 3 drop isolates system integration deficits. The paper does not develop this generalization explicitly, but the pattern it establishes—cumulative levels where each adds exactly one new capability axis—is a template that other evaluation domains can adopt directly.

This diagnostic capability is what distinguishes the hierarchical design from the common practice of simply having "easy," "medium," and "hard" splits in a benchmark. Most benchmarks increase difficulty by combining multiple harder sub-tasks or selecting more challenging instances of the same task type, which confounds capability dimensions. Vision2Web's levels are qualitatively different task types (static reproduction, interactive frontend, full-stack system) that share a common visual grounding requirement. The design principle is orthogonal capability axes, not linear difficulty scaling, and this principle is what enables attribution.

The workflow-based verification paradigm resolves a tension that has blocked progress on end-to-end evaluation. Before Vision2Web, the field was stuck between two unsatisfactory approaches for evaluating open-ended software outputs: unit tests with hardcoded selectors (brittle to implementation diversity, false negatives on correct but differently-structured code) and autonomous LLM-based evaluators given vague objectives (unreproducible, unconstrained, poor coverage). The paper's key move—constraining how the agent interacts ($A_i$: guided actions) while preserving flexibility in what is verified ($V_i$: implementation-agnostic validation)—creates a middle path that is both reproducible and tolerant of implementation diversity. This is not a "better prompting" solution; it is a restructuring of the evaluation problem that separates execution control from correctness judgment.

The formalization as a dependency graph with shared context $C_i = \{H_{<i}, O_i, A_i, V_i\}$ makes this separation explicit and executable. The propagation of execution history through sequential nodes solves the state dependency problem that plagued prior testing frameworks—long-horizon tests could not maintain coherent state across steps because each step was either independent (brittle to ordering) or fully scripted (brittle to implementation). By constraining actions while exposing history, Vision2Web's workflow-based approach enables 20+ step test sequences that maintain state coherence without requiring the verifier to be an expert navigator.

This verification paradigm is the paper's most distinctive technical contribution and the one most likely to be adopted by subsequent benchmarks. The specific instantiation choices (GLM-4.6V for the GUI agent, Gemini-3-Pro-Preview for the VLM judge) are not the contribution—the formal framework and the separation of concerns are. Any benchmark evaluating open-ended system outputs where correctness is implementation-agnostic but test procedures are structured can adopt this approach: mobile app development, game UI implementation, data pipeline construction, infrastructure-as-code generation. The annual evaluator update plan (quarterly backbone model upgrades) is a pragmatic accompaniment that keeps the framework relevant as VLMs improve, though it introduces temporal comparability challenges that the paper does not fully resolve.

The construct validity critique of existing benchmarks is actionable rather than merely critical. The paper's Figure 6 comparison—Vision2Web static pages average 1,385 HTML tags, 22 levels of DOM depth, and 40 unique tag types vs. Design2Code-Hard's 251 tags, 10 levels, and 22 unique types—is not just a "ours is harder" brag. It is evidence for a specific claim: existing UI-to-code benchmarks evaluate a qualitatively different task than real-world website reproduction, and scores on those benchmarks may not transfer to realistic settings. The quantitative nature of the evidence (5.5× more structural elements, 2.2× deeper nesting, 1.8× more tag diversity) makes the claim falsifiable and specifies the magnitude of the representativeness gap.

This changes what the field should expect from benchmark construction. A benchmark claiming to measure "web development capability" now has an evidentiary burden: demonstrate that the tasks reflect the structural complexity, interaction diversity, and functional scope of real-world web development, not just that they are "hard." The paper's multi-stage construction pipeline with quantitative filtering criteria (structural assessment removing pages with malformed or overly simple DOMs, VLM-based content screening for functional richness, manual review for quality and category balance) provides a template for meeting this burden. Future benchmarks in this space that do not provide comparable complexity analysis will be justifiably scrutinized for construct validity.

This work also implicitly redirects research attention. The paper's Finding 6 (Table 5) reveals that State Management (43.2% pass rate for the strongest agent), CRUD Operations (43.6%), and File & Media Operations (33.3%) are the critical failure categories, while Navigation (66.3%) and Authentication (61.8%) are comparatively strong. This suggests that research investment in improving agent navigation or authentication capabilities has diminishing returns relative to investment in state management and cross-layer data consistency. Without this category-level diagnostic, researchers might waste effort optimizing already-strong capabilities while the true bottlenecks remain unaddressed.

The finding that framework choice can nearly double performance (GPT-5's full-stack FS: 49.7 under OpenHands vs. 23.4 under Claude Code) redirects attention toward framework design as a first-class research variable, not an implementation detail. The paper's results imply that a significant fraction of the performance gap between frontier models and usable coding agents may be closable through better framework engineering rather than better base models—a hypothesis that the current results cannot test but strongly suggest.

Follow-Up Research This Work Enables

A study characterizing how coding agent performance scales with inference-time compute budget on Vision2Web. The paper evaluates agents at a fixed budget (three deployment attempts, 10-minute timeout each) and reports a single point on the performance curve. A systematic scaling study varying the allowed development time (e.g., 5 min, 10 min, 30 min, 60 min, 120 min), the number of retry attempts (1, 3, 10, unlimited), or the generation budget (limiting total tokens the agent can consume) would characterize the compute-performance curve for coding agents on website development tasks. This would answer the question posed but not addressed by the current paper: does Claude-Opus-4.5's superior performance reflect a genuine capability advantage, or does it merely converge faster to the same asymptote that other models would reach given more compute? The experiment would run a subset of Vision2Web tasks (particularly the 27 full-stack Level 3 tasks) at multiple budget levels for multiple models, producing scaling curves analogous to the reference example paper's test-time compute scaling analysis. If models converge to similar performance at high budgets, the takeaway is that inference-time compute can substitute for model capability in this domain—a practically important finding for deployment decisions. If they diverge, the takeaway is that certain architectural or training properties create hard capability ceilings that compute cannot overcome.

Cross-domain replication of the workflow-based verification paradigm for mobile app development. Vision2Web's verification framework is formalized in domain-agnostic terms (dependency graphs of verification nodes, GUI agent with guided actions, VLM judge with component-level comparison), but it is validated only on web applications. A follow-up benchmark applying the same paradigm to mobile app development from UI mockups would test the framework's generality and reveal whether the capability patterns observed on web tasks (strong navigation/authentication, weak state management/CRUD, catastrophic full-stack degradation) are domain-specific or reflect general coding agent limitations. The benchmark would replace web prototypes with mobile screen mockups (iOS/Android), replace the browser-based GUI agent with a mobile device emulator agent, and retain the same workflow-based test case structure with guided actions adapted to touch interactions (tap, swipe, long-press instead of click, scroll). The key question: do agents exhibit the same hierarchy of functional category difficulty (e.g., is state management still the bottleneck?), or does the mobile context introduce new failure modes (e.g., gesture handling, platform-specific UI conventions, offline-first data patterns)? A strong follow-up would use a comparable number of tasks (~150-200) across the same three-level hierarchy, source tasks from real-world app designs (not synthetic mockups), and compare model performance patterns to Vision2Web's Table 5 to identify domain-specific vs. domain-general weaknesses.

An analysis of verifier bias: do VLM judges systematically favor certain implementation styles? The paper validates the VLM judge's rank consistency with human preferences (median Spearman correlation 0.80), but does not test whether the judge exhibits systematic bias toward or against specific implementation approaches—e.g., penalizing CSS Grid layouts relative to Flexbox, favoring certain color palettes, or misjudging dark-mode implementations. A controlled bias study would construct minimal pairs of implementations that are functionally and visually equivalent by human standards but differ in implementation approach (different CSS frameworks, different HTML structures, different component library choices, different responsive design strategies), then measure whether the VLM judge assigns systematically different scores to the two implementations when compared to the same prototype. A finding of no systematic bias would strengthen confidence in the judge as an unbiased evaluator. A finding of bias would identify specific implementation patterns that achieve inflated or deflated scores, which is crucial information for interpreting Vision2Web leaderboard results—a model that happens to adopt judge-preferred patterns may appear stronger than it genuinely is.

Training difficulty prediction models to triage tasks between agent autonomy and human handoff. One of the paper's most practically relevant findings is the extreme variance in agent performance across website categories: Claude-Opus-4.5 achieves VS/FS of 56.9/60.0 on Public Service websites but 22.9/39.9 on SaaS Platforms (Table 4). A follow-up study would train a classifier that predicts, from the task specification alone (requirement document text, prototype image embeddings, structural features like expected number of pages and test case categories), whether a given full-stack task is likely to fall into a "high agent competence" category (VS > 40) or a "low agent competence" category (VS < 25). The training data would be Vision2Web's 27 full-stack tasks with their category labels and scores, augmented with synthetic tasks generated by varying requirement document parameters. A strong predictor would enable a practical deployment pipeline: route easy tasks (predicted high competence) to the coding agent for autonomous completion, route hard tasks (predicted low competence) to human developers, and route borderline tasks to agent execution with automatic flagging for human review. The key metric is whether the classifier can achieve >80% precision in identifying tasks where the agent will achieve acceptable quality, effectively using Vision2Web's category-level diagnostic to inform deployment decisions on new, unseen tasks.

A study combining PRM-style verifier-guided search with coding agent frameworks for website development. The paper's results show that even the strongest coding agent (Claude-Opus-4.5) achieves only FS = 57.6 on full-stack tasks, with State Management and CRUD Operations below 45% pass rate. A follow-up could test whether process-level verification during development—analogous to the process reward model (PRM) search described in the reference example paper, but applied to code generation rather than math problem solving—improves full-stack task completion. The approach: at each step of the development workflow (after the agent proposes a database schema, after it implements an API endpoint, after it creates a frontend component), a specialized code verifier assesses whether the intermediate output is consistent with the requirement document and prototypes, providing feedback that the agent can use to revise before proceeding. The comparison is against the baseline Vision2Web protocol where the agent receives no intermediate verification. A finding that PRM-guided development improves full-stack FS from 57.6 to, say, 70+ would demonstrate that the bottleneck is not fundamental coding inability but rather the agent's inability to self-verify intermediate outputs in long-horizon development—a finding with direct implications for coding agent architecture. A null result (no improvement from intermediate verification) would suggest the bottleneck is deeper, potentially in the models' ability to reason about multi-layer system consistency even when given feedback, redirecting research toward architectural improvements rather than verification scaffolding.

Human baseline establishment on Vision2Web to calibrate absolute performance interpretation. The paper compares eight models against each other but provides no human reference point, making the absolute scores uninterpretable. A follow-up study would recruit professional frontend/full-stack developers (5-10 participants) to complete a representative subset of Vision2Web tasks (sampled across levels and categories, ~20-30 tasks total) under time constraints comparable to the agent protocol (e.g., 30-60 minutes per full-stack task, with access to the same prototype images and requirement documents). The output would be human VS and FS distributions that calibrate what "good" performance looks like. Key questions: do expert humans achieve VS > 80 and FS > 90 on full-stack tasks, indicating that current agent scores of 38.4/57.6 represent a large capability gap? Or do humans also struggle (VS ~60, FS ~75), indicating that the tasks are genuinely hard and agent performance is more impressive in relative terms? The human baseline would also reveal whether the per-category performance pattern observed in agents (Public Service > Transaction > Content > SaaS Platform) mirrors human difficulty perception, or whether humans show a different pattern—if humans find SaaS Platforms equally difficult, the agent pattern reflects task difficulty rather than agent-specific weakness; if humans find SaaS Platforms relatively easy, the agent pattern reflects a capability gap specific to complex interaction handling.

Practical Applications and Downstream Use Cases

Deployment-time triage for AI-assisted web development pipelines. The most directly actionable application of Vision2Web's findings is a triage system that uses the benchmark's category-level diagnostic to decide, on a per-task basis, whether to delegate to an AI coding agent or route to a human developer. The paper's Table 4 shows that Claude-Opus-4.5 achieves acceptable performance on Public Service websites (VS/FS of 56.9/60.0 under OpenHands) but unacceptable performance on SaaS Platforms (VS/FS of 22.9/39.9). A deployment pipeline could classify incoming development requests by website category (using a lightweight classifier trained on requirement documents and prototype features, as proposed above), route Public Service and Content tasks to the coding agent for first-pass implementation (with human review of the output), and route SaaS Platform tasks directly to human developers. The expected efficiency gain is substantial: if 30% of incoming full-stack tasks fall into the "high agent competence" categories and the agent produces output requiring, say, 50% less human rework than starting from scratch, the pipeline reduces total human developer hours by approximately 15% compared to a uniform human-only workflow. This is a concrete deployment scenario that does not require any model improvements—it uses today's model capabilities, as measured by Vision2Web, to optimize work allocation.

Benchmark-driven evaluation for coding agent platform selection in enterprises. Organizations evaluating whether to adopt AI coding assistants for web development (comparing Claude Code, Cursor, GitHub Copilot, or custom OpenHands-based deployments) can use Vision2Web's framework to measure which agent-framework combination performs best on their specific website category mix. Table 3 demonstrates that the same model can score dramatically differently under different frameworks—GPT-5's full-stack FS is 49.7 under OpenHands but 23.4 under Claude Code. An enterprise whose internal projects skew toward Transaction and Content websites (the majority of real-world web development) would use Vision2Web's category-level results (Table 4) to identify which agent-framework pair maximizes quality in those specific categories, rather than relying on aggregate benchmark rankings that may be dominated by categories irrelevant to their use case. The workflow-based verification framework's reproducibility means this evaluation can be run in-house on proprietary prototypes and requirements, not just on Vision2Web's public tasks, by adapting the test workflow annotation process (Section 3.3) to internal design specifications.

Continuous capability monitoring for coding agent updates. As coding agent platforms release new model versions (e.g., Claude-Opus-4.5 → 5.0, GPT-5 → GPT-6), organizations need to know whether the update improves, degrades, or leaves unchanged the specific capabilities relevant to their workflows. Vision2Web's hierarchical design enables structured regression testing: run the new model version on the full benchmark (or a representative subset) and compare per-level, per-category, and per-functional-category scores to the previous version's baseline. A finding that the new model improves full-stack FS from 57.6 to 65.0 but degrades File & Media Operations from 33.3 to 20.0 (an example hypothetical pattern) would alert the organization that the update is not uniformly beneficial and that media-heavy projects should defer adoption. Without this fine-grained diagnostic—which aggregate benchmarks cannot provide—organizations would either adopt updates blindly (risking undetected regressions) or avoid updates entirely (missing genuine improvements). The paper's validation of the verification framework (87.2% GUI agent node-level agreement, 0.80 median Spearman for the VLM judge) provides sufficient reliability for internal monitoring, and the quarterly evaluator update plan ensures the evaluation framework itself remains current as underlying VLMs improve.

Curriculum design for training coding agents through staged capability acquisition. The paper's finding that capability degrades systematically as task complexity increases—and that the degradation pattern is differential across models (Gemini-3-Pro-Preview excels at static pages but collapses on full-stack, while Claude-Opus-4.5 degrades more gracefully)—suggests a training curriculum organized around Vision2Web's hierarchical levels. An agent trained first on Level 1 tasks (pure visual reproduction, no interactivity) until achieving VS > 70, then on Level 2 tasks (adding multi-page navigation and interaction logic) until achieving VS/FS > 60/70, then finally on Level 3 tasks (adding backend integration and state management), would follow a scaffolded curriculum where each stage builds on mastered skills from the previous stage. This is analogous to how human developers learn: master HTML/CSS layout before adding JavaScript interactivity, master frontend development before adding backend integration. The paper's category-level diagnostics (Table 5) further specify which skills to emphasize at each stage—Level 2 training should oversample tasks requiring State Management and Form Interaction (the categories where agents struggle most in the transition from static to interactive), while Level 3 training should emphasize CRUD Operations and File & Media Operations. This is a concrete training methodology suggested directly by the benchmark's diagnostic structure, not a generic "curriculum learning" proposal.