Code Aesthetics with Agentic Reward Feedback

🎯 Pitch

A 4B parameter model can beat GPT-4.1 at generating visually appealing webpages and plots—not by scaling size, but by giving the model “eyes.” This paper trains tiny coding models with a multi-agent reward system that looks at screenshots, clicks through rendered interfaces, and scores aesthetics, achieving results that match models 100× larger.

1. Executive Summary

This paper introduces agentic reward feedback, a multi-agent reward system for improving the aesthetic quality of LLM-generated code in visually-oriented tasks such as webpage design and plot generation. Using Qwen3-4B and Qwen2.5-Coder-7B as base models, the authors construct AesCode-358K, a 358K-instance supervised fine-tuning dataset, then apply GRPO-AR, which integrates feedback from three specialized agents—an execution agent (verifying basic HTML syntax), a static aesthetics agent (scoring full-page screenshots via GPT-5 on alignment, visual design, and structural coherence), and an interactive aesthetics agent (autonomously navigating rendered webpages to assess usability)—into the GRPO reinforcement learning algorithm for joint optimization of functionality and aesthetics. On the authors' newly introduced OpenDesign benchmark (840 webpage design cases) and the existing PandasPlotBench, AesCoder-4B surpasses GPT-4o and GPT-4.1 while matching open-source models with 480B–685B parameters, establishing that multi-agent reward feedback can align small models with human-perceived code aesthetics only when tasks fall within the base model's capability range—hard problems benefiting from interactive evaluation, easy problems from refined visual scoring.

2. Context and Motivation

The fundamental problem this paper addresses is deceptively simple: large language models trained to write code have no awareness of what their code actually looks like when rendered. When an LLM generates Python for a matplotlib chart or HTML/CSS for a webpage, it operates purely in the textual modality — it sees tokens, not pixels. This creates a glaring capability gap in visually-oriented coding tasks, which the paper defines as programming tasks "in which the correctness or quality of the code is inherently tied to its visual output" (Section 1).

The consequence, as the authors observe, is that even capable coding models produce "poor visual outcomes like overlapping elements, inconsistent color schemes, and disorganized structures" (Section 1). The model might generate syntactically valid HTML that renders into something ugly, or Python plotting code that produces a chart with a legend occluding the data. The model simply doesn't know, because its training signal — typically next-token prediction on text — never taught it to care about visual aesthetics.

This gap matters because visually-oriented coding is not a niche concern. Real-world software development involves substantial UI work. Data scientists spend significant time tuning plot aesthetics. Web developers iterate on layout and design. If LLMs are to serve as genuine coding assistants across the full spectrum of programming tasks, they need some mechanism for understanding and optimizing the visual consequences of their code.

Why This Problem Is Both Practical and Underappreciated

The paper's framing of this as a code aesthetics problem fills a genuine void in the landscape of LLM coding capabilities. Prior work on coding LLMs has focused overwhelmingly on what might be called textual correctness: does the code compile? Does it pass unit tests? Does it produce the right algorithmic output? Benchmarks like HumanEval, MBPP, and SWE-bench all evaluate code as text-in, text-out. Even recent work on reinforcement learning for code (RLTF, RLEF, CodeRL) defines rewards exclusively in terms of executability and functional correctness — did the code run without errors, did it produce the expected output?

The authors argue that this singular focus on function over form creates a systematic blind spot (Section 1):

"reward methods for training coding LLMs often focus on a single textual modality, such as code executability and result correctness. These methods have significant limitations when applied to code aesthetics tasks, as they fail to assess visual aesthetics and are unable to interact with rendered visual interfaces like webpages, making them ineffective as reward sources."

This is not just an aesthetic preference — it's a capability gap with direct practical consequences. A data analyst who asks an LLM to "create a clear bar chart comparing quarterly revenue" receives code that might run correctly but produce an unreadable chart. A developer who asks for a "landing page for a SaaS product" might get functional HTML that looks amateurish. In both cases, the model has failed at the actual task, even though it passed all traditional correctness checks.

The paper also identifies a more subtle dimension: interactive aesthetics. A static screenshot of a webpage might look beautiful, but if the navigation buttons don't work or a 3D visualization can't be rotated, the design has failed. The authors note that existing evaluation methods "rely on evaluating static image(s) and may not capable to assess contents like webpages which need interactions" (Section 2). This is particularly acute for browser-based games, interactive dashboards, and 3D design tools — all domains where static visual quality and interactive functionality must co-exist.

Where Prior Approaches Fall Short

The paper identifies specific limitations in prior work along several axes:

Single-modality reward signals for code. The dominant approaches for aligning coding LLMs — RLTF (Reinforcement Learning from Unit Test Feedback), RLEF (RL with Execution Feedback), and CodeRL — all define reward as a function of whether code executes correctly or passes tests. These are text-in, text-out signals. They cannot tell you whether a matplotlib plot looks good, whether a webpage has proper visual hierarchy, or whether interactive elements respond correctly to user input. This is not a minor oversight; it's a fundamental mismatch between the reward signal and the task objective.

Static image aesthetics research doesn't transfer. The paper acknowledges prior work on assessing and improving the aesthetic quality of AI-generated images (Section 2). However, these methods are fundamentally designed for static content evaluation — they take an image and produce a quality score. They cannot evaluate interactive elements, cannot verify that a user instruction was followed, and cannot navigate a rendered webpage to test functionality. The paper explicitly notes this limitation:

"all these methods rely on evaluating static image(s) and may not capable to assess contents like webpages which need interactions."

LLM-as-a-judge for code misses visual feedback. The use of LLMs to evaluate LLM outputs (LLM-as-a-judge) has become common for text quality assessment, but when applied to code, these judges typically see only the source code — not its rendered output. They cannot visually inspect a webpage or chart, so they are forced to guess at aesthetics from raw HTML or Python, which is a fundamentally impoverished signal.

No datasets or benchmarks for code aesthetics. Critically, prior to this work, there was no dedicated dataset for training models on code aesthetics, and no benchmark for evaluating webpage design quality in an automated, scalable way. The Design Arena platform exists as a human-powered judging platform for webpage design (Section 6), but its community voting process is "time-consuming and impractical for large-scale evaluation." This means no prior work could systematically train or evaluate models for code aesthetics — the infrastructure simply didn't exist.

How This Paper Positions Itself

The paper's contribution is best understood as building an entire pipeline — data, training methodology, and evaluation — where none existed before. It does not propose incremental improvements to existing code-reward methods; it argues that those methods are fundamentally insufficient and proposes a wholly different reward paradigm.

The key conceptual move is the introduction of agentic reward feedback (Section 4). Rather than trying to squeeze visual evaluation into a text-based reward model, the paper decomposes the evaluation into three specialized agents that operate in the modalities where the relevant information actually lives: the execution agent checks code syntax (text), the static aesthetics agent evaluates screenshots (vision), and the interactive aesthetics agent navigates rendered webpages (interaction). This multi-agent decomposition is the paper's answer to the observation that "in complex tasks, relying solely on a single source of reward can induce biased behaviors, ultimately driving optimization in an incorrect direction" (Section 2).

The paper explicitly connects to the growing maturity of AI agents, noting that "as the growing maturity of AI agents, it becomes possible to integrate interactive evaluation into the contents generated by large language models, thereby providing more comprehensive and systematic feedback" (Section 2). This positions the work at the intersection of two trends: the increasing capability of coding LLMs, and the emergence of multimodal agents (like WebVoyager) that can autonomously interact with web interfaces. The agentic reward framework is essentially an attempt to use the latter to train the former — agents evaluating and providing feedback on LLM-generated code by actually rendering and interacting with it.

The paper also positions itself in the lineage of RLHF (Reinforcement Learning from Human Feedback) and its variants (DPO, RFT), but argues that for visually-oriented coding, human preference data — even if collected — would be insufficient because human evaluators cannot efficiently evaluate thousands of webpages for both static aesthetics and interactive functionality. The agentic reward framework is proposed as an automated, scalable alternative that captures the multi-dimensional nature of code aesthetics (executability, visual quality, interactivity) without requiring human annotation at scale.

The introduction of AesCode-358K and the OpenDesign benchmark (Sections 3 and 6) fills the data and evaluation gaps explicitly. The dataset provides the first large-scale supervised training signal for code aesthetics across both plot generation (158K instances) and webpage design (200K instances). The benchmark provides the first automated, LLM-judge-based evaluation protocol for webpage design that correlates strongly with human preferences (Spearman = 0.98 with Design Arena rankings, Section 6.2). Together, they establish the infrastructure for a new subfield — code aesthetics — that the paper is effectively inaugurating.

Finally, the paper positions the GRPO-AR training methodology (Section 5.2) as a natural fit for the problem: GRPO's group-relative advantage computation (comparing outputs within a batch rather than against an absolute reward threshold) is well-suited to aesthetic evaluation, where absolute scores are less meaningful than relative quality judgments. By combining GRPO with the multi-agent reward framework, the paper creates a training pipeline that jointly optimizes for functionality, visual quality, and interactivity — three objectives that prior work treated as separable or simply ignored the latter two.

3. Technical Approach

3.1 Reader Orientation

The system being built is a training pipeline that gives small language models (4B and 7B parameters) a "sense of aesthetics" — the ability to generate visually appealing, interactive webpages and plots, not just functionally correct but ugly ones. The core problem is that LLMs are trained exclusively on text, so they have no mechanism for understanding what their code looks like when rendered; this pipeline solves that by introducing agentic reward feedback: during reinforcement learning, three specialized AI agents — one that checks if the code runs, one that visually inspects a screenshot of the rendered output, and one that actually clicks around the rendered webpage to test interactivity — collectively provide a rich, multi-dimensional reward signal that teaches the model to care about both how its code looks and how it behaves.

3.2 Big-Picture Architecture (Diagram in Words)

The AesCoder pipeline (Figure 2) has five major components connected in a two-stage training flow:

AesCode-358K Dataset Construction (Section 3): A 358K-instance supervised fine-tuning dataset built from two sources — 158K Python plotting examples regenerated and validated from VisCode-200K, and 200K webpage design examples generated via a four-step GPT-driven pipeline with quality filtering. This is the "what good code aesthetics look like" training data.
Stage I: Supervised Fine-Tuning (SFT) (Section 5.1): The base models (Qwen3-4B-Instruct-2507 or Qwen2.5-Coder-7B-Instruct) are fine-tuned on AesCode-358K for 3 epochs using standard next-token prediction. This gives the models initial knowledge of aesthetic code patterns but limited generalization.
Agentic Reward Framework (Section 4): A multi-agent evaluation system that takes raw model-generated HTML code and produces a composite reward score. It contains three agents running in sequence-then-parallel:
- Execution Agent: Extracts HTML from the model output, checks basic syntax rules via HTMLHint, returns $s_{\text{exec}} = 1$ (pass) or $s_{\text{exec}} = -1$ (fail).
- Static Aesthetics Agent: Renders the HTML in headless Playwright, captures a full-page screenshot, sends it to GPT-5 as a multimodal judge, receives scores on three dimensions (instructional alignment, visual elements, layout/cohesion), and returns the total static score $r_{\text{static}}$ .
- Interactive Aesthetics Agent: Launches the webpage in a browser environment, uses a WebVoyager-based GUI agent (powered by GPT-4o) to plan and execute a sequence of interactions (clicking buttons, typing in textboxes, navigating), records binary success/failure for each interaction, and returns the sum as $r_{\text{interact}}$ .
Stage II: GRPO-AR Training (Section 5.2): The SFT model undergoes reinforcement learning using the GRPO algorithm, where the reward for each generated output comes from the Agentic Reward Framework. The agentic reward is a weighted sum: $r = w_{\text{exec}} \cdot r_{\text{exec}} + w_{\text{static}} \cdot r_{\text{static}} + w_{\text{interact}} \cdot r_{\text{interact}}$ , with weights $w_{\text{exec}} = 0.1$ , $w_{\text{static}} = 0.8$ , $w_{\text{interact}} = 0.1$ . GRPO computes advantages by normalizing rewards within each batch of $G = 8$ samples per prompt and optimizes a clipped policy gradient objective.
OpenDesign Benchmark (Section 6): An 840-case evaluation benchmark spanning five webpage categories (General Website, 3D Design, Data Visualization, Game Dev, UI Component) that uses the same static and interactive aesthetics agents to score model outputs, enabling automated, scalable evaluation that correlates strongly with human preferences (Spearman = 0.98 with Design Arena rankings).

Information flows as follows: a user instruction enters → the model (either during SFT data generation or during RL training) generates HTML code → the Execution Agent checks basic validity → if valid, the Static Aesthetics Agent takes a screenshot and GPT-5 scores it, while in parallel the Interactive Aesthetics Agent launches a browser and tests interactions → the three scores are weighted and summed into a single reward → this reward feeds into the GRPO optimizer to update the model → at evaluation time, the same agentic scoring (without the execution penalty, scored on all cases) produces the OpenDesign benchmark scores.

3.3 Roadmap for the Deep Dive

First, the AesCode-358K dataset construction — how the Python plotting and webpage design data are generated, validated, and filtered, since this is the supervised foundation all subsequent training builds on.
Second, the Execution Agent — its rules, its binary pass/fail mechanism, and why strict execution checking is inappropriate for HTML.
Third, the Static Aesthetics Agent — what dimensions it evaluates, how GPT-5 is prompted as a multimodal judge, and how scores are aggregated.
Fourth, the Interactive Aesthetics Agent — the WebVoyager-based architecture, the planning-and-execution loop, the binary scoring per interaction, and the known limitations.
Fifth, the GRPO-AR training algorithm — the formal objective with the clipped advantage, the KL penalty, and how agentic rewards are integrated into the GRPO batch-relative computation.
Sixth, the reward aggregation formula and weight selection rationale — since the relative weighting of executability, static, and interactive aesthetics fundamentally shapes what the model learns.

I explain these in this order because the agents are the novel contribution (the "agentic reward feedback") and must be understood before the training loop that consumes their output. The dataset construction comes first because it generates the SFT checkpoint that serves as the starting policy for RL.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that multi-agent evaluation of rendered code outputs — spanning executability, visual appearance, and interactivity — can serve as a reinforcement learning reward signal that teaches language models to generate aesthetically superior code, even when the base model has no native visual understanding.

AesCode-358K Dataset Construction: Python Plot Data

The Python plotting portion of AesCode-358K is derived from the existing VisCode-200K dataset (Section 3.1), a collection of 200K instruction-code pairs for visualization tasks. However, the authors found that "some of the Python code snippets are either not executable or exhibited sub-optimal aesthetics, such as chart–legend overlap and improper font sizes" (Section 3.1). The core problem is that VisCode-200K prioritized having some code for each instruction, not necessarily good code with clean visual output.

Regeneration with Qwen3-Coder-480B. To upgrade quality, the authors use Qwen3-Coder-480B-A35B-Instruct-FP8 to regenerate the Python code for each instruction. This is a much larger, more capable coding model (480B total parameters, with 35B active via mixture-of-experts) specifically designed for code generation tasks. The intuition is straightforward: a stronger model produces better code, and better code produces better visual output, so regenerating the dataset with a stronger model inherently improves the aesthetic quality of the training data.

The choice of Qwen3-Coder over alternatives (GPT-4o, Claude, DeepSeek-Coder) is not explicitly justified beyond Qwen3-Coder-480B being the model used. However, the pattern is consistent with the paper's overall philosophy: use the strongest available models as generators and judges for training data, then distill that capability into smaller models through fine-tuning and RL.

Quality control via environment restriction and runtime validation. Two mechanisms ensure the regenerated code is actually good:

First, environment restriction: the Python environment is limited to essential visualization libraries — matplotlib, seaborn, and plotly. This prevents the model from importing obscure or deprecated libraries that might cause execution failures or produce undesirable output. The restriction is practical: these three libraries cover the vast majority of real-world plotting use cases, and limiting the search space makes the generation task more tractable.

Second, runtime validation: each regenerated code snippet is executed in a Jupyter Notebook runtime. If the code fails to run without errors and produce the correct visualization, it is discarded. This is a hard filter — only code that actually executes and renders is kept. The paper does not specify the exact pass rate of this filtering step, but the final yield is 158K high-quality plot data points from the original 200K, implying a roughly 79% survival rate (though some original data points may have been discarded for other reasons).

The runtime check is crucial because it catches a whole class of aesthetic failures that are invisible from source code alone: a matplotlib script might reference a nonexistent column name, produce an empty plot due to incorrect data filtering, or generate a legend that completely covers the data. By requiring actual execution, the dataset construction implicitly encodes that "aesthetically good code" is a subset of "executable code."

AesCode-358K Dataset Construction: Webpage Design Data

The webpage design portion of AesCode-358K is built from scratch through a four-step process (Section 3.2) that generates diverse, high-quality HTML/CSS examples across five categories.

Step 1: Seed keyword generation. The authors define five webpage categories: General Website, 3D Design, Data Visualization, Game Dev, and UI Component. Using GPT-4o, they generate a seed keyword corpus: 9,000 keywords for General Website and 2,500 keywords for each of the remaining four categories, totaling 19,000 seed keywords (Table 3).

The category breakdown is deliberate. General Website is the largest category because it covers the most common use cases (landing pages, portfolios, e-commerce, blogs) and requires the most diversity in keywords to cover the space. The four specialized categories each get 2,500 keywords because they represent important but narrower domains where the model needs sufficient coverage without over-representing edge cases. The choice of these specific categories — 3D Design, Data Visualization, Game Dev, UI Component — maps directly to the paper's concept of interactive aesthetics: these are domains where static visual quality is insufficient and interactive functionality is central to the user experience.

Step 2: Instruction generation from keywords. For each seed keyword, GPT-4o generates 20 "non-redundant and semantically diverse instructions" (Appendix B.1). With 19,000 keywords × 20 instructions each, this produces approximately 380,000 raw webpage design instructions. Each instruction is a natural language description of what a webpage should do (e.g., "Create a pizza restaurant website with a focus on the menu and online ordering" from Figure 2), not technical specifications.

The instruction generation prompt (Appendix H.5) explicitly instructs GPT-4o to "ensure summaries are specific, diverse, and avoid repetition in functionality, theme, or wording" and to "explore unique and imaginative themes, avoiding overlap with common website concepts." This deliberate diversity push is important because if the instructions were homogeneous (e.g., 50 variations on "create a blog homepage"), the model would overfit to a narrow distribution and fail on novel requests during RL training.

Step 3: Semantic deduplication. Even with diversity instructions, large-scale GPT generation produces redundancy. To address this, the authors embed all 380K instructions using openai-text-embedding-3-large (3,072-dimensional embeddings), then apply K-Means clustering with $K = 200,000$ clusters. From each cluster, only the instruction nearest to the cluster centroid is retained, yielding 200K deduplicated instructions.

The choice of embedding model (text-embedding-3-large) over alternatives (e.g., sentence-transformers, BGE, E5) is not justified, but OpenAI embeddings are a reasonable default for semantic similarity tasks. The 3,072-dimensional space provides sufficient capacity to capture fine-grained semantic differences between instructions. The K-Means approach with K = 200K is computationally heavy (clustering 380K points in 3,072 dimensions) but has the advantage of being density-aware: dense regions of the instruction space (many similar instructions) get collapsed to a single representative, while sparse regions (unique instructions) are preserved at roughly their natural frequency, since each gets its own cluster.

The t-SNE visualization in Figure 5 provides qualitative evidence that the deduplication works: the raw data shows "significant overlaps across categories, along with several dense clusters," while the filtered data shows "clearer class boundaries and reduced overlap across categories."

Step 4: HTML code generation and quality filtering. For each of the 200K instructions, the authors generate HTML code using GPT-5 and Qwen3-Coder-480B-A35B-Instruct-FP8. The paper does not specify whether one model generates all 200K or they are split, but the quality filtering step clarifies: for each instruction, both models (or both generation runs) produce HTML, and the one with the higher quality score (as judged by GPT-5 based on rendered screenshots) is kept.

This dual-generation with selection is a form of rejection sampling at the data construction stage. The intuition is that even strong models produce variable-quality HTML for the same instruction, and by generating multiple candidates and keeping the best, the dataset's average quality improves, which in turn improves the SFT model trained on it.

The quality check involves: (1) confirming the HTML is executable (renders without crashing), (2) rendering both candidates via Playwright and Selenium, and (3) having GPT-5 score both rendered images and select the higher-scoring one. The paper does not report the exact scoring prompts for this step, but it is conceptually the same static aesthetics evaluation later used in the agentic reward framework.

Final dataset composition. The complete AesCode-358K dataset contains 158K Python plotting examples + 200K webpage design examples = 358K total instruction-code pairs. The name "AesCode-358K" reflects this: Aesthetics Code, 358,000 instances. The dataset covers two distinct but related domains (plotting and web design), both of which require reasoning about visual output, but through different code modalities (Python with visualization libraries vs. HTML/CSS).

The Execution Agent

The Execution Agent (Section 4.1) is the first and simplest of the three reward agents. Its sole responsibility is to verify that the model's output is syntactically valid HTML that can be rendered by a browser. It operates entirely in the text modality — no screenshots, no interaction.

Output extraction. When the model generates a response, it may wrap the HTML in markdown code fences (`````html ... ```), or it may output raw HTML directly. The Execution Agent first attempts to extract HTML from within html code blocks; if none are found, it treats the entire model output as HTML. This handles the common case where instruction-tuned models format their code outputs in markdown.

Rule-based validation. Unlike Python code, where "executability" means running the interpreter and checking for exceptions, HTML executability is fuzzier. Browsers are famously forgiving — they render malformed HTML, close unclosed tags, and generally try to make something displayable from whatever they receive. Strict execution checking (trying to parse and execute the HTML) would therefore be too lenient: almost anything would "pass."

Instead, the Execution Agent uses HTMLHint, a static linting tool, to apply a defined set of validation rules (listed in Appendix H.7). The rules check for:

HTML5 doctype declaration (doctype-html5: true)
Lowercase tag names (tagname-lowercase: true)
Lowercase attribute names (attr-lowercase: true)
Double-quoted attribute values (attr-value-double-quotes: true)
Proper tag pairing (every opening tag has a closing tag, tag-pair: true)
Self-closing tags for void elements (tag-self-close for br, img, input, link, meta)
Unique id attributes (id-unique: true)
Required alt attributes on all <img> tags (alt-require: true)
Required lang attribute on <html> tag (lang-require: true)
UTF-8 charset declaration (meta-charset-utf-8: true)
Viewport meta tag (meta-viewport: true)
Required <title> tag (title-require: true)

Additionally, some rules are explicitly disabled to avoid penalizing legitimate patterns:

head-script-disabled: false — scripts are allowed in <head>
style-disabled: false — inline CSS is allowed
no-inline-style: false — inline styles within HTML elements are allowed
no-inline-script: false — inline JavaScript is allowed
csslint.important: false — !important in CSS is allowed
csslint.order-alphabetical: false — CSS property ordering is not enforced
script-disabled: false — JavaScript usage is allowed

The rationale for these exceptions is that webpage design tasks, especially single-file HTML pages with embedded CSS and JavaScript, frequently use inline styles, inline scripts, and !important declarations. Enforcing strict separation of concerns (external stylesheets, no inline JS) would penalize valid design patterns that happen to be implemented in a single file — which is exactly the format the model is trained to generate.

Binary scoring. The agent returns $s_{\text{exec}} = 1$ if the HTML passes all validation rules, and $s_{\text{exec}} = -1$ if it fails any rule. The choice of $-1$ rather than $0$ for failure is deliberate: it creates a sharper penalty gradient. A model that generates invalid HTML receives a negative reward, actively discouraging that behavior, rather than merely receiving zero reward (which would treat invalid HTML identically to valid but ugly HTML).

The paper does not specify whether partial credit is possible (e.g., passing 9 out of 10 rules), but the description "assigns $s_{\text{exec}} = 1$ if the output passes all validations, and $s_{\text{exec}} = -1$ otherwise" (Section 4.1) implies a hard pass/fail. This is a simplification — a page that fails only one minor rule (say, missing alt text on one image) is treated identically to one that is completely unparseable — but it makes the reward signal binary, which is both simple to implement and creates a clear "feasibility boundary" that the model must cross before static and interactive aesthetics can even be evaluated.

Why not stricter checking? The paper explicitly notes that "given that web browsers tolerate many structural and syntactic errors, strict execution checking is unsuitable for HTML" (Section 4.1). This is a key design insight: if they used a full HTML5 parser and rejected anything that wasn't perfectly compliant, they would reject many pages that render perfectly fine in practice. The HTMLHint rule set strikes a balance — it catches genuinely problematic patterns (unclosed tags, missing doctype, duplicate IDs) while tolerating patterns that are technically violations but functionally harmless (inline styles, scripts in head).

The Static Aesthetics Agent

The Static Aesthetics Agent (Section 4.2) is the primary source of visual quality feedback. It takes the validated HTML, renders it in a browser, captures a full-page screenshot, and has a multimodal LLM (GPT-5) evaluate the screenshot against the original user instruction.

Rendering pipeline. For a given HTML file, the agent uses Playwright in headless mode (no visible browser window) to launch a Chromium instance, load the HTML, and capture a full-page screenshot. Playwright is chosen over alternatives (Selenium, Puppeteer) because it provides reliable headless rendering with good CSS and JavaScript support, and it can capture full-page screenshots that include content below the fold.

The headless rendering ensures the evaluation is deterministic (no display-dependent rendering variations) and scalable (no need for GPU-accelerated displays). However, it introduces a potential gap: some CSS features (animations, certain @media queries, font rendering) may behave differently in headless vs. headed mode, and the screenshot might not perfectly represent what a human user would see.

Evaluation dimensions. The GPT-5 judge evaluates each screenshot on three dimensions, each with a specified point allocation (detailed in Appendix H.2):

Instructional Alignment (40 points): "Evaluates consistency between the page's style and user instructions." The judge checks whether the webpage actually implements what was requested — are all specified elements present, does the content match the described purpose, is the overall style appropriate for the stated topic? This is the "did you build the right thing?" dimension.
Aesthetics and Readability (30 points): "Assesses the effective use of modern design features such as lighting, transparency, and gradients." The judge evaluates visual appeal — color scheme, typography, whitespace, visual hierarchy, use of modern design elements. This is the "does it look good?" dimension.
Layout and Cohesion (30 points): "Examines whether the structure is functional, responsive, and visually coherent, with concise yet design-aware typography." The judge evaluates structural organization — logical flow, component arrangement, overall coherence. This is the "is it well-organized?" dimension.

The point allocation (40/30/30) weights instructional alignment most heavily, reflecting the paper's philosophy that a webpage should first and foremost satisfy the user's request. Within the 100-point total, aesthetics and structural cohesion are equally weighted at 30 points each, suggesting the authors view them as comparably important.

GPT-5 as judge with chain-of-thought. The paper selects GPT-5 as the judge "for its strong multimodal reasoning ability" (Section 4.2). The judge is prompted to use a chain-of-thought approach — it must provide both a score and a rationale for each dimension. The prompt (Appendix H.2) explicitly instructs:

"Use the full range for each criterion (e.g., 0–40, 0–30). Average or flawed webpages should receive average or below-average scores. High scores (top 20% of each range) should be awarded only for work that meets or exceeds professional standards with virtually no flaws."

The chain-of-thought requirement serves multiple purposes. First, it improves scoring reliability: requiring the model to articulate its reasoning before assigning a score reduces impulsive or inconsistent judgments, a well-documented benefit of chain-of-thought prompting. Second, the rationales themselves are retained (though not used for training), providing interpretability — a human can inspect why a particular webpage received a low score. Third, the requirement to justify scores creates pressure toward score calibration: if the model claims a webpage is "outstanding, nearly perfect" but then notes several flaws in the rationale, there's an internal contradiction.

Score retention. The paper notes that "while both scores and explanations are required to ensure reliable evaluation, we retain only the final aggregated score as the output of the static aesthetics agent" (Section 4.2). The rationales are discarded after scoring — they serve only to improve the quality of the scores themselves, not as training data. This is a pragmatic choice: the rationales are text explanations that don't easily convert to scalar rewards, and the downstream GRPO training only needs a scalar.

Score interpretation reference. The prompt provides GPT-5 with explicit score interpretation guidelines (Appendix H.2):

90–100: "Outstanding, professional, nearly perfect."
70–89: "Good but with noticeable issues or minor flaws."
50–69: "Average, with clear limitations or several weaknesses."
30–49: "Below average, significant flaws or missing requirements."
0–29: "Poor, major requirements missing, very low quality."

This is effectively a rubric that anchors the numeric scores to qualitative descriptions, improving consistency across evaluations. Without such anchoring, different invocations of GPT-5 might assign very different scores to the same webpage based on implicit standard shifts.

Why GPT-5 and not a smaller/cheaper model? The paper does not explicitly justify GPT-5 over alternatives like GPT-4o or Claude for the static judge role, but two factors are implied. First, GPT-5 has the strongest multimodal reasoning capability available at the time of writing, and the static aesthetics evaluation — judging visual design quality against a natural language instruction — is a genuinely hard multimodal reasoning task that benefits from the strongest available model. Second, the static aesthetics agent's scores carry the highest weight in the reward aggregation (0.8), making its accuracy disproportionately important. A weaker judge producing noisy scores would inject noise directly into the training signal.

The one-image limitation. The static aesthetics agent evaluates a single full-page screenshot. This means it cannot assess:

Responsive behavior (how the page looks at different screen widths)
Animations or transitions (they're frozen in a screenshot)
Hover states (the cursor isn't visible)
Scroll behavior (smooth scrolling, parallax effects)

These limitations are inherent to screenshot-based evaluation and are partially addressed by the interactive aesthetics agent (which can test some of these). However, the paper does not discuss these limitations explicitly — it treats the static screenshot as a sufficient proxy for visual quality, which is a reasonable assumption for most webpage designs where the "first impression" from a full-page view dominates aesthetic judgment.

The Interactive Aesthetics Agent

The Interactive Aesthetics Agent (Section 4.3) is the most technically complex component of the reward framework. It addresses the fundamental limitation of static screenshot evaluation: a webpage can look beautiful but be non-functional — buttons that don't click, forms that don't submit, 3D scenes that can't be rotated, game controls that don't respond. The interactive agent autonomously navigates, interacts with, and evaluates the behavior of the rendered webpage, producing a score based on whether interactions succeed or fail.

Why interactivity evaluation is necessary. The paper provides a concrete motivation:

"Evaluation based only on static screenshots is insufficient, as it overemphasizes visual appearance while neglecting usability. This issue is particularly critical for interactive webpages such as 3D design platforms or browser-based games." (Section 4.3)

This captures a real alignment concern: if the reward signal only cares about static appearance, the model may learn to produce webpages that look beautiful in screenshots but are non-functional. For example, a "3D design platform" might render a gorgeous static scene but have broken rotation controls, or a "browser-based game" might have polished graphics but unresponsive game mechanics. The interactive agent provides a countervailing pressure: it rewards functionality, ensuring the model doesn't sacrifice interactivity for static aesthetics.

Architecture and model choice. The agent is built on WebVoyager, an existing framework for end-to-end web agents that can perceive webpages (via screenshots with labeled elements), plan actions, and execute them. The paper chooses GPT-4o as the multimodal model powering the agent "for cost considerations" (Section 4.3) — GPT-4o is cheaper than GPT-5 and sufficient for the action-planning and success/failure judgment tasks, which are less demanding than the holistic aesthetic evaluation performed by the static agent.

The agent operates on a screenshot-and-label paradigm: each webpage observation includes a screenshot with numerical labels overlaid on interactive elements in the top-left corner. The agent sees these labeled screenshots and issues actions by referencing numerical labels (e.g., "Click [3]" to click the element labeled 3, "Type [5]; hello" to type "hello" into the textbox labeled 5). This avoids the need for the agent to specify pixel coordinates or CSS selectors, which would be brittle and model-dependent.

The planning-and-execution loop. The agent's operation proceeds through well-defined phases (detailed in Appendix H.3):

Phase 1: Planning. At the start of evaluation, the agent receives the user instruction, the webpage category, and an initial screenshot. It must "think thoroughly about all the ways of interactions with the webpage based on the topic, the detailed description given by the user and the webpage screenshot" and output a planned list of interactions (Appendix H.3). The planning is structured to be instruction-aware: the agent doesn't just randomly click elements; it identifies which interactions are most relevant to the stated purpose of the webpage.

The prompt instructs the agent to "rank these candidates and selects the top N for execution," and during training, the number of interactive elements is limited to 3 (Section E). This constraint serves two purposes: it keeps the evaluation tractable (a full exploration of all possible interactions would be expensive), and it forces prioritization — the agent must identify and test the most critical interactions.

Phase 2: Interaction and scoring. For each planned interaction, the agent executes the action in the browser, observes the result (a new screenshot), and judges whether the webpage responded correctly. The scoring is binary:

1: The interaction succeeded — the webpage changed as expected (e.g., clicking a navigation button took you to a new section, typing in a search box and clicking search filtered results, pressing an arrow key moved the game character).
0: The interaction failed — the webpage didn't change, changed in an unexpected way, or produced incorrect behavior.

For interactions with multi-step validation requirements, the scoring is nuanced. For textbox interactions specifically:

"for textbox, you should plan both typing in the textbox and clicking the search button. It cannot be considered as a successful interaction if only you successfully type in the textbox, but the webpage has not changed at all after clicking the search button." (Appendix H.3)

This means the agent uses a special scoring protocol for text inputs: it outputs NaN (not a number, indicating "no score yet") after typing, and only assigns the actual score (0 or 1) after clicking the associated action button (search, submit, etc.). This prevents the model from getting credit for merely accepting text input without producing a meaningful response.

Phase 3: Score aggregation. After all planned interactions are completed (or the maximum iterations are reached), the agent outputs a binary score list $[s_1, s_2, ..., s_N]$ where each $s_i \in \{0, 1\}$ , and aggregates them into a final interactive score:

$s_{\text{interact}} = \sum_{i=1}^{N} s_i$

where $N$ is the number of planned interactions (limited to 3 during training, variable during evaluation), and each $s_i$ is the binary success/failure judgment for interaction $i$ .

What it computes: a simple count of successful interactions. If the agent planned to test 3 features and all 3 worked correctly, the score is 3. If only 1 worked, the score is 1. If nothing worked, the score is 0.

Why this form: a sum of binary indicators is the simplest possible metric for interactive functionality — it counts "things that work." An alternative would be a continuous quality score (e.g., GPT-5 judging each interaction on a 0–10 scale), but binary scoring has several advantages: it's more objective (did it work or not? is clearer than "how well did it work?"), it's less susceptible to judge miscalibration, and it directly penalizes broken functionality. The sum aggregation means the maximum possible score equals the number of tested interactions, which varies by webpage. This makes $s_{\text{interact}}$ an absolute count, not a normalized rate, which could be problematic if different webpages have different numbers of interactable elements — the paper addresses this only by noting the score is aggregated via the weighted sum with other rewards, not used in isolation.

Action constraints and environment limitations. The agent operates under specific constraints that reflect practical and methodological considerations:

Offline-only interactions. The agent is explicitly instructed to focus only on local, offline functionality: "Interactions requiring internet access (e.g., social media logins) are excluded, focusing only on the core webpage functionality" (Section 4.3). This is both practical (network-dependent features would introduce non-determinism and potential failures unrelated to code quality) and philosophical (the core webpage functionality should work without external dependencies).

Action repertoire. The agent has access to a limited set of actions (Appendix H.3):

Click a Web Element (by numerical label)
Delete and type content in a textbox (by numerical label)
Wait (1 second, for unfinished page processes)
Arrow key presses (UP, DOWN, LEFT, RIGHT — only for Game Dev category)
FINISH (end evaluation)

For Game Dev webpages specifically, the agent is instructed that "it may not have many interactable elements to click. Instead, you can use the up, down, left, right arrow keys to control the game, and plan dynamically when the game running" (Appendix H.3). This category-specific behavior is crucial: a game might have no visually labeled buttons but still be fully functional through keyboard controls, and the agent needs to test those.

Scroll handling. The agent is explicitly told not to attempt scrolling: "If you have seen a scrollbar in the webpage (not for the whole window, since the webpage is always single-paged, but for a certain area or element of the webpage, such as a 3D object to be rotated or zoomed), do not directly try to scroll it. Instead, find if any interactable element such as button '-' or '+' and click the button instead" (Appendix H.3). This avoids a class of interactions (scrolling) that are notoriously difficult for web agents to execute reliably and instead routes the agent toward explicit UI controls.

Pop-up handling. If an interaction opens a pop-up window, the agent is instructed to "close the pop-up window and return to the original webpage after you have finished evaluating the interaction" (Appendix H.3). This prevents the agent from getting stuck in modal states.

Limitations and their treatment. The paper is unusually candid about the interactive agent's limitations (Section 4.3, "Discussions"):

"Current web agents can handle most webpage operations, but may still struggle with certain corner cases, such as confusing webpage elements or being misled by irrelevant textual content. Such agent failures lead to a score of 0 in the corresponding iteration, since we assign a score of 1 only when the webpage responds correctly."

This creates an under-scoring bias: if the agent fails to interact correctly due to its own limitations (not the webpage's fault), the webpage receives a score of 0 for that interaction, even though a human user would have succeeded. The paper acknowledges this directly: "This may cause the agent to make incorrect judgments, resulting in scores lower than the true values."

However, the paper offers a partial defense: "agent failures also partially reveal non-standard or sub-optimal aspects of webpage design." The reasoning is that if an AI agent — designed to be robust — cannot successfully interact with a webpage, that may indicate that the webpage's interaction design is genuinely non-standard or confusing, even if a determined human could figure it out. This reframes the limitation not as pure noise but as a signal about interaction quality, albeit a noisy one.

Mitigation through prioritization. During training, to "mitigate the impact of the GUI agent's limited success rate on our GRPO-AR training" (Section E), the agent is instructed to "prioritize [interactive elements] based on their importance." By testing the most critical interactions first, even if the agent fails on some edge cases, the core functionality gets evaluated. The paper also limits interactions to 3 during training to keep the evaluation focused and reduce the chance of agent failures accumulating.

The GRPO-AR Training Algorithm

GRPO-AR (Section 5.2) is the combination of the GRPO (Group Relative Policy Optimization) algorithm with the Agentic Reward framework. It is applied after supervised fine-tuning on AesCode-358K, using a separate RL dataset to encourage generalization beyond the SFT distribution.

Why RL after SFT? The paper's motivation for adding an RL stage is explicit:

"After supervised fine-tuning in stage I, the model acquires substantial high-quality knowledge. However, the model at this stage still exhibits limited generalization beyond the training distribution, especially in webpage design tasks. This limitation highlights the necessity of reinforcement learning (RL), which allows the model to adapt more flexibly and robustly to diverse and unseen scenarios." (Section 5.2)

This echoes a finding reported in concurrent work (Chu et al., 2025, cited as [CZY+25]): "SFT memorizes, RL generalizes." The SFT stage teaches the model what good code aesthetics look like on the specific instructions in AesCode-358K, but the RL stage — with its exploration via sampling and reward-driven optimization — pushes the model to generalize these aesthetic principles to novel instructions.

RL data preparation. The RL training data comes from WebSight v0.2, a large synthetic dataset of HTML/CSS code with LLM-generated webpage descriptions. The authors select 20K examples from WebSight. However, the original WebSight descriptions are noted to be "relatively homogeneous, which does not align with the natural expression patterns of human users" (Section 5.2). To address this, they use the WebSight descriptions as seeds and have GPT-4o rewrite them into more natural, diverse user instructions (Appendix H.6).

This rewriting step is critical. If the RL prompts are too similar to the SFT prompts, the model can simply regurgitate memorized patterns and won't learn to generalize. By using WebSight descriptions as seeds and rewriting them with GPT-4o, the authors create prompts that are semantically similar (same webpage concept) but stylistically different (different phrasing, structure, level of detail), forcing the model to abstract aesthetic principles rather than pattern-match.

The rewriting prompt (Appendix H.6) instructs GPT-4o to "transform abstract ideas into compelling website concepts" and to "focus on the overall content, purpose, and features of the website, without going into specific layout, design, or visual details." It also requires selecting one of the five webpage categories and producing a JSON output with the category and a 40–60 word instruction.

GRPO objective with agentic reward. For each prompt $p$ in the RL dataset $D_{\text{RL}}$ , the model (old policy $\pi_{\theta_{\text{old}}}$ ) samples a group of $G = 8$ outputs $\{o_1, o_2, ..., o_G\}$ . Each output is evaluated by the Agentic Reward Framework, producing $G$ rewards $\{r_1, r_2, ..., r_G\}$ . The advantage for output $i$ is computed as:

$\hat{A}_{i,t} = \frac{r_i - \text{mean}(r)}{\text{std}(r)}$

where $r_i$ is the total agentic reward for output $i$ (computed via Equation 1, the weighted sum of execution, static, and interactive rewards), $\text{mean}(r)$ is the mean reward across the group of $G = 8$ outputs for the same prompt, and $\text{std}(r)$ is the standard deviation of rewards across the group.

What it computes: the z-score normalized advantage of each output relative to its group. An output with above-average reward gets a positive advantage; below-average gets negative. The normalization by standard deviation ensures the scale of advantages is consistent across prompts with different reward variances (e.g., a prompt where all outputs score similarly gets small advantages, while one with high variance gets large advantages).

Why this form: the group-relative normalization is the key innovation of GRPO over standard PPO. Standard PPO uses an absolute advantage (typically from a learned value function), which requires training a separate critic model and is sensitive to reward scale. GRPO's batch-relative advantage has three properties that make it well-suited for this setting:

No critic needed. Training a value function on agentic rewards — which are produced by an external multi-agent system, not a stationary environment — would be unstable and add complexity. Group-relative advantages eliminate this need.
Automatic reward normalization. The agentic rewards have different scales (execution is binary ±1, static aesthetics is 0–100, interactive is a count typically 0–3). The weighted sum mixes these scales, but the z-score normalization within each group ensures the effective advantage scale is consistent regardless of the absolute reward magnitude.
Relative quality signal. In aesthetic evaluation, absolute scores are inherently noisy (different prompts have different "maximum achievable aesthetics"), but relative comparisons within a group are more reliable. If one output looks better than another, the score difference is meaningful even if both absolute scores are uncertain.

The full GRPO-AR objective is:

$\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{p \sim D_{\text{RL}}, \{o_i\}_{i=1}^{G} \sim \pi_{\theta_{\text{SFT}}}(O|p)} \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \min\left( \frac{\pi_\theta(o_{i,t}|p, o_{i,<t})}{\pi_{\theta_{\text{SFT}}}(o_{i,t}|p, o_{i,<t})} \hat{A}_{i,t}, \text{clip}\left( \frac{\pi_\theta(o_{i,t}|p, o_{i,<t})}{\pi_{\theta_{\text{SFT}}}(o_{i,t}|p, o_{i,<t})}, 1 - \epsilon, 1 + \epsilon \right) \hat{A}_{i,t} \right) - \beta D_{\text{KL}}[\pi_\theta || \pi_{\text{ref}}] \right]$

where:

$G = 8$ is the group size (number of outputs sampled per prompt).
$|o_i|$ is the number of tokens in output $i$ .
$\frac{\pi_\theta(o_{i,t}|p, o_{i,<t})}{\pi_{\theta_{\text{SFT}}}(o_{i,t}|p, o_{i,<t})}$ is the probability ratio between the current policy $\pi_\theta$ and the SFT policy $\pi_{\theta_{\text{SFT}}}$ for token $t$ of output $i$ .
$\hat{A}_{i,t}$ is the group-normalized advantage (same for all tokens in output $i$ , hence the $t$ subscript is somewhat misleading — the advantage is per-output, not per-token).
$\epsilon = 0.5$ is the clipping parameter (Appendix E).
$\beta = 0.001$ is the KL penalty coefficient (Appendix E).
$D_{\text{KL}}[\pi_\theta || \pi_{\text{ref}}]$ is the Kullback-Leibler divergence between the current policy and a reference policy (presumably the SFT checkpoint).

What it computes: the standard PPO-style clipped surrogate objective, averaged over all tokens in all outputs in the group, minus a KL penalty. The min operation implements conservative policy iteration: if the probability ratio $\frac{\pi_\theta}{\pi_{\theta_{\text{SFT}}}}$ moves outside the range $[1-\epsilon, 1+\epsilon] = [0.5, 1.5]$ , the gradient is clipped to zero for that token, preventing the policy from changing too much in a single update.

Why this form: the clipped objective is standard in PPO and addresses the trust-region problem in policy gradient methods — without clipping, a large advantage estimate could cause a catastrophic policy update that destroys the model's language capabilities. The $\epsilon = 0.5$ value is notably larger than the typical PPO default of $\epsilon = 0.2$ , allowing more aggressive updates. This may reflect the observation that aesthetic alignment requires larger policy changes than typical RLHF tasks, or it may be an empirical tuning choice.

The KL penalty $-\beta D_{\text{KL}}[\pi_\theta || \pi_{\text{ref}}]$ provides additional regularization, preventing the policy from diverging too far from the SFT checkpoint (which already produces reasonable HTML). With $\beta = 0.001$ , this penalty is weak — it provides a gentle pull toward the SFT distribution rather than a hard constraint. This weak KL penalty combined with aggressive clipping ( $\epsilon = 0.5$ ) suggests the authors found that substantial policy movement is necessary and beneficial for aesthetics learning, as long as the model doesn't collapse entirely.

Training hyperparameters. The RL training configuration (Appendix E):

Optimizer: AdamW
Learning rate: constant $3 \times 10^{-6}$ (no warmup or decay specified for RL stage)
Batch size: 64 prompts
Micro batch size: 8
Rollout configuration: 64 prompts per rollout, 8 responses per prompt (yielding 512 outputs per rollout)
KL coefficient: $\beta = 0.001$
Clipping parameter: $\epsilon = 0.5$
Computational cost: approximately 7 days on 1 node of 8×MI300 GPUs for the RL phase

The RL dataset of 20K prompts is separate from the AesCode-358K SFT dataset, ensuring the model encounters novel instructions during RL — this tests and improves generalization rather than memorization.

Reward Aggregation

The three agent rewards are combined into a single scalar via a weighted sum (Equation 1):

$r = w_{\text{exec}} \cdot r_{\text{exec}} + w_{\text{static}} \cdot r_{\text{static}} + w_{\text{interact}} \cdot r_{\text{interact}}$

where:

$w_{\text{exec}} = 0.1$ is the weight for the execution reward.
$r_{\text{exec}} \in \{1, -1\}$ is the binary pass/fail from the Execution Agent.
$w_{\text{static}} = 0.8$ is the weight for the static aesthetics reward.
$r_{\text{static}} \in [0, 100]$ is the total score from the Static Aesthetics Agent.
$w_{\text{interact}} = 0.1$ is the weight for the interactive aesthetics reward.
$r_{\text{interact}} \in \{0, 1, 2, 3\}$ is the interaction success count from the Interactive Aesthetics Agent.

What it computes: a scalar reward that balances executability (10% weight), visual quality (80% weight), and interactive functionality (10% weight). The static aesthetics score dominates, reflecting the judgment that visual quality is the primary signal for code aesthetics, with executability and interactivity serving as guardrails.

Why these weights: The paper does not provide an explicit ablation or justification for the specific weights (0.1, 0.8, 0.1). However, the choices make intuitive sense given the scales and roles of each reward:

Static aesthetics at 0.8: This is the richest signal. The static score ranges from 0–100 and captures nuanced aesthetic quality across multiple dimensions. Making this the dominant weight ensures the model primarily optimizes for visual quality.
Execution at 0.1: The execution reward is binary (±1) and serves as a gating mechanism. A model that generates non-executable code gets a penalty that reduces its total reward, discouraging invalid output. However, the low weight (0.1) means executability alone cannot drive optimization — a perfectly valid but ugly page still gets a low total reward because the static aesthetics (0.8 weight) will be poor.
Interactive aesthetics at 0.1: The interactive score is a small integer (0–3 during training) and, like executability, serves primarily as a guardrail. If the model generates beautiful but non-functional pages, the interactive penalty (contributing at most 0.1 × 3 = 0.3 to the total reward) is small compared to the static aesthetics contribution (up to 0.8 × 100 = 80). This weighting implicitly assumes that static aesthetics are the primary objective and interactivity should not be sacrificed for visual quality — but neither should it be completely ignored.

The 0.1/0.8/0.1 split also means the dynamic range of contributions is roughly balanced: a perfect execution score contributes ±0.1, a perfect static score contributes up to 80, and perfect interactivity contributes up to 0.3. This means static aesthetics dominates the absolute reward magnitude, while execution and interactivity provide small additive adjustments. In practice, this means the model's optimization is primarily driven by "does it look good?" with mild pressure toward "does it work?" — a reasonable prioritization for a task explicitly about aesthetics.

Alternative weighting considered? The paper does not discuss alternatives, but one could imagine different weightings for different use cases: a production deployment prioritizing reliability might increase $w_{\text{exec}}$ and $w_{\text{interact}}$ , while a creative design tool might increase $w_{\text{static}}$ even further. The fixed weights represent a specific point on a tradeoff curve, and the paper does not explore sensitivity to these choices.

Why not a learned weighting? An alternative to fixed weights would be to learn the weighting from human preference data — train a reward model that takes all three agent outputs and predicts a human preference score. The paper does not pursue this, likely because the fixed weights are simpler, more interpretable, and don't require additional human annotation (which the paper specifically avoids through its automated agent-based approach).

4. Key Insights and Innovations

Innovation 1: Code Aesthetics as a Distinct Capability Axis, Not a Sub-Problem of Code Correctness

The paper's most fundamental conceptual move is carving out code aesthetics as a distinct capability that sits orthogonal to traditional code correctness metrics. Before this work, the field's implicit assumption was that if code compiles, runs, and passes tests, it's "good code"—period. Visual quality was either ignored entirely or treated as an afterthought that a human would handle in post-processing. The paper argues this dichotomy is false for a large and growing class of programming tasks where "the correctness or quality of the code is inherently tied to its visual output" (Section 1).

What makes this framing distinctive is that it doesn't simply add a new evaluation metric to an existing code-generation benchmark. It identifies a fundamentally different signal modality that the standard training pipeline for coding LLMs is structurally incapable of capturing. A unit test can tell you whether plt.bar(x, y) produces a bar chart; it cannot tell you whether the legend overlaps the bars, whether the color scheme is coherent, or whether the font sizes are readable. These are not bugs in the traditional sense—they're failures of design, not failures of logic—but they're failures nonetheless from the user's perspective.

The paper's diagnostic framing—"do LLMs have a sense of aesthetics?"—is deliberately provocative but precise. It reframes the problem from "how do we improve code generation?" (a performance question) to "what feedback signal is missing from the training pipeline?" (a systems question). This shift explains why prior work on RL for code (RLTF, RLEF, CodeRL) couldn't address this problem even in principle: their reward signals live entirely in the textual modality, and aesthetics information simply doesn't exist in that modality.

The evidence that this distinction matters comes from the baseline results in Table 1: the base Qwen2.5-Coder-7B-Instruct achieves only 46.27 total static score on OpenDesign and 0.38 interactive score—despite being a strong coding model on traditional benchmarks. This gap between textual coding capability (presumably strong, given Qwen2.5-Coder's design) and aesthetic output quality validates that code aesthetics is not automatically conferred by general coding competence. Something structurally different—training on rendered outputs, receiving visual feedback—is required.

Innovation 2: Multi-Agent Decomposition of Aesthetic Evaluation into Executability, Static, and Interactive Dimensions

The paper's second conceptual contribution is the three-agent decomposition of the reward signal. Rather than attempting to build a single "aesthetics reward model" that takes code as input and outputs a holistic quality score, the paper argues that code aesthetics naturally factor into three irreducible dimensions—executability, static visual quality, and interactive functionality—that require fundamentally different evaluation mechanisms operating in different modalities.

This decomposition is significant not because it's technically complex (the individual agents use off-the-shelf components: HTMLHint for linting, Playwright for rendering, GPT-5 for visual judging, WebVoyager for interaction), but because it recognizes that no single evaluation modality can assess all three dimensions. Prior work on reward modeling for code implicitly assumed that a single model—whether human annotators (RLHF), unit tests (RLTF), or learned reward models (CodeRL)—could capture everything that matters about code quality. The paper demonstrates that this assumption fails catastrophically for visually-oriented tasks.

Consider what each dimension requires:

Executability needs syntactic validation against language rules—a purely textual task that a linter handles perfectly.
Static aesthetics needs visual inspection of rendered output—a multimodal task requiring vision-language reasoning that even advanced LLMs find challenging.
Interactive aesthetics needs sequential decision-making in a dynamic environment—an agentic task requiring planning, action, and outcome assessment.

These are not just different evaluation criteria; they're different evaluation paradigms that require different architectures. The execution agent is a deterministic rule-checker. The static aesthetics agent is a multimodal LLM judge. The interactive aesthetics agent is a web-navigation agent. Trying to collapse these into a single model would either lose information (a text-only model can't see the rendered page) or produce unreliable proxy signals (a vision model guessing interactivity from a static screenshot).

The paper's weight allocation (0.1 executability, 0.8 static, 0.1 interactive) further reveals a substantive claim: for code aesthetics, visual quality dominates, but executability and interactivity serve as necessary guardrails. A perfectly beautiful webpage that doesn't render is worthless; a gorgeous static layout with broken buttons is deceptive. The weights encode a specific value judgment about the relative importance of these dimensions, and while the paper doesn't ablate these weights extensively, the decomposition itself makes the tradeoff explicit and adjustable—a feature that a monolithic reward model would obscure.

The ablation in Table 2 provides direct evidence for the necessity of this decomposition. Removing the agentic reward framework and replacing it with a single text-based reward model scoring HTML source code along the same three dimensions ("GRPO-AR w/o Agentic Reward") consistently underperforms the full multi-agent approach. For Qwen3-4B-Instruct-2507, the interactive score drops from 1.04 to 0.71; for Qwen2.5-Coder-7B-Instruct, it drops from 0.94 to 0.72. This confirms that the modality of evaluation matters—a text-based judge cannot adequately assess visual and interactive quality, even when asked to score the same dimensions.

Innovation 3: Interactive Aesthetics as a First-Class Dimension with Agent-Based Evaluation

The introduction of interactive aesthetics as a distinct evaluation dimension—and the use of a web-navigation agent to assess it—is arguably the paper's most forward-looking conceptual contribution. Prior work on aesthetics in AI-generated content (both text and image domains) has been exclusively static: evaluate a text's layout, score an image's composition, judge a design's visual appeal from a screenshot. The paper identifies a fundamental limitation in this static paradigm: for webpages, especially interactive ones (3D scenes, games, data dashboards), "evaluation based only on static screenshots is insufficient, as it overemphasizes visual appearance while neglecting usability" (Section 4.3).

This is not a minor extension—it's a category shift from evaluating appearance to evaluating behavior. A static screenshot can tell you whether a game's graphics look polished; it cannot tell you whether the game is playable. It can show you a navigation bar; it cannot tell you whether clicking the links actually works. The interactive aesthetics agent addresses this by actually using the webpage as a user would, testing whether the promised functionality exists and operates correctly.

What makes this innovation intellectually distinctive is that it treats agent capability as evaluation infrastructure. The paper leverages the recent maturation of web agents (WebVoyager) not to accomplish a task for an end-user, but to judge the quality of an LLM's output during training. This is a meta-application of agent technology: using one AI system (a web agent) to evaluate and provide feedback on another AI system (a code-generating LLM), with the evaluation operating in a modality (interactive web browsing) that neither system could assess individually.

The paper's candid discussion of limitations (Section 4.3) elevates rather than undermines this contribution. The acknowledgment that "agent failures lead to a score of 0 in the corresponding iteration" and "this may cause the agent to make incorrect judgments, resulting in scores lower than the true values" demonstrates an honest engagement with the noise characteristics of agent-based evaluation. The partial defense—that "agent failures also partially reveal non-standard or sub-optimal aspects of webpage design"—reframes the limitation as a feature: a webpage that confuses an AI agent designed for robust interaction is likely confusing to human users as well. This is a subtle but important conceptual point: evaluation difficulty and interaction quality are not independent.

The practical significance is that interactive evaluation opens the door to training models for tasks that were previously untrainable with automated reward signals. Browser-based games, interactive data visualizations, 3D scene explorers—these are tasks where the "correctness" of the output can only be determined through interaction, not static inspection. The paper demonstrates that this is feasible at the scale required for RL training (20K prompts, each evaluated with multiple interactions), establishing a template for future work on interactive code generation tasks.

Innovation 4: GRPO's Group-Relative Advantage as a Natural Fit for Aesthetic Reward Signals

The paper's methodological insight—that GRPO's batch-relative advantage computation is particularly well-suited to aesthetic evaluation—is subtle but important. It's not merely that GRPO is used as a drop-in replacement for PPO; it's that the properties of GRPO align with the properties of aesthetic reward signals in ways that standard PPO would not.

Aesthetic evaluation is inherently relative and noisy. Absolute scores ("this webpage is 73/100") are less meaningful and less reliable than relative comparisons ("this webpage is better than that one"). Two different prompts might have very different ceilings for achievable aesthetics—a request for a "simple contact form" can only be so beautiful, while a request for an "immersive 3D art gallery" has a much higher ceiling. A reward model that assigns absolute scores would need to account for this prompt-dependent scaling, which is difficult.

GRPO's group-relative advantage (r_i - mean(r)) / std(r) automatically handles this. Within each batch of 8 outputs for the same prompt, the model sees which outputs are above-average and which are below-average, relative to what was achievable for that specific prompt. An output scoring 60/100 might receive a positive advantage if all other outputs for that prompt scored 40-55, or a negative advantage if all others scored 70-90. This prompt-conditioning of the advantage signal—without needing to explicitly model prompt difficulty—is exactly what aesthetic evaluation needs.

This is not an incidental implementation choice; it's a conceptual alignment between the optimization algorithm and the nature of the reward signal. Standard PPO with a learned value function would attempt to predict the expected reward for each prompt, but training a value function on agentic rewards—which come from an external, non-stationary multi-agent system—would be unstable and introduce additional complexity. GRPO sidesteps this entirely by making the relative comparison within the batch serve as the implicit value baseline.

The paper doesn't ablate GRPO against standard PPO, so we can't directly attribute performance gains to this choice. But the intellectual contribution is the recognition of the alignment, not necessarily the empirical demonstration of its superiority over alternatives. It represents a case where the structure of the optimization is deliberately matched to the structure of the problem, which is a deeper form of methodological contribution than simply trying different algorithms and picking the best one.

The choice of ε = 0.5 (unusually large for PPO-style clipping) and β = 0.001 (weak KL penalty) further suggests that aesthetic alignment requires more aggressive policy updates than typical RLHF tasks. The paper doesn't justify these hyperparameters explicitly, but the pattern—large clipping range, weak regularization—is consistent with the observation that SFT alone leaves substantial room for improvement (the SFT model still produces mediocre aesthetics on many prompts), and that substantial policy movement is necessary to internalize the aesthetic feedback signal.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on two benchmarks. For Python plot generation, PandasPlotBench (Galimzyanov et al., 2025) is used in its "head descriptor" and "vis" mode configuration; the test set size is not explicitly stated but the evaluation protocol involves scoring each case with GPT-4o on a 0–100 scale. For webpage design, the authors' newly introduced OpenDesign benchmark (Section 6) provides 840 real-world webpage design cases distributed across five categories: General Website (60.9%), 3D Design (14.6%), Data Visualization (4.8%), Game Dev (13.6%), and UI Component (4.9%) (Table 4). Unlike PandasPlotBench, which compares generated plots against ground-truth images, OpenDesign has no ground truth — it relies entirely on LLM-as-a-judge scoring using the same static and interactive aesthetics agents from the training pipeline.
Base model(s). Two base models of different scales from different families are used: Qwen3-4B-Instruct-2507 (4B parameters) and Qwen2.5-Coder-7B-Instruct (7B parameters) (Section 5.1). The dual-model evaluation serves to test generalizability of the training methodology across model families and scales. Qwen2.5-Coder-7B-Instruct is specifically a code-focused model, while Qwen3-4B-Instruct-2507 is a general-purpose instruction-tuned model — this contrast tests whether code-specific pretraining confers advantages for aesthetic tasks or whether the post-training pipeline can compensate.
Metrics. Three metrics are reported on PandasPlotBench: (i) Error rate (lower is better), the fraction of test cases that fail the executability check; (ii) Average score (higher is better), the mean GPT-4o score (0–100) across all test cases, where GPT-4o compares the generated plot image against the ground truth; and (iii) Good rate (higher is better), the proportion of cases scoring above 75. On OpenDesign, two composite metrics are reported: Static Aesthetics, decomposed into three sub-scores — Alignment (instructional alignment with user instruction, out of 40), Aesthetics (visual elements and design quality, out of 30), and Structure (structural cohesion and layout, out of 30) — which sum to a Total score out of 100; and Interactive Aesthetics (InterAes), the sum of binary success/failure judgments across all tested interactions (a count, not a rate). All OpenDesign metrics are averaged across the 840 benchmark cases.
Baselines. The paper compares against both proprietary and open-source models (Table 1). Proprietary baselines include GPT-4o-mini, GPT-4o, GPT-4.1, GPT-5 (minimal configuration), and Claude Sonnet 4. Open-source large-model baselines include Qwen3-Coder-30B-A3B (30B), GLM-4-32B-0414 (32B), GLM-4.5-Air (110B), Qwen3-Coder-480B-A35B (480B), DeepSeek-V3.1 (685B), and DeepSeek-R1-0528 (685B). Open-source small-model baselines are the untuned versions of the base models: Qwen3-4B-Instruct-2507 and Qwen2.5-Coder-7B-Instruct. For ablation comparisons (Table 2), baselines include the SFT-only checkpoint, RFT (Rejection Sampling Fine-Tuning, Yuan et al., 2023), DPO (Direct Preference Optimization, Rafailov et al., 2024), and a GRPO-AR w/o Agentic Reward variant that replaces the three-agent system with a single text-based reward model scoring HTML source code along the same three dimensions. No baseline uses the same GRPO-AR approach on a different model architecture, nor is there a baseline that gives test-time compute budgets to the proprietary models (all proprietary models are evaluated with single greedy or default decoding).
Generation budget / compute accounting. In the RL stage, the generation budget per prompt is fixed at G = 8 outputs per prompt, with 64 prompts per rollout batch, yielding 512 total generations per training step (Appendix E). For evaluation, all models are compared on the same set of prompts with a single generation per prompt — there is no best-of-N or majority voting at evaluation time. This means the comparison between AesCoder models and baselines is single-sample vs. single-sample, making it a test of per-sample quality rather than test-time compute scaling. The paper does not report how many tokens are generated per sample, nor does it compute total inference FLOPs for fair comparison against larger models. The training compute is reported in GPU-days: approximately 2 days on 8×MI300 GPUs for SFT (7B model), and approximately 7 days on 8×MI300 GPUs for the RL phase (Appendix E), but these are not used for any FLOPs-matched comparison.
Cross-validation / statistical protocol. For OpenDesign's reliability validation (Section 6.2), the paper reports Spearman's and Kendall's rank correlation coefficients with p-values for the ranking consistency between OpenDesign scores and Design Arena human-voted rankings across 10 models. For human alignment, 200 HTML page pairs from 10 models were evaluated by both GPT judges and 10 human evaluators (3 professors, 7 graduate students), yielding 2,000 pairwise annotations. For the main results in Table 1, no confidence intervals, standard deviations, or statistical significance tests are reported. The paper does not describe any cross-validation protocol for hyperparameter selection beyond the fixed configuration in Appendix E.

Main Quantitative Results

OpenDesign Benchmark Performance

The headline result appears in Table 1 and Figure 1: AesCoder-4B achieves a static aesthetics total score of 81.92 and an interactive score of 1.04, surpassing GPT-4o (48.08 static, 0.44 interactive), GPT-4.1 (65.79 static, 0.74 interactive), and all listed open-source models including those with 480B–685B parameters. Specifically, Qwen3-Coder-480B-A35B scores 79.90 static, 0.70 interactive; DeepSeek-V3.1 scores 77.72 static, 0.88 interactive; DeepSeek-R1-0528 scores 78.86 static, 0.77 interactive. AesCoder-7B achieves 81.23 static, 0.94 interactive — slightly lower than the 4B variant but still competitive with the largest models.

The improvement over base models is dramatic: Qwen3-4B-Instruct-2507 starts at 73.26 static, 0.67 interactive, and after SFT+GRPO-AR reaches 81.92 and 1.04 — a gain of 8.66 points on static aesthetics (11.8% relative improvement) and 0.37 on interactive score (55% relative improvement). Qwen2.5-Coder-7B-Instruct starts at an even lower baseline of 46.27 static, 0.38 interactive and reaches 81.23 and 0.94 after training — a gain of 34.96 points on static aesthetics (75.6% relative improvement) and 0.56 on interactive score (147% relative improvement). The 7B model's much lower starting point reflects that Qwen2.5-Coder is optimized for textual code correctness, not visual output, making the aesthetic improvement even more striking.

Looking at the sub-score breakdown (Alignment, Aesthetics, Structure), AesCoder-4B's strong performance is consistent across all three dimensions: Alignment 30.42 (out of 40), Aesthetics 26.19 (out of 30), Structure 25.31 (out of 30). These are all within striking distance of GPT-5 minimal (30.38, 25.94, 24.71) and Claude Sonnet 4 (29.60, 25.92, 25.53), and in some sub-dimensions slightly exceed them. The fact that AesCoder-4B scores 30.42 on Alignment vs. GPT-5's 30.38 — a sub-score specifically about following user instructions — suggests that the training pipeline doesn't just improve visual polish but also enhances instruction-following fidelity, possibly because the static aesthetics agent explicitly evaluates alignment.

The most remarkable single number is AesCoder-4B's interactive score of 1.04, the highest among all open-source models and second only to GPT-5's 1.37 among all evaluated models. This is significant because the interactive agent's evaluation (scoring binary success/failure of interactions) is conceptually harder to game than static screenshot evaluation — a model cannot produce a high interactive score by generating visually deceptive HTML; it must generate code that actually functions when an agent tries to click, type, and navigate.

PandasPlotBench Performance

On PandasPlotBench (Table 1), AesCoder-4B achieves an error rate of 0.09, average score of 70, and good rate of 0.63. This represents substantial improvement over the Qwen3-4B baseline (error rate 0.13, average 65, good rate 0.55) and dramatically outperforms Qwen2.5-Coder-7B's baseline (error rate 0.22, average 60, good rate 0.50). Notably, AesCoder-4B matches or exceeds GPT-4o on all three metrics (GPT-4o: error rate 0.09, average 68, good rate 0.60) and approaches GPT-4.1 (error rate 0.09, average 69, good rate 0.61).

The AesCoder-7B results on PandasPlotBench are initially puzzling: error rate 0.09, average 67, good rate 0.57 — worse than AesCoder-4B on all metrics despite being a larger model. The paper does not discuss this inversion, but it might reflect that Qwen2.5-Coder-7B's starting point (error rate 0.22) was much worse than Qwen3-4B's (error rate 0.13), and the SFT+RL pipeline improved it dramatically but not quite enough to surpass the 4B variant within the fixed training budget. Alternatively, it could indicate that Qwen3-4B-Instruct-2507's general instruction-tuning provides a better foundation for plot generation aesthetics than Qwen2.5-Coder-7B's code-specific pretraining, which may overfit to textual code patterns at the expense of visual reasoning.

The fact that both AesCoder variants achieve identical error rates (0.09) on PandasPlotBench, significantly better than their base models, confirms that the execution agent's penalty (s_exec = -1 for non-executable code) is effective at teaching the models to generate valid, runnable plotting code. The improvement in good rate (from 0.55 to 0.63 for 4B, from 0.50 to 0.57 for 7B) suggests that beyond mere executability, the models are learning to produce high-quality plots that score above 75 — that is, genuinely good visualizations, not just passable ones.

Generalization Across Model Families and Scales

The consistent improvement across both base models (Qwen3-4B and Qwen2.5-Coder-7B) on both benchmarks demonstrates that the training pipeline generalizes across model families and scales. The fact that AesCoder-4B outperforms AesCoder-7B on most metrics is an interesting negative result that the paper does not discuss — it suggests that the base model's initial quality (Qwen3-4B starts stronger on these tasks than Qwen2.5-Coder-7B) matters more than model scale within this range, at least for the fixed training budget used.

Comparison with Proprietary Models

While AesCoder-4B surpasses GPT-4o and GPT-4.1 by substantial margins on both benchmarks, it remains slightly behind GPT-5 and Claude Sonnet 4. GPT-5 (minimal) achieves 81.03 static, 1.37 interactive, and Claude Sonnet 4 achieves 81.05 static, 0.92 interactive — both roughly comparable to AesCoder-4B's 81.92 static, 1.04 interactive. The differences are small (within 1 point on static, within 0.33 on interactive) and the paper does not provide confidence intervals, so it's unclear whether these differences are statistically significant. What is clear is that a 4B model trained with agentic reward feedback achieves parity with state-of-the-art proprietary models that are presumably much larger (GPT-5 and Claude Sonnet 4's parameter counts are not publicly disclosed, but they are widely believed to be in the hundreds of billions to trillions).

RL Training Dynamics

Appendix G (Figure 6) shows the reward curves during GRPO-AR training for both models. Both Qwen2.5-Coder-7B-Instruct-SFT and Qwen3-4B-Instruct-2507-SFT exhibit "steadily increasing reward scores with training steps" over approximately 250 steps. The curves display typical RL training behavior — noisy but with a clear upward trend — and the paper notes that this "indicates that the agentic reward framework provides stable and informative feedback, enabling continuous improvement across different model families and sizes." However, the Figure 6 y-axis ranges from roughly 0.25 to 0.55, which appears to be normalized reward (the static aesthetics scores are in the 0–100 range, so the plotted values likely represent some aggregated or normalized signal). The paper does not specify exactly what is plotted on the y-axis of Figure 6.

Human Evaluation

Appendix F reports a human preference study comparing AesCoder models against four baselines: Claude Sonnet 4, GPT-5, GLM-4-32B-0414, and Qwen3-Coder-30B-A3B-Instruct. The results (Figure 5 in the appendix, referenced as Table 5 in the text) show that "AesCoder achieves a win rate of over 55% in comparisons with mid- to large-scale open-source models (GLM-4-32B-0414 and Qwen3-Coder-30B-A3B-Instruct), and maintains a near 50% win rate when compared to state-of-the-art proprietary models (Claude Sonnet 4 and GPT-5)." This human validation is important because the OpenDesign benchmark itself uses LLM-as-a-judge (GPT-5 for static, GPT-4o for interactive), creating a potential circularity where the training reward model and the evaluation judge are the same or similar systems. The human study partially mitigates this concern by showing that human preferences align with the benchmark rankings.

Ablation Studies and Robustness Checks

Agentic reward vs. text-based reward (GRPO-AR w/o Agentic Reward): Replacing the full three-agent system with a single reward model that scores HTML source code textually along the same three dimensions (Instructional Alignment, Visual Design, Structural Coherence) consistently degrades performance across both models (Table 2). For Qwen3-4B-Instruct-2507, the interactive score drops from 1.04 to 0.71; static total drops from approximately 82 to roughly 79 (computed as alignment + aesthetics + structure: 30.42 + 26.19 + 25.31 = 81.92 vs. 29.16 + 25.20 + 24.67 = 79.03). For Qwen2.5-Coder-7B-Instruct, the interactive score drops from 0.94 to 0.72; static total drops from 81.23 to roughly 78.24. The static sub-scores reveal an interesting pattern: the text-based reward variant actually achieves comparable or slightly better alignment scores for Qwen2.5-Coder-7B (29.75 vs. 30.03 for the agentic version on alignment sub-score in the DPO row? — wait, the DPO and ablation rows need careful reading). Looking at Table 2 carefully: for Qwen2.5-Coder-7B, "GRPO-AR w/o Agentic Reward" achieves Alignment 28.81, Aesthetics 25.02, Structure 24.41, InterAes 0.72, while "GRPO-AR w/ Agentic Reward" achieves 30.03, 25.98, 25.18, 0.94. The gap is largest on Alignment (+1.22) and InterAes (+0.22), confirming that visual inspection and interactive testing capture information that text-based evaluation of source code cannot.

SFT-only vs. SFT+RL: The SFT-only checkpoints (Table 2, "SFT" rows) establish the contribution of reinforcement learning over supervised fine-tuning. For Qwen3-4B-Instruct-2507, SFT alone achieves static sub-scores of 28.50, 25.27, 24.36 and InterAes 0.62, compared to 30.42, 26.19, 25.31, 1.04 after GRPO-AR — gains of roughly +1.9, +0.9, +0.95 on static sub-scores and +0.42 on interactive. The Qwen2.5-Coder-7B SFT checkpoint (28.85, 25.23, 24.37, 0.70) improves to 30.03, 25.98, 25.18, 0.94 after GRPO-AR. The static improvements from RL are modest (+1–2 points per sub-dimension), consistent with the paper's characterization that "SFT memorizes, RL generalizes" — SFT already captures most of the aesthetic quality, and RL provides incremental generalization gains. The interactive improvements are more substantial proportionally (+0.42 on a base of 0.62 for 4B, +0.24 on a base of 0.70 for 7B), suggesting that interactive functionality is harder to learn from static SFT data and benefits more from RL exploration.

Comparison with DPO and RFT: Both DPO and RFT underperform GRPO-AR on all metrics for both models (Table 2). For Qwen3-4B-Instruct-2507, DPO achieves static sub-scores of 28.79, 25.31, 24.38 and InterAes 0.70; RFT achieves 29.32, 25.30, 24.67 and 0.71. GRPO-AR outperforms both: 30.42, 26.19, 25.31, 1.04. The pattern is similar for Qwen2.5-Coder-7B: DPO (29.75, 25.33, 24.87, 0.71) and RFT (29.73, 25.35, 24.85, 0.75) are both below GRPO-AR (30.03, 25.98, 25.18, 0.94). The gap is most pronounced on interactive aesthetics, where GRPO-AR nearly doubles the score of DPO/RFT for the 4B model (1.04 vs. 0.70/0.71). This is a strong result because DPO and RFT are standard RLHF methods that use the same reward model to construct preference pairs or select top samples — the fact that GRPO-AR substantially outperforms them demonstrates that the on-policy exploration and batch-relative advantage computation of GRPO provides benefits beyond simply having a good reward signal. It also suggests that the agentic reward signal is sufficiently nuanced that collapsing it into binary preferences (DPO) or best-of-N selection (RFT) loses important information compared to the continuous, group-normalized advantage used by GRPO.

Training stability and reward quality: Figure 6 (reward curves) serves as an implicit ablation of reward stability. The steadily increasing curves for both models indicate the agentic reward framework does not exhibit the reward hacking or over-optimization patterns that often plague RL training with learned reward models. This is notable because the static aesthetics agent (GPT-5) is a learned model that could, in principle, be exploited by the policy. The fact that rewards continue to increase without plateauing or declining over 250 steps suggests either that the agentic reward is robust to optimization pressure, or that 250 steps is insufficient to reach the over-optimization threshold. The paper does not discuss this explicitly.

OpenDesign reliability validation: Section 6.2 provides robustness evidence for the benchmark itself. The rank correlation with Design Arena (Spearman = 0.98, Kendall = 0.91) across 10 models demonstrates that OpenDesign's LLM-judge rankings closely mirror large-scale human preference rankings. Figure 3a visualizes this alignment — model ranks cluster tightly along the diagonal. The human-GPT agreement rate of 80.9% on pairwise comparisons (Figure 3b) compares favorably to human-human agreement of 68.7%, suggesting that GPT judges are actually more consistent with aggregate human preference than individual humans are with each other — a known phenomenon in preference evaluation where LLM judges can serve as low-variance proxies for the "average human." The paper notes these rates are "comparable to MT-Bench results (66% and 70%, respectively)," referencing established LLM-as-a-judge benchmarks.

Critical Assessment

Claim: AesCoder-4B surpasses GPT-4o and GPT-4.1 on code aesthetics

This claim is supported by the Table 1 results, but with important caveats about what "surpasses" means. On OpenDesign, AesCoder-4B (81.92 static, 1.04 interactive) does clearly exceed GPT-4o (48.08 static, 0.44 interactive) and GPT-4.1 (65.79 static, 0.74 interactive). On PandasPlotBench, AesCoder-4B (error 0.09, avg 70, good 0.63) similarly exceeds GPT-4o (error 0.09, avg 68, good 0.60) and GPT-4.1 (error 0.09, avg 69, good 0.61). However, the comparison is not entirely fair: AesCoder-4B is specifically fine-tuned and RL-optimized for these exact tasks, while GPT-4o and GPT-4.1 are general-purpose models evaluated zero-shot. A properly controlled comparison would fine-tune GPT-4o on the same AesCode-358K dataset or give it access to the same agentic reward feedback, but the proprietary models' training APIs do not permit this. The claim as stated — that a small specialized model can outperform large general models on a specific task — is true and practically meaningful (it shows the value of task-specific post-training), but it does not demonstrate that the 4B architecture is inherently superior; it demonstrates that targeted training on aesthetics data defeats general capability at larger scale.

Additionally, the proprietary models are evaluated with single generations and no test-time compute optimization. GPT-5 with best-of-N sampling or iterative refinement might perform substantially better, but this is not tested. The comparison is thus between a highly optimized, task-specific pipeline and a general-purpose model in its default configuration — a comparison that favors the specialized system by construction.

Claim: AesCoder achieves performance comparable to open-source models with 480B–685B parameters

This claim is well-supported by Table 1. AesCoder-4B (81.92 static, 1.04 interactive) exceeds Qwen3-Coder-480B-A35B (79.90, 0.70), DeepSeek-V3.1 (77.72, 0.88), and DeepSeek-R1-0528 (78.86, 0.77). AesCoder-7B (81.23, 0.94) similarly outperforms these models. The claim is stated conservatively as "comparable" rather than "surpasses," which is appropriate given that the differences are modest and confidence intervals are not reported. The fact that a 4B model can match or exceed 480B–685B models on this specific task is genuinely impressive and supports the paper's implicit argument that task-specific training with appropriate reward signals can substitute for raw scale — but only when the base model already possesses the fundamental capability (both Qwen3-4B and Qwen2.5-Coder-7B can already generate HTML and Python; the training improves aesthetics, not basic code generation ability).

A critical unexamined question is whether the 480B+ models would benefit similarly from the AesCode-358K SFT + GRPO-AR pipeline. If they would — and there's no reason to assume they wouldn't — then the absolute ceiling for code aesthetics might be substantially higher than what AesCoder-4B achieves, and the "comparable to 480B models" claim reflects a training disparity rather than a fundamental capability ceiling.

Claim: Multi-agent reward feedback effectively aligns small models with human-perceived code aesthetics

This is the paper's central methodological claim, and the evidence is strong but incomplete. The ablation (Table 2, "GRPO-AR w/o Agentic Reward") demonstrates that removing the multi-agent architecture and replacing it with text-based scoring degrades performance, confirming that the agentic decomposition matters. The human evaluation (Appendix F) demonstrates that the resulting models produce outputs preferred by humans over strong baselines, confirming that the automated reward signal correlates with human preferences.

However, the claim that the multi-agent aspect specifically is responsible has not been fully isolated. The agentic reward framework differs from the text-based ablation in multiple simultaneous ways: (1) it uses visual input (screenshots) rather than text input (source code), (2) it uses interactive testing rather than static evaluation, (3) it uses different models for different evaluation dimensions (GPT-5 for static, GPT-4o for interactive, HTMLHint for execution) rather than a single model. The ablation conflates all three differences. A more rigorous decomposition would test: what if we keep the multi-agent structure but use text-only evaluation? What if we use visual evaluation but without interaction testing? What if we use a single multimodal model for all three evaluations? Without these ablations, we know that the full system works better than a single text-based judge, but we don't know which components (visual modality? interactive testing? multi-model diversity?) drive the improvement.

The "hard problems benefit from interactive evaluation, easy problems from refined visual scoring" claim

The prior sections' executive summary includes this claim, but the paper does not perform a difficulty-based analysis of its results. Unlike the reference example paper (which splits results by five difficulty quintiles), this paper reports only aggregate scores. We cannot determine from the reported experiments whether interactive evaluation helps more on hard problems (e.g., complex 3D scenes or games where static screenshots are insufficient) or easy problems (where interactions are simple and reliable). The OpenDesign benchmark has five categories (General Website, 3D Design, Data Visualization, Game Dev, UI Component), and one might hypothesize that interactive aesthetics matter most for Game Dev and 3D Design while static aesthetics dominate for General Website and UI Component — but the paper does not report per-category breakdowns. This is a significant gap if the claim about difficulty-dependent effectiveness is being made.

Genuine weaknesses

Single evaluation protocol, circularity risk. The OpenDesign benchmark uses GPT-5 as the static aesthetics judge and GPT-4o as the interactive judge — the same models (or close variants) that provide the reward signal during GRPO-AR training. While the human evaluation (Appendix F) partially addresses this by showing human preferences align with model scores, it only evaluates 100 test cases and only for the final models, not for intermediate training dynamics. There remains a risk that the GRPO-AR training is optimizing for quirks of GPT-5's aesthetic preferences rather than universally human-preferred aesthetics, and the human study's 80.9% agreement rate, while good, leaves 19.1% of judgments where human and GPT disagree — potentially representing aesthetic dimensions where the training signal was misleading.

No per-category analysis. The OpenDesign benchmark includes five diverse categories (Table 4: 60.9% General Website, 14.6% 3D Design, 4.8% Data Visualization, 13.6% Game Dev, 4.9% UI Component), but results are reported only in aggregate. AesCoder-4B might excel at General Website design while struggling on Game Dev (where interactive testing is more demanding), or vice versa. Without per-category breakdowns, we cannot assess whether the training pipeline produces uniformly good results or is carried by the dominant General Website category.

Small sample for human evaluation. The human study uses 100 test cases sampled from OpenDesign's 840 cases, evaluated by 10 human raters on pairwise comparisons. While 2,000 total annotations is reasonable, the 100-case sample is small relative to the benchmark's diversity, and per-category human agreement rates (which would reveal whether humans and GPT judges agree more on General Website vs. Game Dev) are not reported. The paper reports 66.7% top-3 and 80.0% top-5 overlap with Design Arena, but doesn't discuss which models are misranked or whether the rankings are stable across benchmark categories.

No sensitivity analysis of reward weights. The fixed weights (0.1 executability, 0.8 static, 0.1 interactive) are stated but never varied. We don't know whether AesCoder's performance is robust to these choices or whether different weightings would produce meaningfully different aesthetic styles (e.g., higher interactive weight might produce plainer but more functional pages). This is a missed opportunity because the weight sensitivity would reveal the shape of the tradeoff surface between static beauty and interactive functionality.

Unclear RL training dynamics details. Figure 6 shows reward curves but doesn't specify exactly what is plotted (normalized total reward? static aesthetics only?). The curves show increasing reward, but without a held-out validation reward curve, we cannot assess whether the model is overfitting to the RL prompts or genuinely improving. The paper doesn't report whether the 250-step stopping point was chosen based on validation performance or simply fixed in advance.

Missing comparison: test-time compute for baselines. All baselines (including proprietary and large open-source models) are evaluated with single greedy generations. Given that the reference example paper demonstrated 4× efficiency gains from test-time compute scaling, it's plausible that GPT-4o with best-of-8 or Claude Sonnet 4 with iterative refinement would close much of the gap with AesCoder. The paper doesn't explore this, making the "4B surpasses GPT-4o" claim potentially misleading if test-time compute is the relevant comparison axis rather than single-sample quality.

PandasPlotBench as a secondary benchmark. The PandasPlotBench results receive limited analysis — no per-plot-type breakdown, no qualitative examples of improved vs. failed plots, no discussion of the strong performance of AesCoder-4B over AesCoder-7B. The benchmark serves primarily as validation that the training pipeline transfers to non-HTML visual coding tasks, but the analysis is shallow.

No latency or cost analysis for evaluation. The agentic reward framework requires: (1) rendering each HTML in Playwright, (2) sending a full-page screenshot to GPT-5 for scoring, (3) launching a browser agent with GPT-4o to perform multiple interactions. For 512 outputs per training step, this is computationally expensive. The paper reports training time (7 days on 8×MI300 GPUs) but doesn't break down what fraction is model forward passes vs. agent evaluation. In practice, the evaluation cost might dominate and limit the scalability of this approach to larger models or longer training.

Missing experiments that would strengthen the paper

Per-category OpenDesign breakdown: Reporting static and interactive scores for each of the five webpage categories would reveal whether the training pipeline produces broad aesthetic competence or is specialized to certain webpage types.
Reward weight sensitivity: Varying w_static and w_interact (e.g., 0.9/0.0, 0.5/0.4, 0.0/0.9) would reveal the Pareto frontier between static beauty and interactive functionality and validate that the chosen weights represent a reasonable tradeoff.
Test-time compute scaling for baselines: Giving GPT-4o or Claude Sonnet 4 best-of-8 or best-of-16 with verifier-based selection would test whether AesCoder's advantage persists when baselines can also leverage additional inference compute.
Scaling SFT data volume: How does performance vary with 50K, 100K, 200K, or the full 358K AesCode instances? This would reveal whether data volume or data quality is the binding constraint.
Applying the pipeline to the 480B models: Fine-tuning Qwen3-Coder-480B with AesCode-358K + GRPO-AR would test whether the pipeline provides additive benefits at scale or whether large models already saturate the achievable aesthetics.
Validation reward curves: Plotting held-out reward during training to check for over-optimization of the agentic reward signal.
Qualitative failure analysis: The case study (Figure 4) shows only successes. Showing cases where AesCoder fails — and whether failures are due to bad aesthetics, broken interactivity, or instruction misunderstanding — would provide a more complete picture of capability boundaries.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Excluded from the Main Efficiency Claim

The headline result — that AesCoder-4B surpasses GPT-4o and GPT-4.1 while matching 480B–685B parameter models — is achieved through a training pipeline whose computational cost is partially obscured by what the paper reports and what it omits.

The assumption: During GRPO-AR training, every generated output (8 per prompt, 64 prompts per batch, over ~250 steps) must pass through the full agentic reward framework: the Execution Agent runs HTMLHint, the Static Aesthetics Agent renders a full-page screenshot in headless Playwright and sends it to GPT-5 for chain-of-thought scoring, and the Interactive Aesthetics Agent launches a browser, plans interactions, executes them with GPT-4o-powered WebVoyager, and scores each attempt. The paper reports training time — approximately 7 days on 8×MI300 GPUs for the RL phase (Appendix E) — but does not disaggregate what fraction of this cost is model forward passes versus agent evaluation overhead. GPT-5 API calls for static scoring and GPT-4o calls for interactive evaluation are not included in the GPU-day accounting and represent an unmeasured external compute cost.

The consequence: A practitioner seeking to replicate this pipeline cannot estimate the true resource requirements. If agent evaluation dominates the training budget (which is plausible: rendering 512 webpages, sending 512 screenshots to GPT-5, and running GPT-4o-powered web agents with multiple interaction steps per output is substantial), then the approach may be economically infeasible for all but well-resourced labs with API budget for frontier models at scale. Additionally, the GPT-5 and GPT-4o API calls introduce an external dependency — the pipeline's reproducibility depends on continued access to specific proprietary model versions whose scoring behavior may change over time. The paper's training cost numbers (2 days SFT + 7 days RL on 8×MI300) are therefore a lower bound that excludes the evaluation infrastructure cost.

What evidence exists: The paper is transparent about the training configuration in Appendix E — 64 prompts per rollout, 8 responses per prompt, ~250 training steps — but the cost discussion is limited to "takes approximately 7 days on 1 nodes of 8xMI300 GPUs." There is no breakdown of time spent in model inference versus agent reward computation, no report of GPT-5 or GPT-4o API call volume, and no discussion of the latency per reward computation. Figure 6 shows reward curves over training steps, providing a temporal view of training dynamics, but the x-axis is in steps, not wall-clock time or compute cost. A reader cannot determine whether extending training to 500 or 1000 steps would be feasible.

Mitigation status: The paper does not address this limitation or propose ways to reduce evaluation cost (e.g., using cheaper models for static judging, reducing the number of interactions, or caching evaluations for repeated outputs). The Discussion section (4.3) acknowledges limitations of the interactive agent's reliability but not its cost. Future work on cheaper difficulty estimation or more efficient agent architectures would directly address this gap.

The Method Is Validated on a Single Task Family with Small Test Sets

All experimental evidence for the paper's claims comes from two benchmarks — PandasPlotBench and OpenDesign — which together cover exactly two visually-oriented coding domains: Python plotting and single-page HTML webpage design. The broader claim that the pipeline aligns models with "code aesthetics" writ large is not tested beyond these domains.

The assumption: The paper makes an implicit generalization claim by introducing "code aesthetics" as a broad concept and evaluating on representative tasks (Python plotting and webpage design). Section 3 states the authors "focus on domains where both the visual outcome and the implementation style matter" and select these two as representative. The paper does not claim universality, but the framing — "code aesthetics," "AesCoder," the OpenDesign benchmark — suggests a general capability.

The consequence: A practitioner working on code aesthetics in other domains — React component design, mobile UI generation (SwiftUI, Jetpack Compose), CSS animation, scientific figure generation beyond matplotlib/seaborn/plotly, interactive dashboard creation (Shiny, Streamlit), or game development beyond simple browser games — has no evidence that AesCode-358K's training signal or GRPO-AR's reward framework transfers. The Interactive Aesthetics Agent, in particular, is built on WebVoyager and operates in a browser environment; it cannot evaluate interactive aesthetics for non-web UI frameworks. The Static Aesthetics Agent evaluates screenshots, which works for any renderable output, but the GPT-5 prompt is tuned for webpage evaluation (referencing "HTML webpage," "full-page screenshot," "responsive design" in Appendix H.2). Adapting it to other visual domains would require prompt engineering whose effectiveness is untested.

What evidence exists: The paper reports results exclusively on PandasPlotBench (for plot generation) and OpenDesign (for webpage design) in Table 1. The OpenDesign benchmark's category breakdown (Table 4) shows strong skew toward General Website (60.9% of cases), with smaller representations for 3D Design (14.6%), Game Dev (13.6%), Data Visualization (4.8%), and UI Component (4.9%). Per-category results are not reported, so we cannot assess whether the pipeline works equally well across webpage types or is carried by the dominant General Website category. For PandasPlotBench, the paper reports aggregate metrics (error rate, average score, good rate) without breakdowns by plot type (bar, line, scatter, etc.), library (matplotlib vs. seaborn vs. plotly), or complexity level. The paper does not test on any other code aesthetics benchmark — because none existed prior to this work.

Mitigation status: The paper acknowledges this limitation implicitly through its focused scope — it doesn't claim universality, and the two-domain coverage (Python plotting + webpage design) is already substantially broader than prior work. The construction of OpenDesign as a new benchmark specifically for webpage aesthetics partially addresses the evaluation gap, but opens a new one: OpenDesign has only 840 cases, and the paper's own reliability analysis (Section 6.2) is based on 10 model rankings and 200 pairwise comparisons, which is reasonable for validation but small for a benchmark intended to support future research. The paper does not discuss extending to other domains or modalities as future work.

The Reward Signal Carries a Structural Circularity Risk from LLM-as-a-Judge Evaluation

Both the training reward and the primary evaluation benchmark use the same or closely related proprietary models as judges, creating a potential for the policy to optimize for judge-specific aesthetic preferences rather than universally human-preferred aesthetics.

The assumption: The Static Aesthetics Agent uses GPT-5 as the judge for both the training reward (providing 80% weight in the aggregated reward) and the OpenDesign benchmark evaluation. The Interactive Aesthetics Agent uses GPT-4o for both training and evaluation — the paper explicitly notes choosing GPT-4o "for cost considerations" (Section 4.3). This means the same model families (GPT-5 and GPT-4o) that define "good aesthetics" during training are also used to measure "good aesthetics" at evaluation time. The GRPO-AR training objective directly optimizes the policy to maximize these judges' scores.

The consequence: Any systematic biases, blind spots, or idiosyncratic preferences in GPT-5's aesthetic judgment become both the optimization target and the evaluation metric. If GPT-5 systematically overvalues certain design patterns (e.g., gradient backgrounds, specific color palettes, particular layout conventions), the policy will learn to exploit these biases, and the evaluation benchmark will confirm the resulting outputs as "high quality" — but a human user might disagree. The paper provides a partial defense through human evaluation (Appendix F), showing AesCoder achieves ~50% win rate against Claude Sonnet 4 and GPT-5 and >55% win rate against open-source baselines in human pairwise judgments. However, this human study uses only 100 test cases (out of 840 in OpenDesign) and 10 raters, and it evaluates only the final trained models. It does not assess whether intermediate training steps were optimizing for genuinely human-aligned aesthetics or for judge-exploiting patterns that happen to correlate with human preferences. Additionally, the human evaluation compares AesCoder against other models that were themselves not optimized for aesthetics — a more rigorous test would compare against models trained with a human-feedback-based reward signal to check whether the agentic reward converges to the same aesthetic optimum.

What evidence exists: Section 6.2 reports that OpenDesign's rankings have Spearman = 0.98 correlation with Design Arena and that GPT-human agreement (80.9%) exceeds human-human agreement (68.7%) on pairwise comparisons. While these are strong numbers, they characterize the benchmark's aggregate ranking alignment, not whether optimization against GPT-5's scores leads to the same endpoint as optimization against human preferences. The steadily increasing reward curves in Figure 6 (Appendix G) are consistent with either genuine improvement or judge over-optimization — without a held-out human evaluation at intermediate checkpoints, the two are indistinguishable. The paper does not report whether reward increases correspond to monotonic improvements in human preference or whether there is a point beyond which reward increases but human preference plateaus or declines (the classic reward over-optimization pattern).

The Interactive Aesthetics Agent introduces an additional circularity concern. The agent uses GPT-4o to both plan which interactions to test and judge whether interactions succeed or fail. If GPT-4o has systematic interaction patterns (e.g., always trying to click navigation links first, expecting certain visual feedback patterns), the policy may learn to generate webpages that satisfy GPT-4o's specific interaction style rather than being generally usable. A webpage that works perfectly for GPT-4o-WebVoyager might confuse a human user who approaches interactions differently. The paper's acknowledgment that "agent failures also partially reveal non-standard or sub-optimal aspects of webpage design" (Section 4.3) partially addresses this by reframing agent-judged usability as a meaningful signal, but does not eliminate the concern that the optimization target may be agent-specific rather than user-general.

Mitigation status: Partial. The human evaluation study provides some evidence that the final models produce human-preferred outputs, but it is too small (100 cases) and too coarse (only final-model pairwise comparisons) to fully characterize the alignment between optimization trajectory and human preference. The paper does not discuss the circularity risk explicitly, does not propose using a different judge model for evaluation than for training, and does not perform a held-out human evaluation at multiple training checkpoints. The strong benchmark alignment statistics (Spearman = 0.98) provide evidence for the benchmark's validity but not for the absence of over-optimization during training.

Interactive Evaluation Has a Known Under-Scoring Bias That Propagates into Training

The Interactive Aesthetics Agent produces systematically noisy reward signals because web agents are imperfect — they sometimes fail to interact correctly with functional webpages due to their own limitations rather than the webpage's faults. The paper acknowledges this explicitly but does not quantify its impact on training or propose corrections.

The assumption: The Interactive Aesthetics Agent assigns a binary score of 0 for any interaction where "the webpage is not changed or the change is not expected" (Appendix H.3). This scoring rule treats agent failures and webpage failures identically. The paper acknowledges in Section 4.3:

"Current web agents can handle most webpage operations, but may still struggle with certain corner cases, such as confusing webpage elements or being misled by irrelevant textual content. Such agent failures lead to a score of 0 in the corresponding iteration, since we assign a score of 1 only when the webpage responds correctly. This may cause the agent to make incorrect judgments, resulting in scores lower than the true values."

The consequence: The interactive reward signal r_interact (contributing 10% weight to the total reward) contains unmeasured downward bias. For any given training example, the true interactive functionality might be perfect (all interactions would work for a human), but the agent-reported score might be 1–2 points lower due to agent errors. More problematically, this bias is unlikely to be uniform — it probably correlates with webpage complexity, visual density, and interaction design patterns that happen to confuse the agent. The GRPO-AR training then receives a reward signal that penalizes certain valid design patterns not because they are bad for users, but because they are hard for GPT-4o-WebVoyager to navigate. Over many training steps, this could push the policy away from complex, agent-confusing designs even if those designs would serve human users well.

The magnitude of this bias is unknown. The paper does not report: (1) the overall success rate of the WebVoyager agent on human-designed webpages (a baseline for "what's the agent's ceiling?"), (2) the rate at which the agent reports failure for interactions that a human would consider successful (the false negative rate), or (3) whether the agent's success rate improves, degrades, or stays constant as the policy's generated webpages become more aesthetically sophisticated. If the policy learns to generate webpages that are both more beautiful and harder for the agent to evaluate (e.g., more visually complex layouts that confuse element detection), the interactive reward could paradoxically decrease even as true functionality improves.

What evidence exists: The paper discusses this limitation qualitatively in Section 4.3 but provides no quantitative measurement. The mitigation attempt — "when the GUI agent lists the interactive elements, we instruct it to prioritize them based on their importance" (Section E) — addresses which interactions get tested but not the accuracy of the success/failure judgments. The fact that the Interactive Aesthetics Agent received only 10% weight in the reward aggregation (vs. 80% for static aesthetics) may reflect the authors' awareness of this noise — giving a noisy signal low weight limits its capacity to distort training — but this also means the training signal for interactive functionality is weak relative to the static aesthetics signal, which could lead the model to underinvest in interactivity.

The interactive score improvements in Table 2 — from SFT-only (0.62/0.70 for 4B/7B) to full GRPO-AR (1.04/0.94) — demonstrate that the interactive reward does provide a learnable signal despite the noise. However, we cannot determine from these numbers whether the remaining gap to GPT-5's interactive score of 1.37 represents genuine interactive functionality limitations or agent evaluation failures on otherwise functional webpages.

Mitigation status: Minimal. The paper acknowledges the limitation in the Discussion subsection of Section 4.3 and suggests a partial defense (agent failures reveal genuine design issues), but provides no quantitative error analysis, no comparison of agent judgments against human interaction success rates, and no proposal for debiasing the interactive reward signal (e.g., using ensemble agents, incorporating confidence scores, or calibrating against human interaction baselines). This is flagged as an inherent limitation of current web agent technology rather than a solvable problem within the paper's scope.

The AesCode-358K Dataset Construction Relies on Frontier Proprietary Models, Limiting Reproducibility and Scalability

The supervised fine-tuning dataset that provides the foundation for all downstream performance is generated primarily by GPT-5, GPT-4o, and Qwen3-Coder-480B-A35B-Instruct-FP8 — models that are either proprietary and access-limited (GPT-5, GPT-4o) or extremely large and computationally expensive to run (Qwen3-Coder-480B with 480B total parameters). This creates a reproducibility bottleneck that is not addressed by the paper's open-source release of the resulting models.

The assumption: The paper's data construction pipeline (Section 3) requires:

GPT-4o to generate 19,000 seed keywords and 380,000 webpage design instructions (Section 3.2, Appendix B.1)
OpenAI text-embedding-3-large for semantic deduplication (Appendix B.2)
GPT-5 to score and select between candidate HTML generations (Section 3.2, "we asked GPT-5 to score the two outputs based on their rendered images")
Qwen3-Coder-480B-A35B-Instruct-FP8 to regenerate Python plot code and generate HTML candidates alongside GPT-5 (Sections 3.1, 3.2)
GPT-4o again to rewrite WebSight v0.2 descriptions for the RL training data (Section 5.2, Appendix H.6)

The final AesCode-358K dataset is obtained after "rigorous filtering" that discards data not meeting quality thresholds, but the filtering criteria are defined by these same frontier models' judgments.

The consequence: A researcher seeking to replicate, extend, or improve upon this work faces two barriers. First, access: GPT-5 and GPT-4o are proprietary models with API access that can change in availability, pricing, and behavior. If a future researcher cannot access GPT-5 (or if a future GPT-5 version produces systematically different quality judgments), the exact AesCode-358K dataset cannot be reproduced. The paper does not commit to releasing the dataset itself — the project page is referenced but the dataset availability is not stated in the paper.

Second, cost: generating 380K instruction-code pairs with frontier models, then scoring and filtering them, and then running embedding-based deduplication at scale, represents a substantial computational investment that the paper does not quantify. The dataset construction cost is not included in the reported training costs (2 days SFT + 7 days RL on 8×MI300 GPUs, Appendix E). For a research group without access to GPT-5 API credits or the compute budget to run Qwen3-Coder-480B for 158K Python regenerations and 200K HTML generations, building an equivalent dataset from scratch is infeasible.

What evidence exists: The paper describes the data construction pipeline in detail (Section 3, Appendices B and H) but provides no cost estimates, no discussion of dataset release plans, and no ablation testing whether a cheaper data construction pipeline (e.g., using only open-source models, using fewer generated instructions, or using weaker models for quality filtering) would produce comparable downstream performance. The AesCode-358K dataset's dependence on frontier models is presented as a quality guarantee — "we use [strong models] to regenerate the Python code" (Section 3.1) — without acknowledging the reproducibility implications.

Mitigation status: None. The paper does not address dataset reproducibility, does not specify whether AesCode-358K will be publicly released, and does not discuss what fraction of the performance gains are attributable to the specific frontier models used in data construction versus the training methodology itself. The release of trained model weights (AesCoder-4B and AesCoder-7B) provides a downstream artifact but does not enable independent verification or improvement of the data pipeline. Future work could address this by releasing the dataset, by constructing an equivalent dataset using only open-weight models to establish a reproducibility baseline, or by ablating the data construction model choices.

Interactive Aesthetics Evaluation Is Limited to 3 Interactions During Training, Constraining the Signal for Complex Webpages

The Interactive Aesthetics Agent is the paper's primary mechanism for ensuring webpages are not just beautiful but functional. However, during RL training, it tests only 3 interactions per webpage, creating a sparse reward signal that may miss critical functionality failures on complex, interaction-rich webpages.

The assumption: Section E states: "Given the currently low success rate of GUI agents, we limit the number of interactive elements to 3 during training." This constraint is motivated by reliability concerns — more interactions means more opportunities for agent failures that introduce noise into the reward. The agent is instructed to "prioritize [interactive elements] based on their importance," so the 3 tested interactions are meant to be the most critical ones.

The consequence: For simple webpages (a landing page with a navigation bar and a contact form), 3 interactions might cover the essential functionality. But for the webpage categories that most need interactive evaluation — Game Dev (13.6% of OpenDesign) where playability requires testing multiple controls and game states, 3D Design (14.6%) where object manipulation might require rotation, zoom, and multiple view controls, and Data Visualization (4.8%) where interactive charts may have filtering, zooming, tooltip, and drill-down interactions — 3 interactions is a severe undercount. A browser-based game might have 10+ interactive elements (movement controls, action buttons, menu navigation, game state toggles). Testing only 3 of them means the model receives positive reward as long as those 3 work, even if the remaining 7 are broken. The model could learn to produce webpages that are functionally polished on a small subset of carefully prioritized interactions while neglecting overall functional completeness.

Additionally, because the agent chooses which 3 interactions to test based on its own prioritization, the training signal is confounded by the agent's judgment of what is "important." If the agent systematically prioritizes surface-level interactions (clicking visible buttons) over deeper functionality (testing edge cases in form validation, verifying game state transitions), the reward signal will not penalize failures in those deeper layers.

What evidence exists: The paper reports interactive scores in Table 1 and Table 2. For AesCoder-4B, the interactive score is 1.04 — meaning that on average across the 840 OpenDesign test cases, roughly 1 interaction succeeded per webpage. For context, GPT-5 achieves 1.37, and the maximum possible score is not specified (it depends on how many interactions the agent plans per test case, which varies by webpage complexity). The gap between AesCoder-4B (1.04) and GPT-5 (1.37) could represent genuine functional inferiority, or it could reflect that GPT-5 as a stronger model generates webpages whose interactions are easier for the GPT-4o-based agent to evaluate successfully.

During training, the 3-interaction limit means the maximum possible training reward from interactivity is 3 × 0.1 = 0.3 (with weight w_interact = 0.1), compared to a maximum static aesthetics contribution of 100 × 0.8 = 80. The dynamic range of the interactive reward is two orders of magnitude smaller than the static reward, meaning the model's optimization is overwhelmingly driven by visual quality, with interactivity as a weak regularizer. This weight allocation may be pragmatically necessary given the noise in interactive evaluation, but it structurally limits how much the model can learn about functionality.

Mitigation status: The paper acknowledges the low success rate of GUI agents as the motivation for the 3-interaction limit and notes the prioritization strategy as mitigation. During evaluation (OpenDesign benchmark), the interaction limit appears to be higher or unspecified — the agent is not constrained to 3 interactions at test time, which is why interactive scores can exceed 3 (though in practice the highest reported score is GPT-5's 1.37, suggesting most webpages have few successfully tested interactions). The paper does not discuss how the training-time interaction limit affects the model's generalization to webpages requiring more extensive interaction testing, and does not ablate the limit (e.g., training with 1, 3, 5, or 10 interactions to measure the impact on interactive scores). This is fundamentally a limitation of current web agent technology rather than a design choice that could easily be changed — until web agents become more reliable, increasing the interaction count would introduce more noise than signal.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose an incremental improvement to existing code-generation reward methods — it argues that those methods are structurally incapable of addressing a whole category of tasks and introduces an entirely different reward paradigm. The magnitude of this shift is best understood as opening a new subfield rather than advancing an existing one.

Code aesthetics as a first-class capability axis. The field's prior consensus — implicit in every code benchmark from HumanEval to SWE-bench — was that "good code" means code that compiles, runs, and produces the correct output. Visual quality was either a human post-processing step or a non-goal for LLM evaluation. This paper demonstrates that this consensus leaves a gap large enough to drive through: a model that scores well on traditional code metrics (Qwen2.5-Coder-7B-Instruct is a strong coder by standard measures) achieves only 46.27/100 on static webpage aesthetics and 0.38 on interactive functionality (Table 1). The paper's central conceptual contribution is carving out code aesthetics as a capability that is orthogonal to code correctness — you can have one without the other, and standard training pipelines optimize only for correctness.

This reframing has downstream consequences for how the field thinks about coding LLM evaluation. A benchmark suite that lacks visual-output tasks is not merely incomplete — it is systematically blind to a capability that end-users demonstrably care about. The OpenDesign benchmark's strong alignment with Design Arena human rankings (Spearman = 0.98, Section 6.2) provides evidence that aesthetic quality is not a vague, subjective preference but a measurable dimension of code-generation performance that large-scale human judgment converges on. The implication is that future coding benchmarks should include visually-oriented tasks by default, just as they currently include unit-test-based tasks.

The multi-agent decomposition as an evaluation paradigm. The paper's most transferable methodological insight is that complex output quality cannot be assessed from a single modality when the output itself spans multiple modalities. Prior work on reward modeling for code implicitly assumed that a single evaluator — whether a unit test suite (RLTF), a learned reward model (CodeRL), or a human annotator (RLHF) — could capture everything that matters. The paper's three-agent decomposition (textual linting, visual judging, interactive testing) demonstrates that this assumption fails catastrophically for visually-oriented tasks: the ablation in Table 2 shows that replacing the multi-agent system with a single text-based judge scoring the same dimensions reduces interactive scores from 1.04 to 0.71 (4B model) and static totals by 1–2 points per sub-dimension.

This is not just a finding about code — it is a general design principle for any task where output quality has dimensions that live in different modalities. Consider: a model that generates UI code for mobile apps (SwiftUI, Jetpack Compose), a model that produces CAD scripts for mechanical design, a model that writes shader code for visual effects. In each case, the "correctness" of the output is partly textual (does it compile?), partly visual (does it look right?), and partly interactive (does it behave correctly under user manipulation?). The paper's agentic decomposition provides a template: identify the irreducible evaluation modalities, build specialized evaluators for each, and combine their signals with task-appropriate weighting. This paradigm is likely to outlive the specific agents used in this paper.

The training-inference asymmetry at scale. Perhaps the most provocative implication of the results is what they say about model scale versus training signal quality. AesCoder-4B matches or exceeds open-source models with 480B–685B parameters on both static and interactive aesthetics (Table 1). The 480B model (Qwen3-Coder-480B-A35B) scores 79.90 static, 0.70 interactive; the 4B AesCoder scores 81.92 and 1.04. This is a ~120× reduction in parameter count with superior performance on the target task.

The standard interpretation of such results is "task-specific fine-tuning beats general capability." But the deeper implication is about what information the larger model is missing, not what capability it lacks. The 480B model has vastly more parameters, was trained on vastly more data, and undoubtedly possesses stronger general reasoning capability. What it lacks is any training signal that connects its code outputs to their visual consequences. The AesCode-358K dataset provides 358K examples of that connection; the GRPO-AR training provides online feedback that explicitly evaluates visual and interactive quality. The 480B model has never been told that overlapping chart legends are bad, because that information doesn't exist in text-only code training data, no matter how much of it you have.

This reframes the scale-versus-data debate for coding LLMs. It is not that scale doesn't matter — it's that for capabilities that require cross-modal feedback, scale in the text modality alone is insufficient. You can make the model arbitrarily large and train it on arbitrarily many text tokens; it will never learn that plt.legend(loc='upper right') sometimes produces an ugly chart unless it receives visual feedback. The implication is that post-training pipelines that provide modality-appropriate feedback (visual inspection, interactive testing) may be necessary for visually-oriented coding tasks, not merely helpful optimizations. Larger pretraining alone cannot substitute for them.

What research directions become more attractive. The paper makes several research directions newly salient:

Verifier robustness for visual tasks becomes a central concern. Just as the reference example paper identified verifier over-optimization as the bottleneck for math reasoning, this paper's dependence on GPT-5 and GPT-4o as judges raises the question: how reliable are multimodal LLMs as aesthetic evaluators, and how does optimization pressure affect that reliability? The paper's training curves (Figure 6) show steadily increasing reward, but without a held-out evaluation at intermediate checkpoints, we don't know whether the model is genuinely improving or learning to exploit judge-specific preferences.
Interactive evaluation as infrastructure becomes a research investment priority. The Interactive Aesthetics Agent's acknowledged limitations — under-scoring bias, 3-interaction limit during training, GPT-4o dependency — are not fundamental; they reflect the current immaturity of web agent technology. As web agents improve (better element detection, more robust interaction planning, lower latency), the interactive reward signal will become richer and more reliable, directly improving the quality of models trained with agentic reward feedback. Investing in better web agents is now also an investment in better code-generation training pipelines.
Multi-modal reward decomposition becomes a design pattern to replicate across domains. The paper's success suggests that other tasks with cross-modal output quality — UI code generation, data visualization, animation authoring, document layout — could benefit from analogous multi-agent evaluation frameworks.

What research directions become less attractive. The paper also implicitly argues against certain approaches:

Single-modality reward models for visually-oriented code are unlikely to close the gap. The ablation (Table 2) shows that text-based scoring of HTML source code underperforms the full agentic system by substantial margins. While text-based rewards are cheaper and simpler, they lack access to the visual and interactive information that defines aesthetic quality.
Scale-alone approaches — the strategy of simply training larger models on more code data and hoping visual aesthetics emerge — face a structural barrier. The paper's 480B-parameter comparison model, despite its enormous scale, underperforms the 4B AesCoder on aesthetics. Unless future large-scale pretraining incorporates rendered visual feedback (which is computationally expensive at pretraining scale), scale alone will not solve the code aesthetics problem.

Follow-Up Research This Work Enables

Per-category aesthetic competence profiling. The OpenDesign benchmark contains five categories with imbalanced representation (General Website 60.9%, 3D Design 14.6%, Game Dev 13.6%, Data Visualization 4.8%, UI Component 4.9%), but the paper reports only aggregate scores. A natural follow-up would measure AesCoder-4B and AesCoder-7B separately on each category, testing whether the training pipeline produces uniform aesthetic competence or whether certain categories (e.g., Game Dev, where interactive testing is most critical) show weaker improvement. The hypothesis: interactive scores should be highest for Game Dev and 3D Design if the Interactive Aesthetics Agent is providing meaningful signal, and static scores should be highest for General Website and UI Component where visual polish dominates. If instead all categories show similar gains, it suggests the training signal is dominated by static aesthetics (80% weight) regardless of category, and the interactive agent is not providing category-differentiated feedback. This experiment requires only re-running the existing evaluation with per-category score disaggregation — no new data collection needed.

Scaling the number of training interactions and measuring the interactive-static tradeoff surface. The paper limits interactive testing to 3 interactions during training due to web agent reliability concerns (Section E), with weights w_static = 0.8 and w_interact = 0.1. A systematic sweep over interaction counts (1, 3, 5, 10, 20) and reward weights (e.g., w_static/w_interact ratios of 0.9/0.0, 0.7/0.2, 0.5/0.4, 0.3/0.6) would map the Pareto frontier between static beauty and interactive functionality. The prediction: as interactive weight increases and interaction count grows, static aesthetics scores should decrease (the model sacrifices visual polish for functionality) while interactive scores should increase, but only up to the point where agent reliability saturates — beyond that, additional interactions add noise that degrades both dimensions. This experiment would reveal whether the paper's chosen weights represent a genuine sweet spot or an arbitrary point on a broad tradeoff curve, and would provide practical guidance for practitioners who need to prioritize static vs. interactive quality for their specific deployment context.

Cross-judge robustness: training with one judge family, evaluating with another. The paper uses GPT-5 and GPT-4o for both training rewards and OpenDesign evaluation, creating a potential circularity. A critical stress-test: train a model using the full GRPO-AR pipeline but with Claude (Sonnet 4 or Opus) as the static aesthetics judge instead of GPT-5, then evaluate on OpenDesign which still uses GPT-5 as judge. If the resulting model scores comparably to AesCoder-4B, the training signal is robust — the model learns genuinely universal aesthetic principles, not GPT-5-specific preferences. If performance drops substantially, the pipeline is overfitting to judge-specific biases, and the reported results overstate alignment with human aesthetics. This experiment is expensive (requiring retraining) but definitive. A cheaper variant: evaluate AesCoder-4B's outputs using Claude as judge instead of GPT-5, and measure the correlation between GPT-5 scores and Claude scores. If correlation is high (Spearman > 0.9), the circularity concern is partially mitigated; if low, it is acute.

Interactive agent calibration against human interaction success rates. The paper acknowledges that the Interactive Aesthetics Agent under-scores webpages due to its own failures but provides no quantification. A calibration study would: (1) sample 100 webpages from AesCoder-4B outputs and 100 from human-designed pages, (2) have the GPT-4o-WebVoyager agent test interactions on each, recording claimed success/failure, (3) have human testers perform the same interactions on the same webpages, recording actual success/failure. This yields a confusion matrix: agent false positive rate (agent claims success, human fails), agent false negative rate (agent claims failure, human succeeds), and overall agreement. The paper's implicit assumption that "agent failures also partially reveal non-standard or sub-optimal aspects of webpage design" predicts that false positives should be rare (if the agent succeeds, the interaction almost certainly works for humans) and false negatives should be non-trivial but concentrated on visually complex pages. Calibrating these rates would allow future work to debias the interactive reward signal — for example, by applying a correction factor based on webpage visual complexity, or by using ensemble agents where majority vote determines success/failure.

Extension to multi-file, multi-page web applications. AesCode-358K trains on single-file HTML pages; the Interactive Aesthetics Agent assumes single-page offline operation. Real web development involves multi-page sites with navigation between pages, shared CSS/JS resources, and server-dependent functionality. A natural extension would construct a dataset of multi-page web applications (e.g., a simple e-commerce site with product listing, detail, and checkout pages; a dashboard with multiple linked views; a blog with article listing and individual article pages) and extend the Interactive Aesthetics Agent to navigate across pages, testing cross-page links, shared state (e.g., a shopping cart that persists across pages), and consistent styling. The challenge: multi-page evaluation requires the agent to maintain state across page transitions and to reason about cross-page functional coherence, which current web agents struggle with. Success would demonstrate that agentic reward feedback scales beyond toy single-page examples to realistic web development tasks.

Distillation of aesthetic judgment into a smaller, faster reward model. The paper's training pipeline depends on API calls to GPT-5 and GPT-4o for every reward computation, creating a cost and latency bottleneck. A practical follow-up: collect a dataset of (HTML code, screenshot, GPT-5 static score, GPT-4o interactive score) tuples from the GRPO-AR training runs, then train a smaller multimodal model (e.g., a fine-tuned LLaVA or Qwen2-VL variant) to predict the agentic reward directly from the screenshot and instruction. If the distilled reward model achieves high correlation with GPT-5 scores (Pearson > 0.9) and the web agent's interaction judgments (accuracy > 85%), it could replace the expensive multi-agent system during training, dramatically reducing cost and enabling longer training runs, larger batch sizes, and experiments that are currently infeasible. The key question: does the distilled model capture the nuanced aesthetic judgment that GPT-5 provides, or does it regress to a coarser signal that produces worse downstream models? This is testable by running GRPO-AR with the distilled reward model and comparing final OpenDesign scores against the full agentic reward baseline.

Practical Applications and Downstream Use Cases

Small-model deployment for web development assistants in resource-constrained environments. The paper's headline result — AesCoder-4B matches or exceeds 480B–685B models on webpage aesthetics — has immediate deployment implications. A 4B-parameter model can run on consumer hardware (a laptop with 8GB VRAM, or even a high-end phone with quantization), while a 480B model requires datacenter-grade infrastructure. For a web development tool targeting individual developers or small teams (e.g., an IDE plugin that generates webpage prototypes from natural language descriptions), AesCoder-4B provides GPT-4o-competitive aesthetic quality at a fraction of the inference cost and without API dependency. The specific numbers: on OpenDesign, AesCoder-4B scores 81.92 static aesthetics vs. GPT-4o's 48.08 — a 70% improvement in visual quality — while running locally. The same deployment logic applies to any setting where users need visually-oriented code generation but cannot depend on cloud API access: offline development environments, privacy-sensitive enterprise settings where code cannot leave the local machine, and educational contexts where students need instant, free code generation feedback.

Automated design iteration for data visualization pipelines. Data scientists and analysts spend significant time iterating on plot aesthetics — adjusting colors, fixing legend overlaps, tuning font sizes, repositioning annotations — after getting the basic visualization working. The PandasPlotBench results show that AesCoder-4B reduces the error rate from 0.13 (base Qwen3-4B) to 0.09 and improves the good rate (scores >75) from 0.55 to 0.63. In practice, this means that when an analyst asks "create a clear bar chart comparing quarterly revenue across regions," the model is substantially more likely to produce a plot that is both correct and visually polished on the first attempt, reducing the need for manual cleanup. The 0.09 error rate also means that 91% of generated plotting code actually executes — a critical reliability threshold for integration into automated reporting pipelines where non-executable code breaks the workflow. AesCoder could be deployed as a drop-in replacement for the plotting-code-generation step in existing data analysis tools, improving both the reliability (lower error rate) and quality (higher good rate) of automatically generated visualizations.

Pre-deployment quality screening for LLM-generated web content. Many organizations are exploring LLM-generated web content — landing pages, product descriptions with layout, interactive documentation — but face a quality assurance bottleneck: human review of every generated page is expensive, and automated checks based on code validity alone miss visual and interactive defects. The agentic reward framework itself can serve as an automated QA pipeline independent of training. For any model generating HTML, the three agents can evaluate executability, static aesthetics, and interactive functionality, producing scores that correlate with human judgment (Spearman = 0.98 with Design Arena, Section 6.2). A deployment could set thresholds (e.g., static score > 70, interactive score > 1) and auto-approve pages that exceed them while flagging borderline cases for human review. The paper's 80.9% GPT-human agreement rate on pairwise comparisons (vs. 68.7% human-human agreement) suggests that the agentic evaluation is actually more consistent than individual human raters, making it suitable as a first-pass filter that reduces — though not eliminates — the need for human QA. The cost, however, is non-trivial: GPT-5 API calls for static scoring and GPT-4o-powered web agent runs for interactive testing. For high-stakes content (e.g., customer-facing product pages where a broken checkout flow has revenue impact), this cost may be justified; for high-volume, low-stakes content, a distilled, cheaper evaluation model would be needed.

When to Prefer This Method

The paper does not explicitly position GRPO-AR against a well-defined set of alternatives with a clear decision framework — it compares against broad baselines (DPO, RFT, text-only reward models) but doesn't articulate a conditional "use X when Y, use Z when W" rule. The comparisons in Table 2 show that GRPO-AR outperforms DPO and RFT on this specific task, but the paper doesn't characterize when the gap would widen or narrow, nor does it claim GRPO-AR is universally preferable over these methods for other tasks. The reward weighting (0.1/0.8/0.1) is presented as a fixed choice rather than a tunable parameter that practitioners would adjust for different deployment priorities.

For these reasons, I do not include a prescriptive decision matrix — it would impose a tradeoff framework the paper does not develop, and the necessary ablation evidence (sensitivity to reward weights, per-category performance, cross-judge robustness) is not provided. A practitioner deciding whether to adopt GRPO-AR should weigh the evidence in Table 2 (consistent improvement over SFT, DPO, RFT, and text-only reward for this task) against the limitations discussed in Section 6, particularly the cost and reproducibility concerns around the proprietary judge models and the unquantified interactive evaluation noise.