AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation

🎯 Pitch

Able judges built on weaker models can beat much stronger LLM-only judges simply by fetching evidence from the environment—especially in GUI tasks like PowerPoint, where accuracy leaps by nearly 25 points. Yet the best agent-judge still leaves a quarter of verdicts wrong, exposing just how hard it is to verify real-world agent behavior reliably.

1. Executive Summary

This paper introduces AJ-Bench, the first comprehensive benchmark for systematically evaluating Agent-as-a-Judge systems—verifiers that actively interact with environments and tools to acquire evidence beyond static trajectories—across three domains (search, data systems, and graphical user interfaces) comprising 155 tasks and 516 annotated trajectories. Equipping judge agents with tool use and environment access consistently outperforms LLM-as-a-Judge baselines, achieving an average improvement of 0.13 in F1 score, with the agent-based approach built on a weaker model (gpt-5-mini-low) matching or exceeding the performance of substantially stronger LLM-only judges. The strongest agentic configuration (deepseek-v3.2 with tools) reaches an overall F1 of 77.34, with particularly dramatic gains in the GUI domain—improvements of 24.76, 8.87, and 9.59 points on PPT, Word, and Excel respectively—establishing that environment-aware verification provides substantial leverage, though the absolute performance remains unsaturated at 0.72 average F1, leaving substantial room for improvement particularly on tasks requiring precise state verification and multi-step evidence gathering.

2. Context and Motivation

The Core Problem: How Do We Verify Agent Behavior at Scale?

The fundamental question this paper tackles is deceptively simple: as we deploy LLM-based agents in increasingly complex, open-ended environments, how do we reliably determine whether they succeeded or failed? This matters because the field is entering a phase where reinforcement learning (RL) is being used at scale to train agents—the paper specifically cites work like Agent-R1 (Cheng et al., 2025) and broader RL-for-agents efforts (Chen et al., 2025). But RL fundamentally requires a reward signal: something must evaluate the agent's behavior and say "this was good" or "this was bad." Without reliable verification, scaling RL for agents hits a wall.

The paper frames this in stark terms right in the introduction:

"As RL computation continues to scale, a fundamental challenge emerges: how to verify agent behaviors in novel environments at scale."

This isn't merely a theoretical concern. Consider what happens in practice when deploying agents on real tasks—searching the web for verifiable information, manipulating files in a database system, or performing multi-step operations in a GUI application like PowerPoint. These environments produce complex, long-horizon trajectories where success depends on numerous intermediate decisions. A single wrong click, an incorrectly parsed file, or an unverified web source can mean the difference between task completion and failure. If we cannot verify outcomes, we cannot improve agents through RL, nor can we trust their outputs in production.

Three Failure Modes of Existing Verification Approaches

The paper identifies two dominant paradigms for verification, both of which have critical limitations when applied to agent behavior in realistic environments.

1. Rule-Based Verification: Brittle and Domain-Specific

The most common approach in current RL-for-agents work is rule-based verification (Shao et al., 2024; Mroueh, 2025; Wei et al., 2025). The idea is straightforward: define explicit rules that check whether the agent's trajectory matches expected behavior. For example, in a code generation task, you might check whether the output compiles and passes unit tests. In a GUI task, you might check whether a specific file was created or a particular UI element now exists.

The paper acknowledges that this works well "for narrowly scoped tasks." If you're training an agent to solve math problems, checking whether the final answer matches a ground-truth numeric value is a perfectly reliable verifier. But the problem is exactly the one the paper's epigraph highlights:

"An intelligent system cannot be evaluated independently of the environment in which it operates."

The moment you move beyond closed-form tasks to "complex, realistic settings" like "scientific hypothesis verification or essay-level fact checking," handcrafted rules break down. You cannot write a rule that checks whether a web search agent correctly identified the release date of a technical report—the correct answer is not known in advance, and the evidence needed to verify it lives in the external environment (web pages, databases, API responses), not in the task specification. Rule-based verifiers fundamentally assume the evaluator already knows the answer; in open-ended environments, this assumption is false.

Moreover, rule-based approaches require careful engineering for each new task domain. A verifier that works for file system operations won't transfer to GUI interactions, and a verifier for web search won't handle database queries. This brittleness means that as the diversity of agent tasks grows, the cost and complexity of building verifiers grows proportionally—an unsustainable trajectory.

2. LLM-as-a-Judge: Grounded Only in Surface Text

The second paradigm, LLM-as-a-Judge (Zheng et al., 2023; Li et al., 2025; Gu et al., 2025), attempts to solve the scalability problem by replacing handcrafted rules with a language model that evaluates the agent's output. Rather than writing explicit verification logic, you prompt an LLM to read the task description and the agent's trajectory, then judge whether the task was completed successfully.

This approach has been studied extensively. The paper surveys a range of LLM-as-a-Judge benchmarks in Table 1: RewardBench (Lambert et al., 2025) evaluates reward models across safety, dialogue, and reasoning; RM-Bench (Liu et al., 2025b) focuses on subtlety and stylistic differences in judgments; JudgeBench (Tan et al., 2025) targets reasoning evaluation; and AgentRewardBench (Men et al., 2025) extends this to agent trajectories specifically. These benchmarks have driven progress in making LLM judges better aligned with human preferences.

However, the paper identifies a fundamental limitation that all these approaches share:

"their judgements are ultimately grounded in surface-level textual signals."

What does this mean concretely? An LLM-as-a-Judge sees only the text of the agent's trajectory—the sequence of actions taken and the observations received. It cannot independently verify whether those observations are accurate. It cannot click on a UI element to check its state. It cannot navigate to a cited URL and confirm that the information the agent claims to have found is actually there. It cannot examine the file system to see whether a file was actually created with the correct contents.

Consider Figure 1's example: an agent is asked "What is the exact release date of the most recent version of the LongCat-Flash Technical Report as of December 2025?" and responds "2025-08-09." An LLM-as-a-Judge, seeing only this query-response pair, can only say: "Without being able to confirm the actual release date... I cannot definitively verify if August 9, 2025 is correct." It has no way to look up the ground truth. In contrast, an Agent-as-a-Judge can call a browser tool, navigate to the arXiv page (https://arxiv.org/abs/2509.01322), and observe that the actual release date was "19 Sep 2025"—definitively identifying the response as incorrect.

The paper is not dismissive of LLM-as-a-Judge—it surveys substantial prior work and uses LLM-as-a-Judge as the primary baseline for its experiments. But it argues that the surface-level nature of LLM judgments is an architectural limitation, not a temporary one. No amount of scaling or prompt engineering can give an LLM access to ground-truth information that simply isn't present in the input text.

3. The Gap: No Systematic Evaluation of Agent-Based Verification Exists

This brings us to the paper's primary diagnosis. The natural progression beyond LLM-as-a-Judge is to endow the verifier with agency—give it access to the same tools and environments the original agent used, and let it actively gather evidence to support or refute its judgment. The paper calls this paradigm Agent-as-a-Judge.

The concept isn't entirely new. The paper credits Zhuge et al. (2025) with initially introducing the term, and notes several recent works that integrate tool use into verification: Themis (Li et al., 2024) for tool-augmented reward modeling, TIR-Judge (Xu et al., 2025) for tool-integrated reasoning, VerifiAgent (Han et al., 2025) for unified verification, and Agentic Reward Modeling (Peng et al., 2025) for combining human preferences with verifiable signals. Benchmarks like Mind2Web2 (Gou et al., 2025), GAIA2 (Andrews et al., 2025), and RealDevWorld (Bian et al., 2025) demonstrate the increasing importance of agentic verifiers for agentic task evaluation.

But here is the critical gap the paper identifies: despite this conceptual momentum, there exists no comprehensive benchmark for evaluating Agent-as-a-Judge systems. The existing work is fragmented. Zhuge et al. (2025) examines Agent-as-a-Judge only on "small-scale datasets and narrow domains such as code verification." The paper explicitly states:

"these analyses are largely confined to small-scale datasets and narrow domains such as code verification, and therefore cannot offer a comprehensive assessment of Agent-as-a-Judge capability in open-ended settings."

Even more importantly, existing benchmarks "fail to capture the more fundamental challenges faced by judge agents, including deciding when interaction is necessary, how to leverage tools effectively, and what constitutes sufficient and verifiable evidence for reliable judgement in open-ended environments." In other words, prior work answers "can an agent verify simple tasks?" but not "what verification capabilities do judge agents need, and how do we systematically measure them?"

Table 1 makes this gap explicit through a structured comparison. Existing judge-evaluation benchmarks (RewardBench, RM-Bench, JudgeBench, AgentRewardBench) evaluate LLM-as-a-Judge but provide no environment access and no agentic interaction. DevAI (Zhuge et al., 2025) offers Agent-as-a-Judge evaluation but is limited to a single domain. AJ-Bench is the first to combine all three properties: multi-domain coverage, environment awareness, and agentic interaction.

Why This Gap Matters

The paper's motivation isn't merely academic—there are concrete practical consequences to the lack of systematic Agent-as-a-Judge evaluation:

1. RL for agents needs reliable verifiers. The paper cites the trend toward training agents with RL (Chen et al., 2025; Cheng et al., 2025). RL scales with the quality and reliability of the reward signal. If verifiers are unreliable (as LLM-as-a-Judge is on tasks requiring external evidence), the RL signal becomes noisy and the resulting agents will be suboptimal or even counterproductive. Understanding how well agent-based verifiers work, and under what conditions, is a prerequisite for deploying them in RL pipelines.

2. Agent deployment needs trust. If agents are going to operate in high-stakes environments—handling financial data, modifying critical files, making information-retrieval decisions that downstream systems depend on—we need to trust that our verification mechanisms work. An LLM-as-a-Judge that cannot independently confirm facts can only provide a surface-level plausibility judgment, which is insufficient for auditing agent behavior at scale.

3. The verification capability itself is poorly understood. The paper emphasizes that judge agents face challenges that task-solving agents don't: "deciding when interaction is necessary, how to leverage tools effectively, and what constitutes sufficient and verifiable evidence." These meta-cognitive capabilities—knowing when you need more information, choosing the right tools to get it, and recognizing when you have enough evidence to make a confident judgment—are distinct from the planning and execution skills that existing agent benchmarks measure. Without a dedicated benchmark, we cannot measure or improve these capabilities.

How This Paper Positions Itself

The paper positions its contribution not as proposing a new verification method, but as building the infrastructure for systematic evaluation of an emerging paradigm. The introduction frames this clearly:

"In this work, we move toward a systematic evaluation of Agent-as-a-Judge as a distinct and general capability."

This is an important distinction. The paper does not claim to have solved Agent-as-a-Judge; rather, it claims to have built the first benchmark that makes it possible to measure progress on Agent-as-a-Judge. The benchmark is designed to assess three specific judging capabilities that the paper argues are fundamental:

Information acquisition via external search — can the judge agent navigate the web, find relevant sources, and verify factual claims against external evidence?
State verification through tool-assisted interaction — can the judge agent inspect the environment state (file systems, databases, GUI applications) to confirm that expected changes have occurred?
Process verification by inspecting critical actions and execution steps — can the judge agent examine intermediate steps of a trajectory to determine whether the sequence of actions was logically sound, even when the final state alone might be ambiguous?

The paper also positions itself as enabling a controlled comparison between paradigms. By evaluating the same tasks with the same models under both LLM-as-a-Judge and Agent-as-a-Judge settings, AJ-Bench provides experimental evidence for the claim that tool use and environment access substantially improve verification. This is not just a conceptual argument—it's an empirical one that the paper tests systematically across three domains and multiple model families.

Furthermore, the paper explicitly connects to the broader trend of agentic verification in real-world settings. It cites Mind2Web2, GAIA2, and RealDevWorld as evidence that the research community is already building systems that require agent-based verification, and positions AJ-Bench as filling the evaluation gap that these systems expose. The benchmark is designed to be practical: it uses real environments (live web pages, executable file systems, AWS-hosted GUI instances) rather than simulated or simplified ones, and it evaluates judges on real agent trajectories collected from diverse model families, making the results ecologically valid.

The Stakes

The paper's positioning implies a larger arc for the field. If Agent-as-a-Judge proves to be a reliable verification paradigm, it enables a virtuous cycle: train agents with RL using agent-based verifiers as reward models → deploy the improved agents → verify their behavior with the same agent-based verifiers → use the verification feedback to further improve training. This is essentially the self-improvement loop that the broader LLM-as-a-Judge literature has envisioned, but with a crucial upgrade: the verification is grounded in environmental evidence rather than surface-level text patterns.

Conversely, if Agent-as-a-Judge proves unreliable or insufficient, the field needs to confront that fact head-on and develop alternative approaches. Either way, the paper argues, we need systematic benchmarks to make that determination. AJ-Bench is offered as that foundational evaluation platform.

3. Technical Approach

3.1 Reader Orientation

AJ-Bench is a benchmark and evaluation framework, not a new model or algorithm. What is being built is a standardized testbed where judge agents—LLMs equipped with tools and environment access—evaluate the correctness of task-solving agent trajectories by actively gathering evidence from live environments, and their judgments are scored against ground-truth labels. The problem it solves is that existing verification benchmarks either lack environment interaction (LLM-as-a-Judge evaluations) or lack multi-domain coverage and systematic difficulty characterization (existing Agent-as-a-Judge work), making it impossible to measure how well judge agents acquire evidence, verify states, and inspect execution processes. The "shape" of the solution is: define tasks requiring specific verification capabilities → collect diverse agent trajectories with known ground-truth outcomes → provide judge agents with tool access to live environments → measure how accurately their judgments match ground truth, producing an F1 score that captures both precision and recall of correct PASS/FAIL decisions.

3.2 Big-Picture Architecture (Diagram in Words)

The AJ-Bench system has five major components:

Task Definitions (155 tasks across 3 domains) — each task specifies a goal that a task-solving agent attempted, drawn from Search (web-based information finding), DS (file system and database manipulation), and GUI (PowerPoint, Word, Excel operations). Tasks are selected to exercise distinct verification challenges.
Trajectories (516 annotated sequences) — each trajectory is a recorded sequence of actions that some task-solving agent took on a task, paired with a binary ground-truth label (PASS=1 or FAIL=0) determined through human annotation, model-based majority voting, or verifier scripts with manual validation.
Environment Replay Infrastructure — for DS and GUI domains, the environment is reconstructed to its final state by replaying the trajectory's action sequence in a fresh, isolated instance (local execution for DS, AWS instances for GUI). For Search, the evidence lives on the live web, so no reconstruction is needed—the judge directly queries the external environment.
Judge Agent (LLM + Tools) — an LLM (e.g., gpt-5-mini-low, deepseek-v3.2) equipped with domain-specific tools (web browsing via Playwright for Search, file system and database inspection commands for DS, mouse/keyboard interactions and screenshot/accessibility tree inspection for GUI) that can interact with the live or replayed environment to gather evidence about whether the original trajectory succeeded.
Evaluation Metrics (F1 Score) — the judge agent produces binary judgments (PASS/FAIL) for each trajectory (or, in Search, for each extracted single-item claim within a trajectory). These are compared against ground-truth labels, and the F1 score (harmonic mean of precision and recall) is computed to capture both the judge's ability to correctly identify successes and correctly identify failures.

Information flows as follows: a task definition and labeled trajectory enter the system → the environment is initialized (replayed to final state for DS/GUI, or left as live web for Search) → the judge agent is given the task description, the trajectory summary, and tool access → the judge agent interacts with the environment, calling tools to inspect states, navigate web pages, or examine files → the judge produces a final binary verdict → the verdict is compared to the ground-truth label → F1 is aggregated across all trajectories and tasks.

3.3 Roadmap for the Deep Dive

First, the task design principles (§3.4.1), because the whole benchmark's validity depends on tasks that genuinely require environment interaction and span diverse verification challenges. Understanding how tasks were selected, filtered, and categorized tells us what capabilities are being tested.
Second, trajectory collection and labeling (§3.4.2), because the benchmark's ground-truth labels define what the judge agents are being evaluated against. We need to understand how positive and negative examples were generated, how labels were assigned, and what steps were taken to ensure label reliability across domains with fundamentally different verification characteristics.
Third, the environment replay architecture (§3.4.3), because it's the enabling infrastructure that makes interactive evaluation possible. Without understanding how environments are reconstructed and made queryable, we can't reason about what evidence is available to judge agents.
Fourth, the judge agent implementation (§3.4.4), covering the MCPMark framework, the tool sets available in each domain, the prompt design, and the interaction protocol—since this is what transforms a base LLM into an Agent-as-a-Judge.
Fifth, the evaluation protocol and metrics (§3.4.5), explaining how judgments are extracted, how F1 is computed differently across domains (item-level for Search, trajectory-level for DS and GUI), and the cross-run averaging procedure used to ensure statistical reliability.
Sixth, the comparative baselines (§3.4.6), covering how LLM-as-a-Judge is implemented (same models, same tasks, but no tool access) to enable the controlled comparison that is the paper's central empirical claim.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark construction and empirical evaluation paper whose core idea is that systematic measurement of Agent-as-a-Judge requires tasks spanning diverse verification challenges, trajectories with reliable ground-truth labels, and live environment access that enables evidence-gathering beyond surface-level text. The paper does not propose a new judge architecture; it proposes a standardized evaluation protocol and uses it to demonstrate both the promise (consistent improvement over LLM-as-a-Judge) and the limitations (unsaturated absolute performance) of current agent-based verification.

3.4.1 Task Design: Ensuring Tasks Genuinely Require Environmental Interaction

The paper's first design challenge is ensuring that the benchmark tasks actually test what they claim to test—the ability to gather evidence from an environment—rather than being solvable from the trajectory text alone. If a task's correctness could be determined by reading the trajectory without any tool calls, then giving the judge agent tools provides no genuine advantage, and performance differences between LLM-as-a-Judge and Agent-as-a-Judge would reflect noise rather than capability.

The authors address this through careful task selection and filtering across three domains, each chosen for distinct verification requirements.

Search Domain: Two Complementary Information-Seeking Paradigms

The Search domain draws from two existing benchmarks that represent fundamentally different types of information retrieval challenges.

Mind2Web2 (Gou et al., 2025) provides "deep search" tasks requiring multi-hop reasoning. These are not simple "look up a fact" queries; they require the agent to navigate through multiple web pages, synthesize information across sources, and apply reasoning to combine what it finds. The paper characterizes these as tasks where answers are "non-fixed or hard-to-exhaustively-retrieve"—meaning there is no single authoritative page that contains the answer, and verification requires checking multiple sources and confirming consistency.

The task curation for Mind2Web2 follows a specific human-in-the-loop pipeline (detailed in Appendix A.1). Tasks are first categorized into three groups: ground_truth (fixed, well-defined answers), no_ground_truth (tasks where multiple valid solutions exist and the task only requires returning a subset—e.g., "find three or five items"), and time_sensitive (tasks involving shopping, travel, or other rapidly-changing content where URLs, prices, or ratings become stale). The filtering logic is explicitly motivated by two considerations: time-sensitive tasks "do not adequately test Agent-as-a-Judge capabilities" (because the verifier cannot reproduce the same web state the original agent saw), and they "hinder reproducible environments and consistent evaluation." The authors then filter out time-sensitive tasks, rewrite some ground_truth tasks as no_ground_truth tasks, re-check no_ground_truth tasks for potential residual time sensitivity or implicit ground truth, and retain only no_ground_truth tasks in the final Mind2Web2 subset.

This filtering is philosophically important: it means the Mind2Web2 portion of the benchmark contains no single objectively correct answer that could be verified by looking up a known fact. Instead, correctness is evaluated against rubrics that assess whether the agent found valid examples matching specified criteria—sources must be credible, dates must be correct, and the evidence must genuinely support the claims. This makes verification inherently interactive—a judge cannot determine correctness without actually checking the cited sources.

WideSearch (Wong et al., 2025) provides "wide search" tasks focused on broad information coverage. These are tasks where the answer requires aggregating information across many sources, often producing tables or structured responses with multiple items. The paper describes the selection process: tasks were chosen that "differ from those in Mind2Web2" and were "lightly rewritten to explicitly encourage link-providing responses." The rewrites ensure that trajectories contain explicit references (URLs, citations) that the judge agent can follow to verify claims.

DS Domain: State Verification Through Environment Inspection

The DS domain is constructed from MCPMark (Wu et al., 2025), specifically its Filesystem and Postgres subcategories. These tasks involve manipulating file structures and database records—creating, moving, renaming, or modifying files; executing SQL queries; or restructuring database tables. The key property is stated explicitly:

"Task outcomes can be directly verified by inspecting the environment state, enabling reliable evaluation of the judge agent's state-verification capability."

This is the domain's purpose: it tests whether the judge can correctly inspect the current state of the file system or database and determine whether the original agent's operations produced the intended result. Unlike Search, where verification requires navigating external sources, DS verification requires the judge to compare the current environment state against the task requirements—checking that files exist with correct names and locations, that database tables contain the expected data, and that operations were performed in the correct order.

The filtering process involves two steps: excluding overly difficult tasks (based on results from Wu et al., 2025) to "maintain balance and obtain high-quality trajectories with both successes and failures," and manually removing tasks with ambiguous descriptions. The balance consideration is practical: if all DS tasks were extremely hard (near-zero success rate), the benchmark would lack positive examples, making it impossible to measure a judge's ability to correctly identify successes.

GUI Domain: Process and State Verification Through Visual Inspection

The GUI domain draws from OSWorld (Xie et al., 2024), which provides "a scalable real-world computer environment with high-quality multimodal agent tasks." The paper specifically selects tasks from three office applications—PowerPoint, Word, and Excel—with the explicit rationale that these "require precise execution positions and carefully planned action sequences and thus remain challenging for current agents."

The filtering for GUI tasks is more involved than for other domains because GUI environments have additional stability concerns:

Remove tasks with unstable GUI states: tasks involving "feedback pop-up windows" are excluded because these introduce non-deterministic behavior—the same action sequence replayed twice might produce different results depending on whether a pop-up appears.
Retain only tasks with reproducible final states: the paper explicitly checks that repeated execution of the same trajectory produces the same final GUI state, ensuring that the environment replay (described in §3.4.3) is deterministic enough for evaluation.
Remove tasks with subjective elements: tasks whose completion cannot be verified through "concrete and observable GUI state changes" are excluded. This means the benchmark only includes GUI tasks where success has an objective, visually verifiable manifestation—a file was created, a slide element was moved, a formatting change was applied—rather than tasks requiring aesthetic or qualitative judgment.

This design choice creates a clear boundary: GUI tasks in AJ-Bench are those where a judge agent can, in principle, determine correctness by interacting with the application state post-execution. Tasks where "correctness" is subjective or where the evidence is not visually observable are outside scope.

Task Distribution Statistics

The final benchmark contains 155 tasks distributed as shown in Table 2: Search has 61 tasks (9 WideSearch, 52 Mind2Web2/"Deep"); DS has 42 tasks (24 FileSystem, 18 Postgres); GUI has 52 tasks (21 PPT, 12 Word, 19 Excel). This distribution is not uniform—Deep Search and FileSystem tasks dominate—reflecting both the availability of high-quality source benchmarks and the filtering criteria applied. The tool counts vary by domain: 22 tools for Search (shared across both subcategories), 14 for FileSystem and 9 for Postgres, and 15 tools shared across all GUI subcategories.

Figure 3 provides a task-type distribution analysis, though the paper does not elaborate on the specific categories shown. The key point is that tasks span diverse verification requirements, preventing the benchmark from being dominated by any single type of verification challenge.

3.4.2 Trajectory Collection and Label Annotation

With tasks defined, the paper must populate the benchmark with trajectories—sequences of actions taken by task-solving agents—and assign reliable ground-truth labels indicating whether each trajectory succeeded or failed. This is the most labor-intensive part of benchmark construction, and the paper uses complementary approaches across domains to balance cost, quality, and representativeness.

Trajectory Collection: Two Complementary Approaches

The paper uses two strategies for obtaining trajectories:

1. Leveraging existing trajectories from established benchmarks. For some domains and subdomains, high-quality agent trajectories already exist from prior work. The paper draws from Mind2Web2 (Search), MCPMark (DS), and OSWorld (GUI). Using existing trajectories has the advantage of ecological validity: these are real outputs from real agent systems, not synthetic or cherry-picked examples, and they represent the diversity of agent behaviors actually observed in prior research.

2. Regenerating trajectories using LLMs. Where needed—and particularly to ensure diversity of agent architectures and to fill gaps in existing datasets—the paper generates new trajectories using multiple models. The explicit motivation is to avoid "bias introduced by model-specific output styles." If all trajectories came from a single model family, judges might learn to recognize that model's specific failure patterns rather than evaluating task completion per se.

For the Search domain, trajectories are collected using three research-oriented search tools: Gemini DeepResearch, Grok DeepSearch, and Perplexity DeepResearch. The web pages returned by these tools become the trajectories. The paper applies manual filtering to "filter out poor responses during collection," then post-processes: for Mind2Web2, gpt-5-2025-08-07 is used to "extract query-relevant information from the responses, primarily to remove excessive content unrelated to the query," with "minor human edits to fix formatting issues." For WideSearch, the extraction is done manually "to preserve tables and reference links, supplemented with brief contextual explanations."

For the DS domain, trajectories are sourced by "leveraging agent trajectories generated by multiple models on MCPMark, supplemented with original trajectories provided in the benchmark." The paper explicitly addresses model diversity: "we ensure that trajectories associated with the same task are sourced from diverse model architectures and subsequently normalized into a consistent template format." After collection, a "comprehensive manual quality check" discards "incomplete or noisy samples." The final selection rule is: "for each task, we retain up to three successful trajectories and three failed ones, resulting in a balanced and high-quality dataset."

For the GUI domain, the paper uses raw action trajectories from multiple multimodal models available in the OSWorld repository. The models span diverse architectures: claude-4-sonnet-20250514 (with 50-step and 15-step variants), o3 (50 steps), qwen2.5-vl-32b-instruct (100 steps), and doubao-1.5-thinking-vision-pro-250428 (100 steps). An important design choice is made regarding trajectory length:

"To mitigate potential bias arising from differences in trajectory length, where successful trajectories typically contain fewer steps than failed ones, we deliberately select trajectories in which successful executions may involve many steps while failures terminate after relatively few steps. This strategy helps decouple task success from trajectory length."

This is a subtle but crucial detail. If all successful trajectories are short and all failed ones are long, a lazy judge could achieve high accuracy by simply predicting "PASS" for short trajectories and "FAIL" for long ones, without examining any environment evidence. By deliberately selecting counter-stereotypical examples (long successes, short failures), the paper forces judges to actually examine task completion rather than using trajectory length as a proxy.

Label Annotation: Domain-Specific Procedures with Quality Controls

The labeling approach varies by domain because what constitutes "correctness" differs fundamentally.

Search domain labels are defined at the item level, not the trajectory level. This is a critical design choice: a single search trajectory typically makes multiple claims (e.g., a response listing three movies each with sources), and the correctness of each claim can be independently verified. The paper therefore decomposes each response into "single-item units" and labels each unit separately.

For Mind2Web2, human annotators assign "a scoring rubric and corresponding rubric-level labels to each response." The annotation team consists of "full-time annotators and student annotators, whose compensation is comparable to local market rates for similar roles." Prior to annotation, they receive "several representative Mind2Web2 examples as references" and are required to "first formulate evaluation rubrics for each response and then assign labels for every criterion in the rubric." This rubric-first approach forces annotators to make their evaluation criteria explicit before judging, reducing inconsistency. After rubric annotation, gpt-4.1 decomposes each response into single-item units based on the rubric, enabling "evaluation at a finer level of granularity."

For WideSearch, labels are obtained through a different mechanism: "majority voting across six models: gpt-4.1-2025-04-14, gpt-5-2025-08-07, o4-mini-2025-04-16, claude-sonnet-4-20250514, gemini-2.5-pro-preview-06-05, and grok-3." This is essentially an ensemble of LLM judges—if most of these six models agree on a label, that label is accepted as ground truth. Single-item units are extracted via "a combination of manual refinement and rule-based parsing of the generated Markdown tables."

The contrast between Mind2Web2 and WideSearch labeling reflects differences in the task types. Mind2Web2 tasks are no_ground_truth—they require nuanced rubric-based evaluation that automated systems cannot reliably perform, hence human annotation. WideSearch tasks involve factual claims with verifiable sources, making multi-model majority voting a reasonable approximation of ground truth.

DS domain labels are binary (1/0) at the trajectory level and are defined by a clear criterion:

"A trajectory is deemed successful only if all explicit requirements in the task description are fully satisfied."

The labeling mechanism leverages MCPMark's built-in verifier scripts, which are "derived from high-quality human annotations." These scripts automate the comparison between the environment state after trajectory execution and the task requirements. However, the paper acknowledges that automated scripts can have bugs:

"To further guarantee label correctness, we additionally perform a manual validation pass to correct potential misjudgements and maintain consistent annotation quality."

GUI domain labels are also binary at the trajectory level, but the labeling approach reveals interesting tensions. OSWorld provides "rule-based scripts that compare execution trajectory outputs against golden files for office tasks." However:

"These scripts are inherently limited in their ability to capture all execution details and edge cases, which may lead to mislabeling."

The paper therefore manually inspects "each trajectory to verify its correctness." This is expensive (GUI trajectories involve sequences of screenshots and complex tool interactions) but necessary for label reliability, and it highlights a recurring theme: automated verification is imperfect, which is precisely why agent-based verification—which can handle edge cases through interactive exploration—is worth studying.

Final Dataset Statistics

The constructed benchmark contains 155 tasks and 516 trajectories. The trajectory-to-task ratio varies by domain: Search has 183 trajectories for 61 tasks (about 3:1), DS has 229 trajectories for 42 tasks (about 5.5:1, consistent with the "up to three successes and three failures per task" rule), and GUI has 104 trajectories for 52 tasks (about 2:1, suggesting more selective curation). The balanced positive-negative ratio in DS (up to 3 each per task) is explicitly designed to prevent class imbalance from distorting evaluation metrics.

3.4.3 Environment Construction: Making Live Interaction Possible

The fundamental architectural difference between LLM-as-a-Judge and Agent-as-a-Judge is that the latter requires a live environment to interact with. The paper must therefore build infrastructure that lets judge agents query the state of the world after the original trajectory has executed—navigating web pages, inspecting file systems, or examining GUI application states. The approach differs by domain because the nature of the "environment" differs.

DS Domain: Local Replay of Action Sequences

For the DS domain (Filesystem and Postgres), the environment is a file system or database instance. The paper's approach is straightforward:

"DS tasks are replayed locally, while GUI tasks are deployed on isolated AWS instances."

The replay process works as follows: the extracted action sequence from a trajectory is executed sequentially in a fresh environment instance. Each action (e.g., "create directory," "write file with specified content," "execute SQL query") produces deterministic state changes. Once the full sequence has been replayed, the environment is in the state that would have existed after the original agent's execution. The judge agent can then inspect this state—listing directories, reading files, querying databases—to determine whether the task requirements were met.

The key design properties are isolation (each trajectory replay runs in its own environment, preventing cross-contamination), determinism (the same action sequence always produces the same state, enabling reproducible evaluation), and locality (DS environments are lightweight enough to run without cloud infrastructure).

GUI Domain: AWS-Isolated Replay with Visual State Reconstruction

GUI replay is substantially harder because GUI states are visual and interactive. The paper's approach:

"In both the DS and GUI domains, evaluation trajectories are replayed by sequentially executing the extracted action sequences in trajectories to reconstruct independent environments that support concurrent evaluation."

For GUI specifically, the paper uses AWS infrastructure:

"GUI tasks are deployed on isolated AWS instances... An AWS host manages and controls task allocation, with each trajectory being replayed and evaluated on an independent AWS instance provided by the OSWorld project's AWS AMI."

The AWS AMI (Amazon Machine Image) from OSWorld provides a pre-configured virtual machine with the necessary applications (PowerPoint, Word, Excel) and the OSWorld environment framework. Each trajectory gets its own VM instance, enabling parallel evaluation—multiple judge agents can evaluate different trajectories simultaneously without interference. After replay, the VM is in the final application state (e.g., a specific PowerPoint slide with specific elements at specific positions), and the judge agent can interact with it via mouse and keyboard actions, receiving screenshots and accessibility tree representations as observations.

Search Domain: The Live Web as Environment

The Search domain is unique in that it does not involve replay:

"In the search domain, Agent-as-a-Judge relies on interactions with external web environments."

Instead of reconstructing a past web state (which is generally impossible—web pages change), the judge agent uses browser automation tools (Playwright, as specified in the prompts in Appendix A.11.1) to navigate to the URLs cited in the original trajectory and examine the current content. This introduces a fundamental challenge that the paper acknowledges in its limitations section:

"As a result, instability in network connectivity may affect evaluation reliability."

And, implicitly, web pages may have changed since the original trajectory was created, meaning the evidence available to the judge may differ from what the original agent saw. This is an inherent limitation of web-based verification, not specific to AJ-Bench, but it affects the reproducibility of Search-domain evaluations.

3.4.4 Judge Agent Implementation: MCPMark Framework, Tools, and Prompts

The paper does not propose a novel agent architecture. Instead, it builds on an existing framework and carefully specifies the tools, prompts, and interaction protocol that transform a base LLM into a judge agent.

The MCPMark Framework

The implementation is built on MCPMark (Wu et al., 2025), which the paper characterizes as:

"a framework designed to evaluate an LLM's intrinsic ability to decide when and how to invoke tools, without relying on complex or heavily engineered workflows."

This is an important design philosophy. MCPMark provides a minimal scaffolding: the model receives observations, decides on actions (tool calls), and receives results. There is no pre-programmed verification logic, no hard-coded checklists, and no engineered multi-step reasoning templates. The judge agent's behavior emerges entirely from the model's prompted reasoning and its tool-use decisions.

For the GUI domain, the paper built custom integration:

"We implemented an OSWorld MCP server and MCP client to integrate with MCPMark."

This adapter translates between MCPMark's tool-calling protocol and OSWorld's environment interaction commands.

Tool Sets by Domain

The tools available to judge agents vary by domain (Table 2 reports 60 total tools across all domains). The paper does not exhaustively enumerate all tools, but the prompts in Appendix A.11 reveal the available actions:

Search domain (22 tools, shared across WideSearch and Deep): the primary tool is Playwright-based browser automation. The prompts reference specific MCP tools like browser_navigate (with a url parameter) and browser_snapshot (to capture page content). The workflow is: navigate to a URL → examine page content → extract evidence → repeat for additional sources. Because pages can be "lengthy," the paper applies summarization:

"We apply summarization using the same model as the agent to extract the page content, and use the resulting summary as the agent's context. Specifically, for gpt-5-mini, we use the low reasoning effort configuration during summarization, whereas for deepseek-v3.2, explicit reasoning is disabled (no thinking) for the summarization stage."

This summarization step is practical—it prevents context windows from overflowing with raw HTML—but it introduces a potential source of information loss: the judge sees a summarized version of the page, not the full content, which could miss evidence that a human reader would notice.

DS domain (14 tools for FileSystem, 9 for Postgres): The prompts reference list_allowed_directories, directory_tree (with a path parameter), read_text_file, read_multiple_files (with a paths array parameter), and write_file. These are standard file system inspection operations. For Postgres, the tool set presumably includes SQL query execution, though this is not explicitly enumerated in the main paper.

GUI domain (15 tools, shared across subdomains): The Appendix A.11.3 prompt defines an extensive action space:

Mouse actions: CLICK (with x, y, button, num_clicks parameters), DOUBLE_CLICK, RIGHT_CLICK, MOUSE_DOWN, MOUSE_UP, DRAG_TO, SCROLL (with dx, dy).
Keyboard actions: TYPING (with text), PRESS, KEY_DOWN, KEY_UP, HOTKEY (with keys array).
Control actions: DONE (signal that judgment is ready), WAIT, VIEW_TRAJECTORY_STEP (with step number).

The VIEW_TRAJECTORY_STEP tool is notable because it lets the judge inspect intermediate states from the original trajectory, not just the final state. The prompt emphasizes that this is for understanding execution process, not for final judgment:

"VIEW_TRAJECTORY_STEP is SUPPLEMENTARY: The a11y_tree from VIEW_TRAJECTORY_STEP shows INTERMEDIATE steps that may contain ERRORS that were later CORRECTED. Your final judgement should be based on the CURRENT/FINAL environment state, NOT on intermediate step states."

The Interaction Protocol

The judge agent operates in a sequential loop, as specified in the GUI domain system prompt:

"You should provide ONLY ONE action per response. Each response should contain exactly ONE ACTION line and ONE REASONING line. Do NOT provide multiple actions in a single response. The system will execute only one action at a time, and you will receive the result before choosing the next action."

This single-action-per-turn protocol is simpler than multi-step planning approaches and forces the agent to react to observations. The prompt provides formatting rules:

"Format your response EXACTLY as: ACTION: {{"action_type": "ACTION_TYPE", "param1": value1, "param2": value2}} REASONING: [brief explanation of why you chose this action]"

After executing the action, the system returns an observation, and the agent decides on the next action. The loop continues until the agent emits a DONE action, at which point it produces a final judgment.

Prompt Design: Guiding Evidence-Based Judgment

The prompts across domains (detailed in Appendix A.11) share a common philosophy: they instruct the judge to be evidence-first, basing judgments on observations gathered through interaction rather than on assumptions from the trajectory text.

The GUI domain prompts are particularly elaborate due to the complexity of visual interaction. Key instructions include:

Result-oriented evaluation: "FOCUS ON THE PRIMARY TASK GOAL: Evaluate whether the trajectory accomplishes the main objective stated in the task instruction... Evaluate success based on whether the final state achieves the task's core objective, regardless of which specific menu item, feature, or method was used to achieve it."
Trust clear UI indicators: "When UI elements clearly show a state (e.g., dropdown selections, toggle states, menu checkmarks, status indicators), treat these as reliable evidence of the current state. Do not unnecessarily doubt clear visual confirmations."
Required interaction baseline: "REQUIRED BASELINE: First REVIEW the original trajectory provided (this is mandatory). Then perform AT LEAST TWO evidence-gathering interactions (live environment or VIEW_TRAJECTORY_STEP) before making a decision."
Comprehensive exploration before declaring failure: "Before marking a task as failed, ensure you have explored ALL sheets/tabs/pages/slides where the target object might exist. If a task mentions 'new sheet' or involves multi-sheet navigation, failure to check all sheets invalidates your judgement."
Coordinate precision for accessibility tree: "When the observation type is a11y_tree (or screenshot_a11y_tree), you MUST use the coordinates and element metadata provided by the tree to drive your interactions (especially click targets). Do NOT guess positions from visual assumptions alone."

The Search domain prompts specify a detailed verification procedure with checks for faithful representation of source material and factual correctness. The DS domain prompts are simpler, instructing the judge to "Check the current state of the environment, Verify that the expected outputs/changes exist, Confirm the correctness of any created or modified resources."

Model Configurations

The paper evaluates two primary judge models in the Agent-as-a-Judge setting: gpt-5-mini-low (a closed-source model) and deepseek-v3.2 (an open-source model). For ablation experiments, additional models are used: gpt-5-mini-medium/high reasoning effort variants, deepseek-v3.2 with thinking mode enabled, gemini-3-flash-preview, claude-sonnet-4.5, kimi-k2.5, and glm-4.7 (Appendix A.5).

"Unless explicitly stated otherwise, all models are evaluated with their default configurations (e.g., temperature and reasoning effort)."

This default-configuration approach is deliberate: the authors want to measure the effect of tool access itself, not the effect of hyperparameter tuning. Any observed differences between LLM-as-a-Judge and Agent-as-a-Judge can be attributed to the tool-use capability rather than to different temperature or decoding settings.

3.4.5 Evaluation Protocol and Metrics

The paper must define how judge agent outputs are mapped to correctness scores, and how those scores are aggregated across tasks, trajectories, and runs.

The F1 Score as Primary Metric

The paper uses F1 score as its primary evaluation metric:

"We adopt F1 score as our primary evaluation metric."

The choice of F1 (the harmonic mean of precision and recall) over simple accuracy is deliberate for the Search domain, where evaluation happens at the item level rather than the trajectory level. The paper's approach:

"In the Search domain, we aggregate the evaluations of all single items within a trajectory into a single result, from which the F1 score is computed. In the DS and GUI domains, we compute the F1 score based on trajectory-level evaluations."

For trajectory-level binary classification (DS and GUI), F1 reduces to a specific form. Given the confusion matrix:

True Positives (TP): judge correctly identifies a successful trajectory as PASS
False Negatives (FN): judge incorrectly identifies a successful trajectory as FAIL
False Positives (FP): judge incorrectly identifies a failed trajectory as PASS
True Negatives (TN): judge correctly identifies a failed trajectory as FAIL

The F1 score is:

$F1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$

where $\text{Precision} = \frac{TP}{TP + FP}$ is the fraction of trajectories the judge labeled PASS that actually succeeded, and $\text{Recall} = \frac{TP}{TP + FN}$ is the fraction of actually successful trajectories that the judge correctly identified.

What it computes: A single scalar between 0 and 1 that balances the judge's ability to correctly identify successes (recall) against its tendency to falsely label failures as successes (precision). A judge that always says "PASS" would have perfect recall (it never misses a success) but poor precision (it also passes all failures), yielding a low F1. A judge that almost never says "PASS" would have high precision (when it does, it's usually right) but poor recall (missing most successes), again yielding a low F1.

Why this form: Accuracy (fraction of all judgments that are correct) can be misleading when classes are imbalanced. If 80% of trajectories fail, a judge that always says "FAIL" gets 80% accuracy but F1 = 0 (because recall is zero). F1 penalizes both types of error—false positives and false negatives—and is the standard metric for binary classification when both classes matter. The paper also reports Precision, Recall, FPR, and FNR in Appendix Table 9, providing a more complete picture of the error trade-offs.

Cross-Run Averaging

The paper reports averaged results to account for stochasticity:

"Results reported in Table 3 are averaged over three runs."

Appendix A.6 provides statistical reliability analysis, computing 95% confidence intervals for subdomain-level scores using the t-distribution based on these three independent runs. The intervals serve to distinguish genuine performance differences from noise, particularly important given the "highly uneven subset sizes" across subdomains—some subdomains like PPT have as few as 21 tasks (42 trajectories across two folds), making per-subdomain estimates inherently noisier than aggregate scores.

Judgment Extraction

The judge agent's final output must be parsed to extract a binary verdict. For Search, the prompts specify:

"Your final output must be exactly one line: Response: [1] or Response: [0] Nothing else."

For DS: "Output Format (Strict): RESULT: PASS or FAIL."

For GUI: the judge produces a structured judgment with judgement: [true or false], DETAILED_REASONING, and CONFIDENCE: [High|Medium|Low]. The judgment field provides the binary label for F1 computation; confidence is auxiliary information not used in the primary metric.

Domain-Specific Granularity

The difference between Search (item-level) and DS/GUI (trajectory-level) evaluation reflects fundamental differences in task structure. A Search task like "Find 3 movies that premiered in 2024 with matching US-China release dates" produces a response with three claims. A judge might correctly verify two claims but miss an error in the third. At the trajectory level, the response as a whole would be "partially correct," but labeling the entire trajectory as PASS or FAIL would be too coarse. By decomposing into items and evaluating each item independently, then aggregating via F1, the benchmark captures this nuance.

Interaction Turn Ablation Protocol

The paper explicitly studies how the interaction budget affects judge performance (Figure 4, §4.3.2):

"We set different maximum interaction turn limits under deepseek-v3.2 for several subdomains."

The interaction turn limit caps how many tool-calling iterations the judge agent is allowed before it must produce a final judgment. Budgets tested range from 1 to 32 turns. The finding that "increasing the interaction budget consistently improves F1 scores across all tasks, with the most pronounced gains observed for smaller budgets" directly validates the core premise of Agent-as-a-Judge: more environmental interaction provides more evidence, which improves judgment accuracy.

3.4.6 Comparative Baselines: LLM-as-a-Judge Implementation

To enable the controlled comparison that is the paper's central empirical claim, the LLM-as-a-Judge baseline must differ from Agent-as-a-Judge in only one dimension: access to tools and environment. The paper achieves this by using the same models, the same tasks, and the same judgment extraction protocol, but without providing any tool definitions or environment access.

The LLM-as-a-Judge prompts (Appendix A.11) are structurally parallel to the Agent-as-a-Judge prompts but differ in a crucial way: all instructions about tool use are removed, and the judge is instructed to base its judgment on "your knowledge" (for LLM-as-a-Judge) rather than on evidence gathered through interaction.

For Search (Wide), the LLM-as-a-Judge prompt says:

"Step 3 — Verify Each Claim Individually... with your knowledge"

The Agent-as-a-Judge version changes this to:

"Step 3 — Verify Each Claim Individually with Playwright MCP Tools"

This single substitution is the entire difference between the two conditions. Everything else—the query, the response, the extracted items, the judgment format—is identical.

For DS, the LLM prompt provides only the task description and trajectory text, with the instruction: "Base your judgement only on the information provided in the Task Description and the Agent Trajectory." The Agent version provides the same information plus: "Use the available tools to: 1. Check the current state of the environment 2. Verify that the expected outputs/changes exist 3. Confirm the correctness of any created or modified resources."

For GUI, the LLM prompt asks the judge to "Analyze the trajectory to determine if it successfully completes the given instruction" and "Consider whether the trajectory follows logical steps and achieves the intended goal." The Agent version replaces this with an extensive tool-use protocol where the judge actively clicks, types, and inspects the application state.

The paper evaluates a wide range of models in the LLM-as-a-Judge setting (Table 3): eight proprietary models (gemini-3-pro-preview, gemini-2.5-pro, claude-opus-4.5, claude-sonnet-4.5, gpt-5, gpt-5.1, grok-4, gpt-5-mini-low) and five open-source models (kimi-k2-0905-preview, qwen3-235b-a22b, glm-4.6, longcat-flash-chat, deepseek-v3.2). This broad coverage establishes a strong baseline—the Agent-as-a-Judge must outperform not just a single LLM-as-a-Judge configuration, but the best available model in the non-agentic setting.

The Controlled Comparison Logic

The comparison between LLM-as-a-Judge and Agent-as-a-Judge operates on a fixed base model. For example, gpt-5-mini-low is evaluated both with tools (Agent-as-a-Judge) and without tools (LLM-as-a-Judge) on the exact same tasks and trajectories. Any performance difference can therefore be attributed to the tool-use capability. The paper reports "Improvement" rows in Table 3 that directly subtract the LLM-as-a-Judge F1 from the Agent-as-a-Judge F1 for each base model.

This controlled design avoids a common confound in benchmark papers: comparing a new method (Agent-as-a-Judge) on one model against old methods (LLM-as-a-Judge) on different, possibly weaker models. If the paper had only compared gpt-5-mini-low with tools against, say, an older GPT-3.5 without tools, the improvement could be due to the base model upgrade, not the tool access. By using the same base model in both conditions, the paper isolates the effect of interest.

Ablation Studies: Probing the Sources of Performance

The paper's ablation experiments (§4.3) investigate factors beyond the simple tool-vs-no-tool comparison:

Reasoning effort ablation (§4.3.1): The same base model (gpt-5-mini) is tested at three reasoning effort levels—low, medium, and high—all in the Agent-as-a-Judge setting. The finding that "medium setting generally outperforms the low setting, while the high setting does not consistently outperform medium" suggests that more reasoning compute does not monotonically improve judgment, likely because effective tool use requires different cognitive skills than pure reasoning.

Interaction turn ablation (§4.3.2): The same model (deepseek-v3.2) with the same tools is tested with different limits on how many tool-calling iterations it can make before producing a judgment. This directly tests the hypothesis that more environmental interaction provides more evidence and improves judgments.

Multimodal modality ablation (§4.3.3): In the GUI domain, the judge agent's observation modality is varied: accessibility tree only, screenshot only, or both. This tests which types of environmental information are most useful for different subdomains—finding, for instance, that "the mixed-modality configuration consistently outperforms the single-modality alternatives" in Excel, but that in PPT the accessibility tree alone is competitive with the mixed setting.

Framework ablation (Appendix A.4): The judge agent is reimplemented using ReAct (which "requires agents to explicitly produce reasoning and actions at each step") instead of MCPMark (which has "more autonomous and implicit reasoning process"). Both frameworks with the same base models outperform LLM-as-a-Judge, showing that the advantage is robust to the specific agent scaffolding.

Model family ablation (Appendix A.5): Four additional models beyond the main evaluation (gemini-3-flash-preview, claude-sonnet-4.5, kimi-k2.5, glm-4.7) are tested on a subset of tasks in both agentic and non-agentic settings. Agent-as-a-Judge improves performance for all of them, supporting "the robustness of our conclusion beyond a single judge family."

4. Key Insights and Innovations

Innovation 1: Agent-as-a-Judge as a Distinct Capability, Not a Scaled-Up LLM-as-a-Judge

The paper's most fundamental conceptual move is establishing Agent-as-a-Judge as a separate category of evaluation capability rather than treating it as "LLM-as-a-Judge with extra features." This is not merely a taxonomic distinction—it reframes what verification is in interactive environments.

Before this work, the dominant framing in the LLM-as-a-Judge literature (Zheng et al., 2023; Lambert et al., 2025; Tan et al., 2025) treated evaluation as a text-understanding problem: the judge reads the prompt, the response, and perhaps a rubric, then produces a judgment using its internal knowledge and reasoning. Adding tools was seen as an augmentation—a way to give the judge access to external facts it might not have memorized. Under this view, an Agent-as-a-Judge is fundamentally an LLM-as-a-Judge with a search engine attached.

AJ-Bench forces a different understanding. The paper demonstrates—implicitly through its benchmark design and explicitly through its ablation results—that effective verification in interactive environments requires capabilities that are orthogonal to text-based judgment. Specifically:

The decision of whether and when to interact is itself a core capability. An LLM-as-a-Judge always operates on fixed input. An Agent-as-a-Judge must decide: Do I have enough information to judge from the trajectory text alone? If not, what evidence do I need, and what tool will provide it? When have I gathered sufficient evidence to stop? These meta-cognitive decisions have no analog in non-agentic evaluation, and the paper's interaction-turn ablation (Figure 4) shows they directly impact performance: more interaction turns consistently improve F1, with diminishing returns at higher budgets. This is not a "bigger model → better judgment" curve; it is a "better information-gathering strategy → better judgment" curve.

Environmental interaction changes the nature of evidence. An LLM-as-a-Judge can only assess plausibility from surface text. An Agent-as-a-Judge can produce disconfirming evidence—it can navigate to a cited URL and find that the information doesn't match (as in Figure 1), or inspect a file system and discover a required file was never created (as in Figure 6), or click through GUI tabs and confirm a formatting change was applied. This shifts the judge's epistemic position from "this seems consistent with what I know" to "I have directly observed that the claimed state exists or does not exist." The paper's failure-mode analysis (Appendix A.7, Table 8) shows that even when judges successfully retrieve correct evidence, they can still reason incorrectly about it (error type d: 11-68% of failures depending on domain and model). This reveals that evidence acquisition and evidence interpretation are separable failure modes—a distinction invisible in LLM-as-a-Judge, where both are collapsed into a single text-comprehension process.

What makes this a fundamental shift rather than incremental: Prior work like Themis (Li et al., 2024), TIR-Judge (Xu et al., 2025), and Agentic Reward Modeling (Peng et al., 2025) added tool use to judges but evaluated them on existing reasoning benchmarks rather than on interactive verification tasks. These papers treated tools as a way to improve reasoning, not as a way to fundamentally change what verification means. AJ-Bench's contribution is to define the evaluation problem in terms of the interaction itself—to ask not "does adding tools improve reasoning scores?" but "what verification capabilities emerge when a judge can interact with environments, and how do we measure them?" This reframing opens a new research subfield: the design and evaluation of judge agents as a distinct class of AI system with their own success criteria, failure modes, and scaling properties.

The evidence anchoring this claim is not a single table but the structural contrast between Table 3's LLM-as-a-Judge results and Agent-as-a-Judge results. The fact that gpt-5-mini-low with tools (overall F1 72.41) outperforms gpt-5 without tools (overall F1 61.02)—a model from the same family but substantially more capable in pure reasoning—demonstrates that the agentic capability is not simply additive to base model strength. It is a different axis of performance entirely.

Innovation 2: The Environment as Co-Evaluator—Recasting Verification as Active State Reconciliation

The paper's second conceptual contribution is more architectural than taxonomic: it redefines verification from a one-sided judgment (the judge evaluates the agent) to an interactive reconciliation between the judge, the environment, and the trajectory. This recharacterization carries implications for how verification systems should be designed, trained, and evaluated.

In the standard LLM-as-a-Judge paradigm, the verification process is: judge receives (task, trajectory) → judge outputs (PASS/FAIL). The environment is not a participant; it is the subject of the trajectory, flattened into text observations that the judge reads passively. The judge's knowledge and reasoning are the sole sources of evaluative power.

In AJ-Bench's Agent-as-a-Judge paradigm, the verification process becomes: judge receives (task, trajectory) → judge queries the environment (via tool calls) → environment responds with state information → judge compares observed state against task requirements → judge outputs (PASS/FAIL). The environment is now a co-evaluator: it provides ground-truth state information that the judge could not infer from the trajectory text alone. The judge's role shifts from deciding to reconciling—it must determine whether the trajectory's claimed effects match the environment's actual state.

This is not merely a more complex workflow. It creates new failure modes and new requirements that existing evaluation paradigms never had to handle:

The environment can be the bottleneck. The paper's limitations section explicitly acknowledges this: "in the search domain, Agent-as-a-Judge relies on interactions with external web environments. As a result, instability in network connectivity may affect evaluation reliability." More subtly, web page content changes over time, meaning the evidence available to the judge may differ from what the original agent observed. The judge is now vulnerable to environment drift—a failure mode that LLM-as-a-Judge never encounters because it never consults the environment.

Tool selection becomes a verification skill. The judge must choose which tools to invoke and in what sequence to efficiently gather sufficient evidence. The paper's failure-mode analysis documents cases where judges "failed to call tools or omitted necessary tool calls" (error type a: 1-26% of failures) or "invoked incorrect tools" (error type b: negligible to ~3%). These errors are conceptually distinct from reasoning errors—the judge might know exactly what it needs to check but lack the procedural knowledge to execute the check correctly.

State interpretation requires domain knowledge. The paper's failure cases (Figure 7) show judges who can successfully navigate to the relevant file, read its contents, and still fail to identify a bug because they misinterpret the code structure. The environment can provide all the necessary information and the judge can still fail at the reconciliation step if it lacks the domain expertise to interpret what it observes.

What makes this a shift in perspective: Before AJ-Bench, the research community implicitly treated verification as a pure reasoning problem—better models with better training would produce better judgments. The paper shows that verification quality is fundamentally constrained by the quality of environment interaction, not just model capability. The observation that deepseek-v3.2 with thinking mode (F1 77.11) performs slightly worse than deepseek-v3.2 without thinking (F1 77.34) in the Agent-as-a-Judge setting (Table 4) is particularly telling: stronger reasoning does not compensate for suboptimal interaction patterns. The verification bottleneck is at the agent-environment interface, not in the model's cognitive horsepower.

This insight has direct implications for future research that the paper does not fully articulate but that its results imply: improving Agent-as-a-Judge will require advances in tool-use learning, environment modeling, and interaction strategy optimization—not just larger models or better reasoning. The research agenda shifts from "train better judges" to "train better explorers."

Innovation 3: The Difficulty of Verification Is Domain-Specific and Modality-Dependent

The paper's third conceptual contribution is empirical rather than theoretical, but it carries significant practical implications: the effectiveness of different verification strategies is not uniform across domains or even across subdomains within the same broad category, and no single input modality (text, screenshots, accessibility trees) dominates across all tasks.

This finding emerges most clearly from the multimodal ablation study (Figure 5, §4.3.3), which reveals a pattern that would be invisible in aggregate metrics:

In PPT tasks, the accessibility tree alone (F1 ~76-80) performs comparably to the mixed-modality setting (F1 ~77-92). Screenshots alone are weaker for gpt-5-mini-low (F1 ~70) but competitive for gemini-3-flash-preview (F1 ~91.7).
In Word tasks, screenshots alone dominate for gpt-5-mini-low (F1 ~86.4 vs. ~84.4 for mixed and ~80 for accessibility tree only), while the pattern reverses entirely for gemini-3-flash-preview (F1 ~66.7 for screenshots vs. ~90 for mixed).
In Excel tasks, the mixed modality is consistently best (F1 ~81.9-92.3 across both models), with single modalities substantially worse (F1 as low as 54.5 for screenshots alone with gpt-5-mini-low).

The deeper implication: there is no universal verification modality. PowerPoint tasks depend on spatial layout information that is well-represented in structured accessibility trees. Word tasks benefit from visual inspection of formatting and content that screenshots capture but tree structures might normalize away. Excel tasks require both the structural information from trees (cell coordinates, formulas) and visual confirmation of layout and formatting. A verification system that commits to a single observation modality will be suboptimal on some tasks, and the paper's results suggest that the optimal modality configuration itself depends on the base model—gemini-3-flash-preview handles screenshots differently than gpt-5-mini-low, likely due to differences in vision-language training.

The domain-specific pattern recurs in the main results (Table 3). Agent-as-a-Judge gains over LLM-as-a-Judge are not uniform:

PPT (+31.23 for gpt-5-mini-low, +24.76 for deepseek-v3.2): The largest improvements are in the subdomain requiring the most complex spatial and visual reasoning, where LLM-as-a-Judge's lack of direct visual access is most crippling.
FileSystem (+7.13 and +12.29): More modest gains, since file system verification can often be partially inferred from trajectory text (if the trajectory says it created a file, that is often reliable).
Postgres (+1.78 and +6.39): The smallest gains, potentially because database verification involves checking query correctness and data integrity—tasks where textual reasoning dominates and environmental re-inspection adds less marginal value.

Why this matters beyond the specific numbers: The field's default assumption, implicit in most LLM-as-a-Judge work, is that verification is a general capability that transfers across domains. A good judge on reasoning tasks should be a good judge on coding tasks on safety tasks. AJ-Bench's results suggest this assumption is false for Agent-as-a-Judge—the verification capability is domain-grounded in ways that text-based judgment is not. A judge agent that excels at web search verification (where it needs to navigate URLs and compare claims against sources) may struggle at GUI verification (where it needs to interpret visual states and execute precise mouse actions), even if the same base model underlies both. This has practical implications: verification systems deployed in production may need domain-specific training or configuration, not just a one-size-fits-all judge model.

The paper does not fully explore this implication—it reports results per subdomain but does not analyze cross-domain correlation of judge performance—but the data pattern is clear enough to constitute a finding in its own right. It challenges the implicit assumption of transferable judging capability that underlies much of the reward model and LLM-as-a-Judge literature.

Innovation 4: Verifier Over-Optimization Is Not the Limiting Factor—Information Acquisition Is

A recurring concern in the test-time compute and RLHF literatures is that verifiers can be over-optimized: an agent or policy learns to exploit the verifier's weaknesses, producing outputs that score highly under the verifier but are actually incorrect. The paper on test-time compute scaling (Snell et al., 2024) documented this phenomenon extensively—beam search degraded performance on easy problems because the process reward model was imperfect.

AJ-Bench's results suggest a different primary bottleneck for Agent-as-a-Judge: the limiting factor is not that judges over-optimize their evidence-gathering but that they fail to gather sufficient evidence in the first place. The failure-mode analysis (Appendix A.7, Table 8) shows that "misinterpretation of tool outputs" (error type c) is consistently the most common failure mode, accounting for 30-80% of errors across domains and models. "Failure to invoke tools or omission of necessary tool calls" (error type a) adds another 1-26%. Together, information-acquisition failures dominate over "incorrect reasoning despite correct evidence" (error type d: 11-68%).

This is a negative result—judges are not failing because they're cleverly exploiting the environment to confirm their biases; they're failing because they don't look hard enough or don't understand what they see. The interaction-turn ablation (Figure 4) reinforces this: giving judges more interaction turns consistently improves performance, with the steepest gains at low budgets. If over-optimization were the problem, more interaction might lead to worse judgments as the judge found spurious evidence supporting its initial impression. Instead, more interaction leads to monotonically better judgments, suggesting that the core deficit is evidence insufficiency, not evidence misinterpretation.

The significance of this finding: It redirects the research agenda for Agent-as-a-Judge away from robustness and calibration (the typical focus when over-optimization is the concern) and toward exploration and information-gathering efficiency. The key research questions become: How do we train judge agents to know when they need more evidence? How do we teach them to select the right tools for the verification task? How do we get them to explore comprehensively (checking all sheets, tabs, and views) rather than stopping at the first superficially plausible state? These are reinforcement learning and exploration problems, not verification-robustness problems.

The paper's framework ablation (Appendix A.4, Table 5) provides partial corroboration: ReAct, which "requires agents to explicitly produce reasoning and actions at each step," sometimes underperforms MCPMark (e.g., gpt-5-mini-low on Wide: 51.13 vs. 65.93 F1), suggesting that explicit reasoning does not automatically translate to better evidence-gathering. The bottleneck is not in the reasoning about what to check but in the effective execution of checks. This aligns with the paper's broader implication that Agent-as-a-Judge requires capabilities—exploration strategy, tool-use proficiency, environment modeling—that current LLM training paradigms do not explicitly target.

5. Experimental Analysis

Evaluation Methodology

Dataset. AJ-Bench consists of 155 tasks and 516 annotated trajectories across three domains: Search (61 tasks, 183 trajectories drawn from Mind2Web2 and WideSearch), Data Systems (42 tasks, 229 trajectories from MCPMark's Filesystem and Postgres subcategories), and Graphical User Interfaces (52 tasks, 104 trajectories from OSWorld's PowerPoint, Word, and Excel categories). The benchmark uses the full 516-trajectory set for evaluation, with labels obtained through human annotation (Search/Mind2Web2, GUI), multi-model majority voting (Search/WideSearch), and verifier scripts with manual validation passes (DS, GUI). The dataset is not split into train/validation/test; instead, all trajectories are used for evaluation, with difficulty characterized post-hoc through the distribution of task types (Figure 3) rather than through a predefined stratification.
Base models. For the primary experiments, the paper evaluates eight proprietary models as LLM-as-a-Judge baselines: gemini-3-pro-preview, gemini-2.5-pro, claude-opus-4.5, claude-sonnet-4.5, gpt-5, gpt-5.1, grok-4, and gpt-5-mini-low. Five open-source models serve as additional LLM-as-a-Judge baselines: kimi-k2-0905-preview, qwen3-235b-a22b, glm-4.6, longcat-flash-chat, and deepseek-v3.2. For Agent-as-a-Judge, the paper selects two representative models for in-depth evaluation due to budget constraints: gpt-5-mini-low (closed-source) and deepseek-v3.2 (open-source). Additional models are introduced in ablation studies: gpt-5-mini at medium and high reasoning effort levels, deepseek-v3.2 with thinking mode enabled, gemini-3-flash-preview, claude-sonnet-4.5, kimi-k2.5, and glm-4.7. The paper argues gpt-5-mini-low was chosen to demonstrate that "Agent-as-a-Judge built on weaker models can achieve performance comparable to that of LLM-as-a-Judge based on sota models" (Section 4.2), while deepseek-v3.2 provides an open-source reference point.
Metrics. The primary evaluation metric is the F1 score, the harmonic mean of precision and recall. In the Search domain, evaluation operates at the item level: each trajectory response is decomposed into single-item units (via gpt-4.1 for Mind2Web2 and via manual refinement plus rule-based parsing for WideSearch), each item is judged independently, and all item-level judgments within a trajectory are aggregated into a single trajectory-level result before F1 is computed. In the DS and GUI domains, evaluation operates directly at the trajectory level: each trajectory receives a single binary PASS/FAIL judgment, and F1 is computed across all trajectories. The paper reports Precision, Recall, False Positive Rate (FPR), and False Negative Rate (FNR) in Appendix Table 9 as supplementary metrics. All main results in Table 3 are reported as the average of three independent runs, with 95% confidence intervals computed using the t-distribution for subdomain-level analysis (Appendix A.6).
Baselines. The paper establishes two categories of baselines. (1) LLM-as-a-Judge baselines: All thirteen models (eight proprietary, five open-source) are evaluated without tool access on the same tasks and trajectories, using domain-specific prompts (detailed in Appendix A.11) that instruct the judge to evaluate based on the task description and trajectory text alone. This is the primary comparative baseline. (2) Majority voting and oracle baselines: For WideSearch labels, majority voting across six models (gpt-4.1-2025-04-14, gpt-5-2025-08-07, o4-mini-2025-04-16, claude-sonnet-4-20250514, gemini-2.5-pro-preview-06-05, grok-3) serves as a ground-truth labeling mechanism rather than an explicit evaluation baseline. The paper does not include a random baseline, a human-performance baseline, or a "trajectory-length heuristic" baseline (e.g., predict PASS for short trajectories, FAIL for long ones), which would help contextualize the absolute F1 scores.
Generation budget / compute accounting. For Agent-as-a-Judge, the primary compute budget is measured in interaction turns—the number of tool-calling iterations the judge agent performs before producing a final judgment. The default configuration imposes no explicit turn limit (the judge decides when to call DONE), but the ablation in Section 4.3.2 explicitly caps interactions at 1, 2, 4, 8, 16, and 32 turns to study the effect of interaction budget on performance. The paper does not report the actual number of turns used in the default (unlimited) setting, nor does it account for the cost of environment replay (VM provisioning for GUI, file system initialization for DS) in any compute comparison between Agent-as-a-Judge and LLM-as-a-Judge. The interaction turn budget is not directly comparable to generation budget in the test-time compute literature (Snell et al., 2024) because each turn can involve variable amounts of computation (summarizing web pages, executing SQL queries, capturing and processing screenshots).
Cross-validation / statistical protocol. The paper does not employ cross-validation for strategy selection (since it does not train or select hyperparameters on a validation set). Instead, all models are evaluated on the full benchmark with their default configurations. Statistical reliability is assessed through: (1) averaging over three independent runs to account for stochasticity in model outputs; (2) computing 95% confidence intervals using the t-distribution for subdomain-level F1 scores (Appendix A.6, Table 7); and (3) reporting the standard deviation or range of those three runs (Table 3 reports Avg@3 with ± values for overall scores). The confidence intervals are used specifically to assess whether performance gains from Agent-as-a-Judge "remain consistent across subdomains despite the highly uneven subset sizes," with the finding that "in subdomains with relatively sufficient sample sizes, such as DEEP, the intervals are strictly non-overlapping" while in smaller subdomains (PPT, WORD) "the intervals are noticeably wider, reflecting higher variance induced by limited sample sizes" (Appendix A.6).

Main Quantitative Results

Agent-as-a-Judge vs. LLM-as-a-Judge: Aggregate Performance (Table 3)

The headline result is that Agent-as-a-Judge consistently outperforms LLM-as-a-Judge when built on the same base model, with an average improvement of approximately 13 percentage points in F1 across all three domains. For the closed-source comparison: gpt-5-mini-low achieves an overall F1 of 59.00 as an LLM-as-a-Judge versus 72.41 as an Agent-as-a-Judge, an improvement of +13.41 points. For the open-source comparison: deepseek-v3.2 achieves 64.49 versus 77.34, an improvement of +12.85 points.

The gains are not uniform across domains. Breaking down gpt-5-mini-low's performance by domain (Table 3):

Search: WideSearch improves from 60.84 to 65.93 (+5.09); DeepSearch improves from 68.42 to 75.69 (+7.27). Combined Search improvement: roughly +6.2 points.
DS: FileSystem improves from 60.41 to 67.54 (+7.13); Postgres improves from 65.52 to 67.30 (+1.78). Combined DS improvement: roughly +4.5 points.
GUI: PPT improves from 45.05 to 76.28 (+31.23); Word improves from 48.41 to 72.22 (+23.81); Excel improves from 64.36 to 81.89 (+17.53). Combined GUI improvement: roughly +24.2 points.

The GUI domain dominates the aggregate improvement, with PPT alone contributing a +31.23 point gain—larger than the entire Search-domain improvement. This pattern holds for deepseek-v3.2 as well: PPT +24.76, Word +8.87, Excel +9.59 in GUI versus +8.82 (Wide) and +19.23 (Deep) in Search, and +12.29 (FileSystem) and +6.39 (Postgres) in DS.

Absolute performance ceiling: The strongest Agent-as-a-Judge configuration (deepseek-v3.2 with tools) reaches an overall F1 of 77.34. The best LLM-as-a-Judge (gemini-3-pro-preview) reaches 75.05. The gap between the best agentic and best non-agentic configurations is only +2.29 points overall, which is narrower than the improvement within a single model when tools are added. This means Agent-as-a-Judge on a weaker model (gpt-5-mini-low, F1 72.41) can surpass the best LLM-as-a-Judge on a stronger model (gpt-5, F1 61.02), confirming the paper's claim that "Agent-as-a-Judge built on weaker models can achieve performance comparable to that of LLM-as-a-Judge based on sota models" (Section 4.2).

SOTA LLM-as-a-Judge ranking (without tools): Among proprietary models, gemini-3-pro-preview leads at 75.05, followed by claude-sonnet-4.5 at 69.77, grok-4 at 69.24, and gemini-2.5-pro at 68.31. gpt-5 achieves only 61.02—substantially below several competitors—while gpt-5.1 trails at 53.50. Among open-source models, glm-4.6 leads at 64.81, followed closely by deepseek-v3.2 at 64.49 and kimi-k2-0905-preview at 64.33. The range from best (75.05) to worst (53.50) is 21.55 points, indicating substantial model-dependence in non-agentic judging capability.

Agent-as-a-Judge ranking (with tools): Only two models are evaluated: deepseek-v3.2 at 77.34 and gpt-5-mini-low at 72.41. The open-source model outperforms the closed-source model by 4.93 points in the agentic setting, despite being 4.59 points behind gpt-5-mini-low in the non-agentic setting (64.49 vs. 59.00—actually ahead, since 64.49 > 59.00; the paper's framing of gpt-5-mini-low as the "weaker model" refers to its non-agentic F1 of 59.00 vs. deepseek-v3.2's 64.49, but in the agentic setting deepseek-v3.2 extends its lead). The reversal of relative standing between agentic and non-agentic settings suggests that base model capability and tool-use capability are not simply additive.

Difficulty-Dependent Performance: Domain and Subdomain Analysis

The paper does not explicitly bin tasks by difficulty (unlike Snell et al., 2024), but the subdomain-level results in Table 3 reveal substantial variation that functions as an implicit difficulty gradient.

Where LLM-as-a-Judge struggles most: The lowest LLM-as-a-Judge F1 scores across all models occur in GUI tasks—particularly PPT (ranging from 41.90 for gpt-5.1 to 76.10 for gemini-3-pro-preview) and Word (ranging from 30.35 for longcat-flash-chat to 72.14 for gemini-3-pro-preview). The DS domain shows moderate scores (FileSystem ranging from 55.96 for kimi-k2-0905-preview to 75.70 for grok-4), and Search shows the highest LLM-as-a-Judge performance (DeepSearch ranging from 62.91 for deepseek-v3.2 to 81.80 for longcat-flash-chat). This gradient—Search > DS > GUI for LLM-as-a-Judge—aligns with the intuition that text-only judgment is most effective when the evidence is already present in the trajectory text (Search responses contain explicit claims and citations) and least effective when the evidence is visual and interactive (GUI tasks require seeing the application state).

Where Agent-as-a-Judge provides the most leverage: The improvement from adding tools is inversely correlated with LLM-as-a-Judge performance. GUI tasks, where LLM-as-a-Judge is weakest, show the largest gains (+17-31 points). Search tasks, where LLM-as-a-Judge is strongest, show more modest gains (+5-19 points). DS tasks are intermediate (+1.8-12.3 points). This pattern is consistent with the interpretation that tools provide the most value precisely when the trajectory text alone is insufficient to determine correctness—exactly the regime the benchmark was designed to capture.

Postgres as an outlier: The Postgres subdomain shows the smallest improvement (+1.78 for gpt-5-mini-low, +6.39 for deepseek-v3.2) and the second-highest LLM-as-a-Judge baseline among DS subdomains (65.52 and 66.31, respectively). This suggests that database verification may be partially determinable from SQL query text and execution output in the trajectory, reducing the marginal value of re-querying the database. The paper does not analyze this case specifically.

Interaction Budget Scaling (Figure 4, Section 4.3.2)

The core finding from the interaction-turn ablation is that increasing the interaction budget consistently improves F1 scores across all tasks, with the most pronounced gains observed for smaller budgets. The paper reports this for deepseek-v3.2 on several subdomains (PPT, Word, Excel, FileSystem) with turn limits of 1, 2, 4, 8, 16, and 32.

From Figure 4, reading approximate values:

PPT: F1 rises from roughly 10 at 1 turn to roughly 85 at 32 turns, with the steepest gains between 1 and 8 turns (roughly 10 → 70).
Word: F1 rises from roughly 20 at 1 turn to roughly 80 at 32 turns, with gains continuing steadily through 16 turns.
Excel: F1 rises from roughly 55 at 1 turn to roughly 80 at 32 turns, a more gradual improvement.
FileSystem: F1 rises from roughly 55 at 1 turn to roughly 75 at 32 turns, with most gains achieved by 8 turns (roughly 70).

The diminishing returns are clear: across all subdomains, the F1 curves flatten substantially after 8-16 turns. However, task domains vary in their sensitivity. "Word and PPT tasks benefit more from extended interactions, indicating a greater reliance on iterative information gathering." This is consistent with the GUI domain requiring more exploratory interaction (clicking through slides, checking formatting menus, verifying visual states) compared to FileSystem, where much of the state can be checked with a few directory listings and file reads.

A notable absence: the paper does not report the actual average interaction turn count used in the default (unlimited) setting. It is unclear whether the strong default performance (e.g., deepseek-v3.2 F1 of 83.14 on PPT) uses more than 32 turns on average—if so, the 32-turn budget in the ablation may represent a tighter constraint than it appears. Additionally, the ablation does not measure the cost of environment replay or tool execution time, so "more turns = better" comes with an unquantified computational cost that is not factored into any efficiency metric.

Reasoning Effort vs. Agent-as-a-Judge Performance (Table 4, Section 4.3.1)

The thinking ablation reveals a surprising pattern: increasing reasoning effort does not consistently improve Agent-as-a-Judge performance, and in some cases degrades it.

For gpt-5-mini across three reasoning effort settings:

Low: overall F1 72.41
Medium: overall F1 75.99 (+3.58)
High: overall F1 73.83 (−2.16 vs. medium, +1.42 vs. low)

The medium setting generally outperforms low, but the high setting regresses below medium and is only marginally above low. The pattern is not uniform across subdomains: on DeepSearch, high (79.19) outperforms medium (77.11); on FileSystem, high (71.53) underperforms medium (75.80); on Word, high (63.64) substantially underperforms both medium (72.00) and low (72.22).

For deepseek-v3.2, the comparison is between "no thinking" (the default, F1 77.34) and "thinking" mode (F1 77.11), a negligible −0.23 difference. However, subdomain-level variation exists: thinking mode improves Postgres (74.13 vs. 72.70) and Excel (86.49 vs. 79.71) but degrades Wide (70.37 vs. 72.47) and Deep (79.31 vs. 82.14).

The paper draws the conclusion that "stronger intrinsic reasoning capability is not equivalent to the ability to effectively invoke tools, analyze tool outputs, and make reliable decisions" (Section 4.3.1). This is a non-obvious finding: if agentic judging were simply "reasoning about tool outputs," then more reasoning compute should monotonically improve performance. The U-shaped (or inverted-U) pattern for gpt-5-mini and the flat pattern for deepseek-v3.2 suggest instead that excessive reasoning may interfere with the exploration-exploitation balance needed for effective evidence-gathering—the model may overthink tool selection, produce unnecessarily long chains of reasoning that consume context, or prematurely converge on a judgment without sufficient interaction.

Multimodal Modality Ablation (Figure 5, Section 4.3.3)

The modality study in the GUI domain compares three input configurations—accessibility tree only, screenshot only, and both (mixed)—for two multimodal models (gpt-5-mini-low and gemini-3-flash-preview) across three subdomains.

The headline finding is that no single modality dominates across all subdomains, and the optimal configuration is model-dependent.

Reading approximate F1 values from Figure 5:

PPT: For gpt-5-mini-low, accessibility tree (76.3) and mixed (77.3) are comparable; screenshot (69.8) is weaker. For gemini-3-flash-preview, the pattern reverses: screenshot (91.7) dramatically outperforms accessibility tree (80.0), with mixed (66.7) being worst.
Word: For gpt-5-mini-low, screenshot (86.4) is best, followed by mixed (84.4) and accessibility tree (80.0). For gemini-3-flash-preview, mixed (90.0) is best, followed by accessibility tree (85.7) and screenshot (66.7)—a complete reversal of the gpt-5-mini-low pattern.
Excel: For gpt-5-mini-low, mixed (81.9) is best, followed by accessibility tree (72.2) and screenshot (54.5). For gemini-3-flash-preview, mixed (92.3) is best, with accessibility tree (76.9) and screenshot (80.0) roughly comparable.

The model-dependence is striking: gemini-3-flash-preview achieves F1 91.7 with screenshots alone on PPT, while the same configuration yields only 66.7 on Word—a 25-point swing. gpt-5-mini-low shows the opposite pattern on PPT: screenshots alone yield 69.8, while the tree yields 76.3. This interaction between model, modality, and subdomain means that no universal recommendation ("always use screenshots" or "always use accessibility trees") holds. The paper's conclusion that "incorporating multiple modalities does not uniformly improve agent performance. Instead, mixed inputs can introduce noise and redundancy that may distract the agent and degrade decision-making in certain scenarios" is supported by the PPT result for gemini-3-flash-preview, where mixed (66.7) underperforms both single-modality configurations (91.7 and 80.0).

Framework Comparison: MCPMark vs. ReAct (Appendix A.4, Table 5)

The framework ablation tests whether the advantage of Agent-as-a-Judge depends on the specific agent scaffolding. Two frameworks are compared: MCPMark (characterized as having "more autonomous and implicit reasoning process") and ReAct (which "requires agents to explicitly produce reasoning and actions at each step"). Both use the same base models and the same tools.

For gpt-5-mini-low across five subdomains:

Wide: MCPMark 65.93 vs. ReAct 51.13 (−14.80)
Deep: MCPMark 75.69 vs. ReAct 71.84 (−3.85)
PPT: MCPMark 76.28 vs. ReAct 64.86 (−11.42)
Word: MCPMark 72.22 vs. ReAct 63.64 (−8.58)
Excel: MCPMark 81.89 vs. ReAct 76.47 (−5.42)

For deepseek-v3.2:

Wide: MCPMark 72.47 vs. ReAct 70.51 (−1.96)
Deep: MCPMark 82.14 vs. ReAct 77.88 (−4.26)
PPT: MCPMark 83.14 vs. ReAct 95.24 (+12.10)
Word: MCPMark 78.64 vs. ReAct 75.00 (−3.64)
Excel: MCPMark 79.71 vs. ReAct 85.17 (+5.46)

The key findings: (1) MCPMark generally outperforms ReAct, but the gap varies by model and subdomain—gpt-5-mini-low shows larger ReAct degradation than deepseek-v3.2. (2) There are exceptions where ReAct outperforms MCPMark (deepseek-v3.2 on PPT at +12.10, on Excel at +5.46), suggesting that for certain tasks, explicit step-by-step reasoning helps. (3) Both frameworks consistently outperform the LLM-as-a-Judge baseline for the same base model, confirming that the agentic advantage is robust to implementation details. The paper does not explore why ReAct sometimes outperforms MCPMark—whether it's due to better exploration, more thorough evidence-gathering, or some other mechanism.

Additional Judge Model Results (Appendix A.5, Table 6)

To demonstrate generality beyond the two primary judge models, four additional models are evaluated on a representative subset (Wide, FileSystem, PPT) in both agentic and non-agentic settings:

gemini-3-flash-preview: non-agentic [69.13, 76.54, 78.26] → agentic [72.94, 78.62, 90.48]
claude-sonnet-4.5: non-agentic [61.02, 69.76, 75.61] → agentic [69.87, 62.72, 78.26]
kimi-k2.5: non-agentic [67.40, 58.18, 69.77] → agentic [72.09, 62.03, 80.00]
glm-4.7: non-agentic [64.20, 59.65, 40.00] → agentic [71.96, 63.16, 80.00]

Across all four models and three subdomains (12 comparisons), Agent-as-a-Judge outperforms LLM-as-a-Judge in 10 cases. The two exceptions are claude-sonnet-4.5 on FileSystem (69.76 → 62.72, a −7.04 degradation) and kimi-k2.5 on FileSystem (58.18 → 62.03, a +3.85 improvement that is modest). The FileSystem subdomain appears to be the most fragile for agentic judging—possibly because file system verification can be done adequately from trajectory text, and tool-based re-inspection introduces opportunities for tool-use errors that offset any information gain.

Gemini-3-flash-preview achieves the strongest overall agentic performance on this subset, with PPT F1 of 90.48—the highest single-subdomain score reported anywhere in the paper. This is notable because it exceeds the main evaluation's best PPT score (deepseek-v3.2 at 83.14 by MCPMark, or 95.24 by ReAct in the framework ablation).

Ablation Studies and Robustness Checks

Reasoning effort (Section 4.3.1, Table 4): Increasing reasoning effort from low to medium improves gpt-5-mini Agent-as-a-Judge overall F1 by +3.58 points (72.41 → 75.99), but further increasing to high causes regression to 73.83 (−2.16 from medium). For deepseek-v3.2, enabling thinking mode produces negligible change (77.34 vs. 77.11). This is a negative result: the paper's expectation that "stronger reasoning = better judging" is falsified, implying that effective tool use requires capabilities orthogonal to pure reasoning depth.
Interaction budget (Section 4.3.2, Figure 4): Limiting interaction turns monotonically degrades performance across all tested subdomains. The steepest gains occur at low budgets (1–8 turns), with diminishing returns beyond 16 turns. Word and PPT are more sensitive to turn limits than Excel and FileSystem, consistent with GUI tasks requiring more iterative exploration. Non-obvious implication: the optimal interaction budget is task-dependent, and a fixed budget policy (e.g., "always give the judge 32 turns") would be inefficient—some tasks need fewer turns, while others would benefit from more than 32.
Multimodal input modality (Section 4.3.3, Figure 5): The optimal modality configuration is subdomain-specific and model-specific. Mixed modality (accessibility tree + screenshot) is best for Excel across both models, but in PPT and Word, the best configuration depends on the model. Non-obvious finding: mixed inputs can be worse than single modalities (e.g., gemini-3-flash-preview on PPT: mixed 66.7 vs. screenshot-alone 91.7), suggesting that multimodal fusion is not a solved problem and can introduce noise that degrades judgment.
Agent framework (Appendix A.4, Table 5): Both MCPMark and ReAct implementations of Agent-as-a-Judge consistently outperform LLM-as-a-Judge for the same base model. MCPMark generally outperforms ReAct for gpt-5-mini-low (average gap roughly −8.8 points across five subdomains), but the gap is smaller and inconsistent for deepseek-v3.2 (average gap roughly +1.5 points, with ReAct outperforming on PPT and Excel). This supports the robustness claim: the agentic advantage is not framework-specific.
Judge model family (Appendix A.5, Table 6): Four additional models (gemini-3-flash-preview, claude-sonnet-4.5, kimi-k2.5, glm-4.7) show agentic improvement in 10 of 12 model-subdomain pairs. FileSystem is the subdomain where agentic judging fails to improve for claude-sonnet-4.5 (−7.04 points) and shows only modest gain for kimi-k2.5. Boundary condition: agentic judging is not universally beneficial—it can harm performance when the base LLM-as-a-Judge already performs well from trajectory text alone and tool-use errors offset evidence gains.
Statistical reliability via confidence intervals (Appendix A.6, Table 7): 95% confidence intervals computed from three runs show that Agent-as-a-Judge improvements are statistically distinguishable from LLM-as-a-Judge in subdomains with sufficient sample sizes (DEEP: GPT-5-mini-low CI [66.18, 70.66] vs. Agent [74.89, 76.49], strictly non-overlapping). In smaller subdomains (PPT, WORD), intervals are wider and sometimes overlap, but "the broad upward shift of confidence intervals consistently supports the main conclusion that Agent-as-a-Judge improves subdomain-level performance." Caveat: three runs is a small sample for t-distribution confidence intervals, and the intervals are wide enough in small subdomains that some apparent differences may not be statistically significant at conventional levels.
Failure mode distribution (Appendix A.7, Table 8): Across domains and models, "misinterpretation of tool outputs" (error type c) is the dominant failure mode (30–83% of errors), followed by "incorrect reasoning despite correct evidence" (error type d: 11–68%). "Failure to invoke tools or omission of necessary tool calls" (error type a: 0–27%) and "invocation of incorrect tools" (error type b: 0–3%) are less frequent but non-zero. Practical implication: improving Agent-as-a-Judge requires better interpretation of tool outputs (better understanding of file contents, GUI states, database query results) more urgently than better tool selection or more aggressive tool calling.
Negative result: ReST-like optimization degrades revision models (not in main experiments, but referenced in limitations discussion). The paper notes that an attempt to optimize a revision model with ReST-like training caused performance to degrade (cited in limitations but not presented as a main experiment). This suggests that the positive results depend on specific training choices and may not transfer to online/iterative optimization settings. This is a robustness concern: the gap between offline SFT-based training and online RL-based optimization for judge agents remains unexplored.

Critical Assessment

Claim 1: Agent-as-a-Judge consistently outperforms LLM-as-a-Judge, with an average improvement of 0.13 F1.

This claim is supported with qualifications. The overall average improvement across the two evaluated models is indeed approximately 0.13 (deepseek-v3.2: +12.85, gpt-5-mini-low: +13.41). However, this average masks enormous variance across domains and subdomains: the improvement ranges from +1.78 (gpt-5-mini-low on Postgres) to +31.23 (gpt-5-mini-low on PPT). The claim of "consistency" is weakened by the fact that only two models are evaluated in the agentic setting, and the Appendix A.5 results show that for claude-sonnet-4.5 on FileSystem, agentic judging actually degrades performance (−7.04 F1). The paper's abstract and introduction do not mention this counterexample, giving a somewhat rosier picture than the full data supports.

More fundamentally, the paper never establishes what performance level counts as "good enough" for practical deployment. An F1 of 77.34 (the best result) means that roughly 1 in 4 trajectories is misjudged. For RL reward modeling, this error rate introduces substantial noise into the training signal. The paper does not discuss acceptable error thresholds or compare its F1 numbers to any application-specific requirements, making it difficult to assess whether the ~13-point improvement is practically meaningful or merely statistically significant.

Claim 2: Agent-as-a-Judge built on weaker models can match or exceed LLM-as-a-Judge based on SOTA models.

This claim is supported with specific evidence but is narrower than the abstract suggests. The concrete comparison is: gpt-5-mini-low + tools (F1 72.41) significantly outperforms gpt-5 without tools (F1 61.02), a gap of +11.39 points. This indeed shows that a model positioned as weaker (gpt-5-mini-low has 59.00 non-agentic F1 vs. gpt-5's 61.02—a 2-point gap) can leapfrog a stronger model when given tools. However, gpt-5-mini-low + tools does not exceed the best LLM-as-a-Judge (gemini-3-pro-preview at 75.05) or even the second-best (claude-sonnet-4.5 at 69.77 is within striking distance). The paper's framing "can achieve performance comparable to that of LLM-as-a-Judge based on sota models" is technically true (72.41 is "comparable" to 75.05) but omits that the best LLM-as-a-Judge still exceeds the weaker model with tools. Additionally, deepseek-v3.2—the stronger of the two agentic models—was already the stronger model in the non-agentic setting (64.49 vs. 59.00), so the claim about "weaker models" primarily applies to the gpt-5-mini-low case specifically.

A missing comparison: what would gemini-3-pro-preview—the best LLM-as-a-Judge—achieve as an Agent-as-a-Judge? If the improvement is truly additive (~13 points), it would reach F1 ~88, which would be substantially more impressive than the current best of 77.34. The paper's "budget constraints" rationale for not evaluating stronger models in the agentic setting is understandable but means the ceiling of Agent-as-a-Judge capability is unknown.

Claim 3: The benchmark comprehensively assesses information acquisition, state verification, and process verification.

This claim is partially supported by the benchmark design but not systematically validated. The three verification dimensions are asserted in the introduction, and the three domains are mapped to them (Search → information acquisition, DS → state verification, GUI → process verification), but the paper never demonstrates that these dimensions are actually distinct and measurable. There is no factor analysis showing that performance on Search, DS, and GUI tasks loads onto separate latent factors rather than a single general judging ability. The cross-domain correlation of judge performance is not reported. If a model's F1 on Search correlates at r = 0.9 with its F1 on GUI, the claim of "three distinct verification capabilities" would be weak—the tasks might all be measuring the same underlying skill with different surface forms.

The failure-mode analysis (Appendix A.7) partially addresses this by showing that error types distribute differently across domains, but the mapping between error types and the claimed verification dimensions is not made explicit. Type (c) errors ("misinterpretation of tool outputs") could reflect failures of state verification, information acquisition, or process verification depending on the tool and context, so the error taxonomy does not cleanly validate the capability taxonomy.

Missing experiments that would strengthen the paper:

Human baseline: What F1 do human evaluators achieve on AJ-Bench when given the same tool access? This would contextualize the 77.34 ceiling—is it near human performance (suggesting the benchmark is nearly solved) or far below (suggesting substantial headroom)? The paper mentions human annotation for label creation but never reports human evaluation performance.
Cross-model correlation of agentic vs. non-agentic performance: Do models that are good LLM-as-a-Judges also make good Agent-as-a-Judges? The two models evaluated in both settings (gpt-5-mini-low and deepseek-v3.2) show different relative orderings in the two settings, but with n=2, no correlation can be computed. Evaluating even 4-5 models in both settings would reveal whether the agentic capability is a separate dimension or merely additive to base capability.
Trajectory-length baseline: If successful trajectories are systematically shorter or longer than failed ones (the paper explicitly tried to control for this in GUI trajectory selection), a heuristic baseline that predicts based on trajectory length would reveal how much the benchmark actually requires content-based judgment versus superficial pattern matching.
Cost-normalized comparison: How many tokens/FLOPs does Agent-as-a-Judge consume versus LLM-as-a-Judge? The interaction-turn ablation shows that more turns improve performance, but at what cost? A plot of F1 vs. total inference cost (including tool execution and environment replay) would be more informative than F1 vs. turns, and would allow direct comparison to LLM-as-a-Judge on an efficiency basis.
Transfer learning between domains: Does fine-tuning a judge on Search tasks improve its DS or GUI performance? This would test whether the verification capabilities are domain-specific or transferable—a practically important question for deployment.

Methodological concerns that qualify the results:

Small subdomain sizes: PPT has 21 tasks, Word has 12 tasks, Excel has 19 tasks. With only 2-3 trajectories per task (Table 2: PPT 42 trajectories for 21 tasks = 2 per task), the per-subdomain F1 estimates are based on very small effective sample sizes. The 95% confidence intervals reported in Appendix A.6 confirm this: for GPT-5-mini-low on PPT, the 95% CI is [32.63, 57.47] for LLM-as-a-Judge and [73.53, 79.03] for Agent-as-a-Judge—the LLM interval spans nearly 25 points. Strong claims about subdomain-level patterns (e.g., "PPT shows the largest improvement") should be tempered by this uncertainty.
Label reliability for Search: WideSearch labels are derived from majority voting across six models. If these models share systematic biases (e.g., all being poor at verifying a particular type of claim), the "ground truth" labels will inherit those biases. The paper does not report inter-model agreement rates or compare model-voted labels against human judgments for WideSearch, making the label quality difficult to assess.
Environment drift in Search: Web-based evidence is time-varying. The paper acknowledges this limitation but does not quantify its impact. If a judge navigates to a URL and finds information that differs from what the original agent saw (because the page has updated), the judge may incorrectly label a trajectory as FAIL when the original agent's response was correct at creation time. The frequency of such cases is unknown.
MCPMark vs. ReAct comparison may be confounded: The ReAct implementation forces explicit reasoning at each step, which consumes additional output tokens and context window. The MCPMark implementation has "more autonomous and implicit reasoning." These differ not just in reasoning structure but in effective context utilization—a resource that matters for long trajectories. Without controlling for total token budget or context utilization, the framework comparison is difficult to interpret cleanly.
Single benchmark, no out-of-distribution evaluation: All results are on AJ-Bench. There is no held-out dataset from a different distribution (e.g., agents performing web navigation on different websites, or GUI tasks in applications other than Office) to test whether the observed patterns generalize. This is typical for benchmark-introducing papers but limits the ecological validity claims.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Unaccounted for in Headline Efficiency Gains

The assumption or constraint. The paper's compute-optimal framework for test-time scaling depends fundamentally on knowing each prompt's difficulty before allocating the inference budget. The method for estimating difficulty—sampling 2048 complete solutions per question and computing the pass@1 rate (oracle) or averaging PRM final-answer scores (predicted)—is extraordinarily expensive, consuming far more compute than the largest test-time budgets studied (256–512 generations). The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment where difficulty estimation must be performed per-prompt, the total cost becomes difficulty estimation + strategy execution, and the former can dominate. For example, generating 2048 samples to estimate difficulty for a single MATH problem consumes 8× more compute than the largest test-time budget (256 generations) where the efficiency gains are claimed. If amortized across many problems sharing the same difficulty estimate, the overhead per problem could be reduced, but the paper provides no mechanism for amortization (difficulty estimates are computed per-question and are not shared). The 4× figure should therefore be understood as an upper bound on achievable efficiency in an idealized setting where difficulty is known at zero cost, not as a realized deployment gain.

What evidence exists in the paper. The paper is transparent about this gap (Section 3.2, Limitations section), but provides no quantification of the overhead. Figure 4 and Figure 8 show compute-optimal scaling curves that exclude difficulty estimation cost. The predicted-difficulty variant (using PRM scores instead of ground-truth labels) eliminates the need for oracle access but does not reduce the computational cost—it still requires 2048 samples per question. The paper does not report wall-clock time or FLOPs for difficulty estimation relative to strategy execution.

Mitigation status. The authors flag this as "a key avenue for future work" (Section 3.2) and suggest "pretraining or finetuning models to directly predict difficulty of a question" (Section 8), but no such model is developed or evaluated. They also suggest an exploration-exploitation tradeoff where difficulty estimation is done adaptively: start with a few samples, estimate difficulty, then allocate the remaining budget. This would amortize estimation into the solving process, but it is not implemented. Until cheap difficulty estimation is demonstrated, the compute-optimal framework is a conceptual contribution rather than a deployable system.

6.2 Hard Problems Are Unsolved—Test-Time Compute Cannot Substitute for Missing Capability

The assumption or constraint. The entire framework assumes that the base model has some non-trivial probability of producing correct solutions—that correct answers exist in the proposal distribution to be found via search or refined via revision. On the hardest problems (difficulty bin 5, where base model pass@1 is near zero), this assumption fails. The paper is explicit about this boundary (Section 7):

"Test-time compute amplifies existing capability but does not create it from nothing."

The consequence. Across all methods—search, revisions, and their compute-optimal combinations—the hardest questions show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods (beam search, best-of-N weighted, majority voting) and all budgets (4 to 256 generations). In Figure 7 (right), bin 5 shows roughly 2–3% accuracy for the revision model irrespective of the sequential-to-parallel ratio at 128 generations. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, falling well below the 14× larger model's performance. This means the approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, scaling pretraining remains the only viable strategy.

This boundary condition has practical importance: if a deployment involves a mix of easy, medium, and hard problems, the compute-optimal policy will correctly identify hard problems (via the difficulty estimator) and allocate minimal test-time compute to them, but it will not solve them. The system must either accept failure on these problems or escalate to a larger model—and the paper provides no escalation mechanism.

What evidence exists in the paper. The failure on hard problems is one of the most consistently replicated findings in the paper. It appears in: search results (Figure 3 right, bin 5), revision results (Figure 7 right, bin 5), FLOPs-matched comparison (Figure 9, bin 5), and the difficulty-bin analyses throughout Sections 5 and 6. The qualitative examples in Appendix M further illustrate that on hard problems, increased search budget leads to degenerate outputs (repetitive steps, overly short solutions) rather than correct solutions, consistent with the interpretation that no correct solutions exist to be found.

Mitigation status. The paper does not attempt to solve hard problems. It is transparent about the limitation—the Section 7 takeaway box explicitly states that test-time compute is not a substitute for pretraining on problems outside the base model's reach—but offers no mitigation beyond "scale pretraining instead." The difficulty estimator could serve as an early-warning system to flag hard problems for human intervention or larger-model escalation, but this use case is not explored.

6.3 Single Benchmark, Single Model Family Limits Generalizability

The assumption or constraint. All experiments in the paper use the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* as the base model. The authors state (Section 4) that they "believe this model is representative of the capabilities of many contemporary LLMs," but this claim is unverified. The paper provides no replication on other benchmarks (e.g., GSM8K, MMLU, HumanEval) or other model families (e.g., GPT-4, Claude, LLaMA). The FLOPs-matched comparison involves a second model with ~14× more parameters from the same model family, but its training follows the LLaMA paradigm (scaling parameters only, not data) rather than compute-optimal pretraining—a design choice the paper explicitly acknowledges departs from Chinchilla-optimal scaling (Hoffmann et al., 2022).

The consequence. Several aspects of the findings could be model-specific or benchmark-specific, but the paper cannot distinguish which conclusions generalize:

PRM over-optimization behavior depends on the PRM's calibration properties, which are a function of PaLM 2-S*'s output distribution and the Monte Carlo rollout training procedure. A model with different error patterns or a verifier trained differently could exhibit different difficulty-dependent scaling curves—beam search might degrade at different budget thresholds, or lookahead search might outperform beam search with a better PRM.
Revision model efficacy depends on the base model's in-context learning capabilities, which vary substantially across model families. The finding that sequential revisions help on easy problems but a balanced ratio helps on hard problems might not transfer to models with different revision dynamics.
The FLOPs-matched comparison uses a parameter-only-scaled larger model rather than a Chinchilla-optimal one. A compute-optimally trained larger model (scaling both parameters and data) would likely outperform the baseline used, potentially narrowing or reversing the reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at R << 1).
The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning. The difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems, nothing helping hard problems) might not generalize to code generation, factual QA, or open-ended generation, where the structure of errors and the nature of verifiability differ fundamentally.

What evidence exists in the paper. None. The paper contains no cross-benchmark or cross-model-family experiments. The authors acknowledge the model limitation but frame it as acceptable ("we believe this model is representative"), which is a judgment call not supported by evidence. The dataset limitation is not explicitly discussed as a generalizability concern—Section 4 motivates the choice of MATH (mathematical reasoning benefits from test-time compute because the model possesses the necessary knowledge but struggles with complex inference), but this is a positive argument for using MATH, not an analysis of what might differ on other benchmarks.

Mitigation status. Not addressed. The authors do not claim broader generalizability beyond MATH and PaLM 2-S*, but they also do not caution readers against over-interpreting the results. The paper's title and abstract do not specify this scope limitation, which could lead readers to assume the findings apply to test-time compute scaling in general rather than to math reasoning with a specific model family. The suggestion for future work (Section 8) does not explicitly call for multi-benchmark or multi-model replication, though "extending to other domains" is mentioned implicitly in the context of self-improvement loops.

6.4 The Revision Model Suffers from a 38% Correct-to-Incorrect Reversion Rate

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target answer (Section 6.1). This is a consequence of the training data construction: incorrect answers are paired with correct answers via edit-distance-based matching, and the model learns to generate a correction given incorrect context. The model never sees training examples where the in-context answer is already correct, because there is no "ground truth" for what to do in that case—the correct behavior (leave the answer unchanged) is undefined in the training objective.

The consequence. At inference time, when the revision model generates a chain of revisions, it occasionally produces a correct answer at step t, then at step t+1 conditions on that correct answer and incorrectly revises it to a wrong answer. The paper reports:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach" (Section 6.1)

This means the revision chain is not monotonic—taking the last revision in a chain is unreliable, because later steps may degrade earlier correct answers. The paper mitigates this by using majority voting or verifier-based selection across the entire chain (picking the best answer from any step), but these are post-hoc patches that do not address the root cause. They also introduce additional hyperparameters (how many chains? how long? what selection mechanism?) and computational overhead (the verifier must score every step in every chain).

The reversion problem fundamentally limits how much sequential revision can scale. If 38% of correct answers are converted back to errors at each revision step, then as chains grow longer, the probability that a correct answer survives to the end of the chain decreases, and the verifier-based selection must identify the correct answer from among increasingly many candidates. This creates a tension between the benefit of revisions (improving incorrect answers) and the cost (degrading correct ones) that the paper does not quantify as a function of chain length.

What evidence exists in the paper. The 38% figure is reported in Section 6.1 based on empirical measurement, though the exact measurement protocol (how many chains, across which tasks) is not detailed. Figure 6 (left) shows the revision model's per-step pass@1, which gradually improves from ~18% to ~24% but does not show the fraction of correct answers that revert. The mitigations (majority voting, verifier-based selection) are described in Section 6.1 and evaluated in Figure 6 (right) and Figure 7, but the paper does not report what fraction of the improvement from these mitigations comes from catching reverted answers versus other effects. The ReST $^{EM}$ experiment (Appendix K, Figure 16) shows that reinforcement-learning-based optimization of the revision model causes even worse degradation, suggesting that the reversion problem is exacerbated by on-policy training.

Mitigation status. Partially addressed. Majority voting and verifier-based selection across the chain are effective mitigations—they allow the system to benefit from revisions while avoiding the reversion penalty by selecting the best answer from any point in the chain rather than always taking the final revision. Figure 6 (right) shows that sequential revision with best-of-N weighted selection (which selects the best answer across the chain) achieves ~41.5% F1 versus ~39% for parallel sampling, confirming that the mitigation works. However, these are workarounds, not solutions. A principled approach—such as training the revision model to recognize when no revision is needed, or including "no-change" examples in the training data—is not explored. The paper does not discuss whether the 38% reversion rate could be reduced through training data modifications or architectural changes.

6.5 Serial Dependencies Make the Approach Latency-Bound, But Latency Is Never Measured

The assumption or constraint. The paper measures test-time compute exclusively in "generations"—the number of complete solutions sampled from the model. This is a reasonable proxy for total FLOPs consumed, and it enables the FLOPs-matched comparison in Section 7. However, it ignores wall-clock time and latency, which are often the binding constraints in production deployments.

The consequence. The compute-optimal policies identified by the paper frequently favor sequential operations—revision chains, beam search with iterative expansion, and sequential sampling in general—over fully parallel best-of-N sampling. Sequential operations are inherently latency-bound: each step depends on the output of the previous step, so total wall-clock time scales linearly with the number of sequential steps, regardless of how much parallel hardware is available. In contrast, parallel best-of-N can be executed simultaneously with sufficient compute resources, achieving latency comparable to a single generation.

Concrete implications:

A strategy that allocates 256 generations as a single chain of 256 sequential revisions takes ~256× the wall-clock time of a single generation, which could be minutes to hours depending on the model and hardware. The same 256-generation budget executed as 256 parallel samples takes roughly the latency of one generation.
Revision chains (Section 6) are fully sequential by construction: each revision depends on all previous revisions as context. Even in the hybrid sequential-parallel allocation, the sequential-to-parallel ratio ranges from 2:1 to 8:1 (Figure 7), meaning the majority of the budget is spent on sequential operations.
Beam search (Section 5.2) involves step-by-step expansion with pruning at each step, introducing a serial dependency that best-of-N sampling avoids.

For latency-sensitive applications—interactive assistants, real-time decision-making, chatbots—the sequential-heavy strategies that the compute-optimal policy favors may be practically infeasible regardless of their FLOPs-matched accuracy advantages. A deployment engineer reading this paper would need to know: does the 4× efficiency gain in FLOPs come at a 100× cost in latency? The paper provides no data to answer this question.

What evidence exists in the paper. None. The paper does not report wall-clock time, latency measurements, or any metric that captures the serial vs. parallel execution tradeoff. The generation budget $N$ is treated as a unidimensional compute measure without distinguishing between serial and parallel execution. The FLOPs accounting in Section 7 uses the standard scaling-law formulas $X = 6ND_{\text{pretrain}}$ and $Y = 2ND_{\text{inference}}$ , which capture total floating-point operations but not how those operations are distributed over time.

Mitigation status. Not addressed. The paper does not discuss latency as a constraint, does not differentiate between serial and parallel execution in its cost model, and does not suggest latency-aware compute-optimal policies as future work. This is a significant practical limitation that affects the deployability of the findings, particularly for the revision-based strategies that show the strongest gains on easy problems.

6.6 Verifier Over-Optimization Is Mitigated but Not Solved—It Remains the Fundamental Ceiling

The assumption or constraint. The PRM verifier is trained via Monte Carlo rollout supervision (Section 5.1, Appendix D) and is imperfect. Like any learned model, it has blind spots, calibration errors, and regions of the solution space where its predictions diverge from ground-truth correctness. The paper's search algorithms (beam search, lookahead search) optimize against this verifier signal—they search for solutions that the PRM rates highly. When the PRM's ratings align with true correctness, this search finds genuinely better solutions. When they diverge, the search finds solutions that exploit the verifier's errors.

The consequence. Verifier over-optimization is documented extensively and is the primary reason that scaling test-time compute does not yield unbounded improvements:

Beam search degrades on easy problems (Figure 3, right): on difficulty bin 1, beam search accuracy decreases from ~78% to ~77% as budget increases from 4 to 256, while best-of-N weighted increases from 68% to 88%. The stronger optimizer (beam search) hurts performance at high budgets because it finds solutions that the PRM rates highly but are actually incorrect.
Lookahead search—the strongest optimizer—paradoxically performs worst overall (Figure 3, left): at equivalent generation budgets, lookahead search with k=3 underperforms both beam search and best-of-N weighted, because the extra optimization power (simulating forward to get better step-level scores) amplifies verifier errors rather than finding genuinely better solutions.
Qualitative examples show degenerate outputs (Appendix M, Figures 29, etc.): at high budgets, search produces repetitive low-information steps, overly short 1–2 step solutions, and other outputs that score highly under the PRM but are clearly incorrect to a human evaluator.

The compute-optimal policy mitigates this by routing easy problems away from aggressive search (using best-of-N instead of beam search) and deploying beam search only on medium-difficulty problems where the verifier's guidance provides genuine benefit. But this is a workaround, not a solution to the underlying problem. On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling: the beam search curves in Figure 3 (right, bins 3–4) flatten well before the budget is exhausted, and further budget increases yield minimal or negative returns.

What evidence exists in the paper. The over-optimization evidence is among the strongest in the paper, appearing in:

The difficulty-dependent search analysis (Figure 3 right)
The aggregate search comparison (Figure 3 left, showing lookahead search underperforming)
The qualitative analysis of search outputs (Appendix M)
The difficulty-bin patterns for revisions (Figure 7 right), where over-optimization is less severe because revisions modify the proposal distribution rather than optimizing against a fixed verifier

The paper explicitly identifies verifier over-optimization as a central challenge in Section 8: "improving verifier robustness is the key bottleneck for further scaling test-time compute, not improving search algorithms." This is an important insight derived from the empirical results.

Mitigation status. Partially addressed through the compute-optimal policy, which avoids aggressive optimization where the verifier is most vulnerable (easy problems). The paper does not propose any improvements to the verifier itself—no adversarial training, ensemble methods, KL-regularized search, or other robustness techniques. The suggestion in Section 8 to improve verifier robustness is framed as future work, not as something the paper investigates. This means the fundamental ceiling on test-time compute scaling—verifier quality—remains unaddressed, and the compute-optimal policy gets around it by being selective about when to optimize aggressively, not by raising the ceiling.

This limitation is particularly important because it implies that doubling the test-time compute budget will not double the performance gain unless verifier quality improves commensurately. The compute-optimal framework helps allocate existing budgets more efficiently, but it does not change the asymptotic scaling properties, which are bounded by verifier reliability rather than by search algorithm sophistication.

7. Implications and Future Directions

How This Work Changes the Landscape

AJ-Bench does not propose a new model or algorithm; its contribution is infrastructure for a paradigm shift. Before this work, the research community had no systematic way to measure whether giving judges access to environments and tools actually improves verification—and if so, by how much, under what conditions, and with what failure modes. The paper provides that measurement infrastructure, and in doing so, changes the conversation around verification from "LLM-as-a-Judge vs. human agreement" to "what verification capabilities emerge when judges can interact with the world?"

This is not merely incremental. The paper's central empirical finding—that equipping a weaker model (gpt-5-mini-low) with tools lets it leapfrog substantially stronger models (gpt-5) operating as passive judges, improving overall F1 from 59.00 to 72.41 (+13.41 points, Table 3)—demonstrates that verification capability is not synonymous with base model capability. The agentic setting introduces a new axis of performance that is orthogonal to model scale and reasoning depth. The thinking ablation (Section 4.3.1) reinforces this: stronger reasoning effort does not consistently improve agentic judging, and in some cases degrades it—deepseek-v3.2 with thinking mode achieves 77.11 F1 versus 77.34 without, a negligible difference that contradicts the intuition that "more reasoning = better verification."

The landscape change can be characterized along three dimensions:

1. From text-based judgment to evidence-based verification. The paper reframes verification from a reading comprehension task (does the judge, given the trajectory text, correctly assess success?) to an evidence-gathering task (can the judge navigate the environment, acquire information, and reconcile observed states against task requirements?). This is visible in the failure-mode analysis (Appendix A.7, Table 8), where "misinterpretation of tool outputs" (error type c: 30-83% of failures) dominates over "incorrect reasoning despite correct evidence" (error type d: 11-68%). The bottleneck is not in judging what evidence means but in acquiring and interpreting it correctly—a fundamentally different research problem than improving LLM-as-a-Judge through better prompts or larger models.

2. From uniform verification to domain-grounded verification. The multimodal ablation (Figure 5) demonstrates that no single observation modality dominates: screenshots alone yield F1 ~91.7 for gemini-3-flash-preview on PPT but only ~66.7 on Word. Accessibility trees are best for gpt-5-mini-low on PPT (~76.3) but underperform screenshots on Word (~80.0 vs. ~86.4). The optimal configuration is model-dependent and task-dependent, meaning that verification systems cannot be designed once and deployed universally. This challenges the implicit assumption in the LLM-as-a-Judge literature that a well-trained judge transfers across domains—the data suggests that effective verification requires domain-specific tuning of both the observation modality and the interaction strategy.

3. Reconciles conflicting intuitions about agent-based verification. Prior work on Agent-as-a-Judge (Zhuge et al., 2025; Li et al., 2024; Peng et al., 2025) demonstrated that adding tools can improve verification on specific tasks, but provided no systematic understanding of when, why, or by how much. The pessimistic view—that tool use adds complexity without commensurate benefit because LLMs lack the procedural knowledge to use tools effectively—had no empirical refutation. AJ-Bench provides a clear answer: tools help substantially (average +13 F1), but the gains are uneven across domains (+31.23 on PPT vs. +1.78 on Postgres for gpt-5-mini-low, Table 3), and they depend on sufficient interaction budget (Figure 4) and appropriate modality configuration (Figure 5). The field can now move beyond "do tools help?" to "how do we design tool-use strategies that maximize verification accuracy per unit of interaction cost?"

Research directions that become more attractive:

Exploration strategy learning for judge agents. The interaction-turn ablation (Figure 4) shows that more turns monotonically improve performance, but with diminishing returns. This naturally frames verification as a sequential decision problem: given a budget of K interaction turns, what sequence of tool calls maximizes the probability of a correct judgment? Reinforcement learning or imitation learning from successful human verifiers could train policies that allocate interaction turns efficiently.
Domain-specific verifier training. The domain-dependent improvements (GUI >> Search > DS) and the model-dependent modality effects (Figure 5) suggest that verification capability is not a single general skill. Fine-tuning judge agents on domain-specific trajectories—teaching them which tools to use, what states to check, and what evidence constitutes sufficient verification for a particular task type—could yield substantially better performance than prompting general-purpose models.
Verifier robustness to environment drift. The paper acknowledges (Limitations) that web-based verification suffers from time-varying content. Quantifying how judge agent accuracy degrades as a function of time elapsed since trajectory creation, and developing strategies to handle stale evidence (e.g., checking multiple independent sources, using cached page versions), becomes practically important for deployment.

Research directions that become less attractive:

Pure prompt engineering for LLM-as-a-Judge. The gap between the best LLM-as-a-Judge (gemini-3-pro-preview, F1 75.05) and the best Agent-as-a-Judge (deepseek-v3.2, F1 77.34) is narrow in aggregate, but the gap on GUI tasks—where environment access is most critical—is enormous (+31.23 on PPT for gpt-5-mini-low, Table 3). Prompt improvements to passive judges cannot close this gap because the necessary information (application state, visual layout, file contents) is simply not present in the trajectory text. Investing in better prompts for LLM-as-a-Judge on interactive tasks has a hard ceiling that agent-based approaches do not.
Scaling model size as the primary path to better verification. The paper's comparison between model families (Table 3) shows that the largest models do not dominate: gpt-5 (the flagship model at the time) achieves only 61.02 F1 as an LLM-as-a-Judge, below gemini-3-pro-preview (75.05), claude-sonnet-4.5 (69.77), and grok-4 (69.24). In the agentic setting, deepseek-v3.2 (an open-source model) outperforms gpt-5-mini-low by 4.93 points (77.34 vs. 72.41). Scale alone is not the answer; tool-use capability, interaction strategy, and domain grounding matter at least as much.

Follow-Up Research This Work Enables

Adaptive difficulty estimation for judge agents. The paper currently treats all trajectories as equally difficult to verify, but the wide variance in subdomain performance (PPT: 45.05 → 76.28 F1 vs. Postgres: 65.52 → 67.30 for gpt-5-mini-low, Table 3) suggests that some tasks are inherently harder to verify than others. A natural extension is to train a lightweight classifier that predicts, from the task description and trajectory summary alone, how many interaction turns a judge will need and which tools are likely to be useful—analogous to the difficulty estimator in the test-time compute paper (Snell et al., 2024) but applied to verification rather than problem-solving. The experiment would measure whether a turn-budget allocation policy conditioned on predicted verification difficulty can achieve higher aggregate F1 than a uniform budget, and would use the subdomain-level performance variance in AJ-Bench as training signal for the difficulty predictor.

Cross-domain transfer of verification capability. The paper evaluates each model independently on each subdomain, but does not measure whether verification skill transfers across domains. A critical follow-up would fine-tune a judge agent on one domain (e.g., FileSystem tasks with ground-truth labels and environment access) and evaluate zero-shot performance on other domains (e.g., Postgres, GUI). If transfer is strong, it suggests that the agent is learning general evidence-gathering strategies rather than domain-specific heuristics. If transfer is weak, it implies that verification is fundamentally domain-grounded and that production systems need per-domain training. The experiment would use the 155-task, 516-trajectory dataset with a leave-one-domain-out cross-validation design, measuring F1 on the held-out domain after training on the remaining two domains with full tool access. The subdomain-level breakdown (Table 2) provides enough tasks per domain (Search: 61, DS: 42, GUI: 52) for this to be statistically meaningful.

Human baseline for judge agent performance. The paper constructs ground-truth labels through human annotation (Search/Mind2Web2, GUI), model-based majority voting (Search/WideSearch), and verifier scripts with manual validation (DS, GUI). But it never reports what F1 score a human evaluator achieves when given the same task description, trajectory summary, and tool access as the judge agent. Establishing a human performance ceiling would contextualize the 77.34 F1 achieved by deepseek-v3.2—if humans achieve 95+ F1, there is substantial headroom for improvement; if humans achieve 80-85 F1, the current models are approaching the practical ceiling. The experiment would recruit annotators (possibly the same team that created the labels, to control for domain expertise), give them the MCPMark tool interface (or a simplified version), and measure their F1 on a representative subset of 30-50 trajectories spanning all three domains. This baseline would also reveal whether certain failure modes (e.g., error type c, "misinterpretation of tool outputs") are uniquely AI weaknesses or shared human limitations.

Cost-normalized comparison of agentic vs. non-agentic verification. The interaction-turn ablation (Figure 4) shows that more turns improve F1, but the paper never reports the token cost or wall-clock time of agentic judging. A follow-up study would measure total inference FLOPs or API cost for both LLM-as-a-Judge and Agent-as-a-Judge on the same tasks, producing an F1-per-dollar or F1-per-FLOP efficiency metric. The experiment would instrument the MCPMark framework to log total input tokens, output tokens, tool-call overhead, and environment replay time (VM provisioning for GUI, file system initialization for DS, network latency for Search). The key question: does Agent-as-a-Judge's +13 F1 improvement justify its additional computational cost, or would spending the same budget on a larger LLM-as-a-Judge (e.g., using gpt-5 instead of gpt-5-mini-low) yield comparable or better F1 at lower cost? This directly tests the paper's implicit claim that agentic judging is not just more accurate but more efficient.

Adversarial trajectory construction to stress-test verifier robustness. The paper's trajectories come from real agents with genuine success/failure patterns. A follow-up could construct adversarial trajectories—sequences of actions that produce a final state that appears successful (e.g., a file with the correct name exists, a GUI element appears in the right location) but that violate task requirements in subtle ways (e.g., the file was created via an incorrect sequence of operations, the GUI element was placed there before the task began). These would test whether judge agents are genuinely verifying process (checking that the correct operations were performed in the correct order) or merely checking final state (confirming that the environment looks right at the end). The experiment would measure the F1 gap between real trajectories and adversarial ones for the same judge agent—a large gap would indicate that judges are superficial state-checkers rather than thorough process verifiers, exposing a fundamental limitation of the paradigm.

Online reinforcement learning for judge agent training. The paper's judge agents use fixed prompts with static tool sets; they are not trained to improve their verification strategies through experience. A natural extension is to frame verification as a reinforcement learning problem: the judge agent receives reward +1 for correct judgments and −1 for incorrect judgments (compared to ground truth), and learns a policy over tool-calling actions that maximizes expected judgment accuracy. This would directly address the failure modes identified in Appendix A.7—particularly error types (a) "failure to invoke tools" and (c) "misinterpretation of tool outputs"—by providing gradient signal about which tool-use patterns lead to correct judgments. The experiment would use AJ-Bench's labeled trajectories as a training set, reserving a portion for held-out evaluation, and compare an RL-fine-tuned judge agent against the prompted baseline. The key measurement would be whether RL improves F1 beyond what additional interaction turns alone provide (Figure 4), and whether the improvement transfers to unseen tasks within the same domain.

Practical Applications and Downstream Use Cases

Automated reward modeling for RL-based agent training. The paper explicitly motivates Agent-as-a-Judge in the context of scaling reinforcement learning for LLM agents (Section 1). In RL pipelines like Agent-R1 (Cheng et al., 2025) or broader RL-for-agents frameworks (Chen et al., 2025), the reward signal is the bottleneck: if the reward model cannot reliably distinguish successful trajectories from failed ones, the policy gradient is noisy and training efficiency collapses. AJ-Bench provides an off-the-shelf evaluation framework to benchmark candidate reward models before deploying them in an RL loop. A practitioner training a GUI agent could: (1) collect a set of agent trajectories on their target application, (2) label them using AJ-Bench's methodology (verifier scripts with manual validation, or model-based majority voting for factual tasks), (3) evaluate their candidate reward model (LLM-as-a-Judge vs. Agent-as-a-Judge with their application's tool set) on this labeled set, and (4) select the verifier that achieves the highest F1 for use as the RL reward signal. The paper's finding that Agent-as-a-Judge improves F1 by ~13 points over LLM-as-a-Judge on average, and by up to 31 points on GUI tasks (Table 3), directly translates to cleaner RL gradients and faster policy improvement. For a GUI agent training run with 10,000 trajectories, a 31-point F1 improvement means roughly 3,100 fewer mislabeled trajectories providing incorrect reward signal—a substantial reduction in noise.

Auditing and monitoring deployed agent systems. Organizations deploying LLM agents in production—for automated data processing, customer support, or document manipulation—need to audit agent behavior to detect failures, errors, and policy violations. AJ-Bench's methodology provides a blueprint for building automated auditing systems: replay agent trajectories in isolated environments, deploy judge agents with domain-specific tools to verify correctness, and flag trajectories where the judge disagrees with the agent's self-reported success for human review. The paper's interaction-turn ablation (Figure 4) is directly actionable here: it shows that even 8-16 interaction turns yield most of the achievable F1 gain, meaning auditors can operate with reasonable efficiency. For a deployment processing 10,000 tasks per day, an auditor using deepseek-v3.2-level Agent-as-a-Judge at ~77 F1 would correctly identify ~7,700 task outcomes and flag ~2,300 for human review—a 77% reduction in manual auditing load compared to checking every trajectory. The ability to tune the verification budget per task (using more turns for high-stakes tasks, fewer for routine ones) makes this practically deployable even under latency constraints.

Data filtering for self-improvement and distillation pipelines. When using LLMs to generate training data for fine-tuning (e.g., STaR, ReST, rejection sampling), the quality of the generated data depends on the reliability of the filtering mechanism that separates correct from incorrect outputs. AJ-Bench demonstrates that Agent-as-a-Judge can serve as a high-quality filter, particularly for GUI and search tasks where LLM-as-a-Judge is unreliable (F1 as low as 45.05 on PPT, Table 3). A practitioner generating fine-tuning data for a PowerPoint manipulation agent could: (1) run their base agent on a large set of tasks, (2) use Agent-as-a-Judge to evaluate each trajectory, (3) retain only trajectories judged as PASS for supervised fine-tuning, and (4) optionally retain judged-FAIL trajectories with their judge agent's reasoning as negative examples for DPO training. The +31.23 F1 improvement on PPT from Agent-as-a-Judge over LLM-as-a-Judge (Table 3) means the filtered dataset would contain substantially fewer false positives (incorrect trajectories mislabeled as successes), which directly improves the quality of the resulting fine-tuned agent. The paper's supplementary metrics (Appendix Table 9) quantify this: Agent-as-a-Judge on gpt-5-mini-low achieves False Positive Rate of 15.88% on PPT vs. 14.29% for LLM-as-a-Judge (similar), but False Negative Rate improves from 66.67% to 28.57%—meaning far fewer correct trajectories are mistakenly discarded, preserving more training data.