Evaluation2026~104 minarXiv:2604.10866

OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language World Models

score 8Xiaomeng Hu, Yinger Zhang, Fei Huang +7Qwen↑ 58★ 14Apr 13, 2026cs.CLhttps://github.com/GregxmHu/OccuBench

🎯 Pitch

Implicit data faults (truncated fields, missing values) defeat AI agents more thoroughly than overt errors like timeouts, because they offer no error signal to trigger recovery. OCCUBENCH reveals that no single model excels across all 65 professional domains—each has a unique occupational fingerprint invisible to existing benchmarks—and that stronger simulators are essential for reliable evaluation, as weaker ones produce inconsistent rankings.

1. Executive Summary

This paper introduces OCCUBENCH, a benchmark covering 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains, enabled by Language Environment Simulators (LESs)—LLM-driven simulators that generate domain-specific tool responses for evaluation without requiring real environment infrastructure. The benchmark evaluates 15 frontier models along two complementary axes: task completion across professional domains and environmental robustness under controlled fault injection, including explicit errors (timeouts, 500s), implicit data degradation (truncated data, missing fields), and mixed faults. The central finding is that no single model dominates all industries—each exhibits a distinct occupational capability profile invisible to single-domain benchmarks—while implicit faults prove harder than explicit ones because they lack overt error signals, establishing that agent evaluation must consider cross-industry specialization and environmental resilience rather than aggregate scores alone. GPT-5.2 achieves the highest overall completion rate (79.6%) and improves 27.5 points from minimal to maximum reasoning effort, while pairwise ranking agreement between capable simulators reaches 85.7%, confirming that LES-based evaluation is reliable only when the simulator is sufficiently capable.

2. Context and Motivation

The Core Problem: Professional AI Agents Are Untestable in the Domains That Matter Most

This paper addresses a fundamental evaluation gap in AI agent research: the professional domains where autonomous agents would deliver the highest value—healthcare, finance, legal, manufacturing, energy, governance, logistics—are precisely the domains where no benchmarks exist to evaluate them. The authors frame this not as a minor limitation but as the default state for the vast majority of professional work, which they term the "Untestable Majority."

The problem is structural, not incidental. It arises from a mismatch between how agent benchmarks are built (by deploying or accessing real environments) and how professional work is conducted (within closed, proprietary, safety-critical systems). An agent benchmark requires an interactive environment where the model can take actions—tool calls, API requests, state queries—and receive feedback. But for emergency department triage, nuclear reactor monitoring, customs import processing, or greenhouse irrigation control, no public environment exists, and building one is either practically impossible (proprietary systems with no external API access) or prohibitively dangerous (irreversible real-world consequences).

This is not a collection of edge cases. As the paper states in Section 1:

"This is not a collection of edge cases; it is the default state for the vast majority of professional work."

The result is a severe evaluation blind spot. The research community can evaluate agents on web browsing (WebArena), desktop operations (OSWorld), code repositories (SWE-bench), and a handful of API domains like retail and airline customer service (TAU-bench)—but cannot answer basic questions about whether an AI agent could competently triage a patient, audit a financial report, schedule a factory production line, or respond to a wildfire evacuation alert. These are the highest-stakes, highest-value applications of agent technology, yet they are systematically excluded from evaluation.

Why This Problem Matters: Real-World Impact and Unanswered Questions

The paper motivates this gap through a series of specific, practical questions that no existing benchmark can address (Section 1):

"• Can an agent triage patients in an emergency department? No public environment exists. • Can an agent manage a nuclear reactor safety alert? No benchmark covers this. • Can an agent process customs import declarations? No API is available. • Can an agent control greenhouse irrigation based on sensor data? No testbed exists."

These are not hypotheticals. Each maps to a real human job role with significant economic and safety implications. The inability to evaluate agents in these domains has several downstream consequences:

Deployment risk is unquantified. Organizations considering whether to deploy AI agents for professional work have no principled way to assess whether current models are ready. An agent might score 90% on existing web-browsing or code-editing benchmarks while being entirely incapable of handling the state tracking, domain knowledge, error recovery, and multi-step reasoning required for professional work. The benchmarks cannot tell you this.

Research direction is skewed. Because benchmarks drive research attention, the dominance of web, code, and OS benchmarks means the agent research community primarily optimizes for capabilities in those visible domains. The "untestable" domains—despite representing the bulk of economic value—receive disproportionately little research attention because there is no measurement apparatus to drive iterative improvement.

Model selection is uninformed. Organizations deploying agents must choose which model to use for which task. But single-domain benchmarks produce a single aggregate ranking that obscures occupational specialization. A model that excels at code generation may struggle with healthcare protocol compliance, and a model strong in business operations may stumble on transportation logistics. Without cross-industry evaluation, these blind spots are invisible to practitioners.

Environmental robustness is ignored entirely. Even within the narrow set of domains that have benchmarks, all existing evaluations operate on the "happy path"—perfectly functioning environments with no timeouts, no degraded data, no service interruptions. Real professional environments are noisy, unreliable, and adversarial. An agent that works flawlessly in a clean benchmark may catastrophically fail when an API returns a 500 error or a sensor reports truncated data. Without systematic fault injection, deployment readiness cannot be assessed.

Prior Approaches and Their Structural Limitations

The paper identifies several categories of existing agent benchmarks and explains why each is structurally unable to address the professional domain gap.

Web environments: WebArena (Zhou et al., 2024) deploys real websites for browser-based tasks; VisualWebArena (Koh et al., 2024) extends to multimodal web interaction; Mind2Web (Deng et al., 2023) covers cross-domain web tasks; WorkArena (Drouin et al., 2024) targets enterprise knowledge work on the ServiceNow platform; BrowseComp (Wei et al., 2025) evaluates deep web navigation. These benchmarks are tied to the web paradigm—they evaluate whether an agent can navigate, click, type, and reason about web content. They cover zero professional domains that operate through specialized, non-web interfaces (hospital management systems, industrial control panels, customs databases).

OS and mobile environments: OSWorld (Xie et al., 2024) provides full operating system virtual machines; AndroidWorld (Rawles et al., 2025) benchmarks mobile app automation; MobileBench (Deng et al., 2024) evaluates mobile agents; Terminal-Bench (Merrill et al., 2026) tests agents in real terminal environments. These evaluate desktop and mobile operations but not domain-specific professional workflows. Being able to use a file system does not demonstrate the ability to triage patients or monitor nuclear safety systems.

Code environments: SWE-bench (Jimenez et al., 2024) evaluates repository-level issue resolution; InterCode (Yang et al., 2023) provides interactive coding with execution feedback. These evaluate software engineering capability, which is one narrow slice of professional work.

Tool and API environments: TAU-bench (Yao et al., 2024) evaluates tool-agent-user interaction in retail and airline domains; BFCL (Patil et al., 2025) benchmarks function calling; AgentBench (Liu et al., 2024) covers 8 distinct environments; ToolLLM (Qin et al., 2024) evaluates across 16,000+ real-world APIs; GAIA (Mialon et al., 2023) tests general assistant capabilities; MINT (Wang et al., 2024) evaluates multi-turn tool interaction; and MCP-Bench (Wang et al., 2025), MCP-Atlas (Bandi et al., 2026), MCPMark (Wu et al., 2025), and Toolathlon (Li et al., 2026) benchmark tool-use competency through real MCP servers. The TAU-bench domains (retail, airline) involve real APIs, but only in two narrow domains. ToolLLM's 16,000 APIs and the MCP-based benchmarks cover more breadth, but the APIs are general-purpose public services (weather, news, translation), not professional domain-specific systems (hospital management, customs processing, industrial control).

The paper identifies four shared limitations across all existing benchmarks (Section 2):

"1. environments require substantial engineering to construct and maintain; 2. test sets are static and vulnerable to data contamination; 3. no systematic environmental robustness evaluation; and most critically, 4. domain coverage is extremely limited: all existing benchmarks combined cover only web browsing, code editing, desktop operations, and a handful of API domains, leaving the vast majority of professional occupational tasks untestable."

The fourth point is the central one. The engineering overhead of building and maintaining real environments creates a hard scalability wall. Adding a new domain to WebArena requires deploying and configuring entire web applications—a multi-week engineering effort per domain. Extending TAU-bench requires integrating new real APIs or manually writing simulators. This makes scaling to dozens or hundreds of professional domains practically infeasible, which is exactly why the existing benchmarks collectively cover perhaps a dozen domains while hundreds of professional occupations remain untestable.

Existing Evaluations of Professional Work

The paper acknowledges several recent benchmarks that target economically valuable professional work and positions OCCUBENCH as complementary to them.

GDPVal (Patwardhan et al., 2025) covers 44 occupations across 9 industries with 1,320 tasks graded by industry experts, focusing on output-quality tasks—writing legal briefs, creating presentations, generating reports. These assess the quality of a static deliverable, not the interactive decision-making process that produces it.

$OneMillion-Bench (Yang et al., 2026) evaluates 400 expert-curated tasks across Law, Finance, Industry, Healthcare, and Natural Science, where each task is assigned a monetary value based on senior professional hourly rates. This benchmark measures deliverable quality through rubric-based grading.

TheAgentCompany (Xu et al., 2025) evaluates agents as digital workers performing consequential real-world tasks, and SWE-Lancer (Miserendino et al., 2025) maps agent performance to monetary value through 1,400+ real freelance software engineering tasks. These focus on software-adjacent professional work.

Claw-Eval (Ye et al., 2026) introduces an end-to-end evaluation suite of 300 human-verified tasks spanning 9 categories with trajectory-aware grading over 2,159 fine-grained rubric items, evaluating completion, safety, and robustness of autonomous agents.

CL-bench (Dou et al., 2026) evaluates models' ability to learn from task-specific context containing new knowledge beyond pre-training, covering 500 complex contexts with 1,899 tasks. While CL-bench tests context-dependent reasoning, OCCUBENCH tests context-dependent action—agents must not only understand domain-specific contexts but execute multi-step tool-use workflows within them.

The paper positions OCCUBENCH as distinct from all of these along a key dimension: it measures interactive decision-making across 65 specialized domains, requiring multi-step tool use, state tracking, and error handling in stateful environments, rather than measuring static output quality. A model might produce an excellent written legal brief (evaluable by GDPVal) while being incapable of the interactive process that would typically produce it in a real law firm—consulting case management systems, querying statutes, coordinating with colleagues. OCCUBENCH evaluates that interactive process.

The Prohibitive Engineering Cost of Scaling Benchmarks

The paper identifies a prohibitive scaling cost as the root cause of limited domain coverage. Even within covered domains, each benchmark is constrained by its environment implementation. Adding a new domain requires substantial engineering:

For WebArena: deploying and configuring entire web applications, including setting up databases, mock users, and realistic data
For TAU-bench: integrating new real APIs or manually writing detailed simulators that encode domain-specific business logic
For OSWorld/AndroidWorld: configuring VMs or emulators with the appropriate software and data

This engineering overhead makes scaling to hundreds of professional domains practically infeasible under the traditional benchmark construction paradigm. Each new domain is a new engineering project.

The paper's key observation is that this entire paradigm can be circumvented. The bottleneck is not that professional domains are inherently unmodelable—it is that building physical or software environments for them requires manual engineering. If the environment itself can be simulated by an LLM, then domain coverage becomes a configuration problem rather than an engineering problem.

The No Robustness Evaluation Gap

A separate but equally important motivation is the absence of environmental robustness testing in existing benchmarks. The paper identifies this as a gap that is "critical for production deployment decisions" (Section 1). Real-world deployments encounter:

Explicit errors: API timeouts, HTTP 500 errors, connection refusals, service unavailability—failure modes with clear error signals where the agent at least knows something went wrong
Implicit degradation: Truncated data, missing fields, incomplete lists, stale cached values, null/empty fields—failure modes with no error signal where the response appears superficially correct
Mixed faults: Real environments combine both types

Existing benchmarks evaluate agents exclusively on clean environments. This means a model that retries gracefully on errors, detects data truncation, and verifies response completeness receives no credit for these capabilities, while a brittle model that crashes on the first timeout receives no penalty. For organizations deciding whether to deploy an agent in production, this is precisely the information they need and precisely what existing benchmarks cannot provide.

How This Paper Positions Itself

The paper introduces Language Environment Simulators (LESs) as the enabling technology that addresses both the domain coverage gap and the robustness evaluation gap. The core insight is elegant:

"Given a configuration c = (system prompt, tool schema, initial state, state description), an LLM becomes a stateful, interactive environment simulator... This transforms environment construction from an engineering problem into a configuration problem, extending benchmark coverage from 'domains with public environments' to 'any domain an LLM can understand.'" (Section 1)

Rather than building real environments or manually coding simulators for each domain, the paper defines each environment through a declarative configuration that an LLM reads and follows to generate tool responses. This approach has several key properties that position it as a genuine solution rather than a shortcut:

Scalability: Adding a new professional domain requires writing a configuration (system prompt, tool schema, initial state, state description), not building or deploying software. This enables coverage of 100 scenarios across 65 domains—orders of magnitude more than any prior benchmark.
Realism through domain knowledge: LLMs encode operational logic for hundreds of professional domains through their pre-training on documentation, protocols, and procedural descriptions. The configuration constrains this knowledge to maintain causal consistency (state transitions, inventory management, patient queues) across multi-turn interactions. The paper argues (Section 3.3) that LLMs have the necessary format priors (API documentation), domain knowledge (hospital triage protocols, network firewall rules), state maintenance ability (through the combination of system prompt constraints and in-context tracking), and edge-case handling (generating reasonable error responses for out-of-bounds parameters).
Robustness evaluation by design: Because the LES is software rather than a fixed environment, fault injection becomes a configuration change—appending fault rules to the system prompt at evaluation time—rather than requiring separate environment deployments. This enables the paper's systematic evaluation of explicit, implicit, and mixed faults across all 100 scenarios.
Solvability guarantees through the multi-agent synthesis pipeline: The paper does not simply ask an LLM to generate tasks. It employs a multi-agent synthesis pipeline (Section 4) that generates environment configurations, task instructions, tool definitions, solution plans, and verification rubrics; executes each task multiple times with and without a reference plan to verify solvability and calibrate difficulty; uses a majority-vote verifier to assess trajectories against rubrics; and includes a repair module that diagnoses and fixes failures. Tasks that are trivially easy (100% autonomous success), unsolvable (0% success), or have invalid tool schemas are filtered out. This ensures the 382 evaluation instances that make it into OCCUBENCH are genuinely solvable, discriminative, and diverse.

The paper draws a connection to work on LLMs as world models (Section 2), including Gu et al. (2024)'s finding that LLMs can serve as world models of the internet for web agent planning, WebWorld (Xiao et al., 2026)'s training of open-web simulators at scale, ViMo (Luo et al., 2025)'s generative visual world models for GUI agents, and Chen et al. (2025)'s self-play approach to internalizing world models for agentic RL. However, the paper explicitly distinguishes its LES approach as occupying "a distinct niche: using LLMs to simulate tool-response-level environment interaction for evaluation rather than training, supporting stateful multi-step professional tasks with realistic action spaces across 100 scenarios and 65 specialized domains." Prior world-model work focuses on training (generating training data for agents) or planning (simulating forward rollouts to plan actions). OCCUBENCH repurposes the simulation capability for evaluation—and, critically, at a breadth (65 domains) and with systematic fault injection that no prior benchmark or simulator achieves.

In summary, the paper positions itself as solving a problem that is simultaneously obvious, widespread, and previously intractable: the inability to evaluate AI agents on the professional tasks that matter most. It does so not by incrementally extending existing benchmarks but by changing the fundamental approach to environment construction, from engineering to configuration, exploiting LLMs' pre-existing domain knowledge as the simulation engine. The resulting benchmark, OCCUBENCH, is the first to provide systematic cross-industry evaluation with environmental robustness testing, enabling findings—occupational specialization profiles, implicit fault vulnerability, scaling patterns—that no prior benchmark could reveal.

3. Technical Approach

3.1 Reader Orientation

OCCUBENCH is an evaluation benchmark consisting of 100 interactive professional task scenarios, each defined by a configuration of system prompt, tool schema, initial state, and state description, where an LLM acting as a Language Environment Simulator (LES) generates tool responses during evaluation. The system solves the fundamental problem of evaluating AI agents in professional domains that lack public environments by replacing real environment infrastructure with LLM-driven simulation—transforming environment construction from an engineering problem (building software, deploying APIs) into a configuration problem (writing descriptions that an LLM interprets).

3.2 Big-Picture Architecture (Diagram in Words)

The OCCUBENCH system has five major components, organized into a synthesis pipeline (which creates evaluation instances) and an evaluation loop (which tests agents):

Scenario Design Module — defines 100 professional job roles across 10 industry categories and 65 domains, each with 16 non-overlapping sub-topics and a professional reference document covering domain terminology, workflows, state variables, edge cases, and constraints.
Multi-Agent Synthesis Pipeline — powered by Gemini-3-Flash-Preview as the LES, this pipeline generates environment configurations (system prompt, tool schema, initial state, state description), task instructions, solution plans, and verification rubrics for each scenario. It executes each task multiple times, verifies solvability, calibrates difficulty, and filters out unsolvable or trivially easy instances.
Language Environment Simulator (LES) — the LLM that, given an environment configuration and conversation history, generates the next observation $o_{t+1}$ in response to the agent's tool call $a_t$ . This is the central enabling technology: it replaces real environments at evaluation time.
Fault Injection Module — at evaluation time, appends fault rules to the LES's system prompt, causing it to inject explicit errors (timeouts, 500s), implicit degradation (truncated data, missing fields), or mixed faults at controlled frequencies and durations.
Evaluation Loop — the agent under test issues tool calls; the LES generates observations; the complete trajectory is scored by an automated rubric-based verifier against pre-defined success criteria.

Information flows as follows: scenarios are designed → the synthesis pipeline generates environment configurations, task instances, and rubrics → quality filtering removes unsolvable or trivially easy instances → the evaluation set of 382 instances is fixed → at evaluation time, an agent model iteratively calls tools, the LES generates responses (with optional fault injection), and the trajectory is verified.

3.3 Roadmap for the Deep Dive

First, the Language Environment Simulator formalization (Equation 1), because it is the foundational abstraction that enables everything else—understanding what an LES is computationally and why LLMs can serve as simulators is prerequisite to understanding the benchmark construction and evaluation.
Second, the environment configuration structure (system prompt, tool schema, initial state, state description), since each component encodes a specific aspect of the domain simulation and constrains the LES's behavior.
Third, the multi-agent synthesis pipeline, which transforms scenario designs into solvable, verifiable, discriminative, and diverse evaluation instances—this is the "factory" that produces the benchmark and ensures quality.
Fourth, the evaluation loop and automated verification, since they define how agents interact with the LES and how success is determined.
Fifth, the environmental fault injection framework, because it is a separate module that operates at evaluation time, modifies the LES's behavior through system prompt augmentation, and defines the four fault conditions (E0–E3).
Sixth, the scenario design principles and dataset composition, since they provide the scope, coverage, and scale of the benchmark.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark construction and evaluation methodology paper whose core idea is that LLMs can serve as interactive environment simulators for professional domains, enabling scalable evaluation of AI agents across domains where no public environments exist, without building or deploying any real software infrastructure.

The Language Environment Simulator (LES) Formalization

The LES is the enabling abstraction. It is defined as a function:

$(s_{t+1}, o_{t+1}) = f_\theta(s_t, a_t; c)$

where $f_\theta$ is the LLM parameterized by $\theta$ , $c$ is the environment configuration (a tuple of system prompt, tool schema, initial state, and state description), $s_t$ is the latent environment state at time $t$ (maintained implicitly by the LLM through its context window), $a_t$ is the agent's action at time $t$ (a structured tool call with a function name and typed arguments), and $o_{t+1}$ is the observation returned to the agent (a structured JSON tool response). The function produces both the next latent state $s_{t+1}$ and the next observation $o_{t+1}$ .

What it computes: given a configuration describing a professional environment, a conversation history representing the current state, and an agent's tool call, the LES generates the next tool response and updates its internal state representation. The key difference from traditional world models is that the LLM does not learn a state transition function from data—it leverages its pre-trained knowledge of domain-specific operational logic, constrained by the configuration $c$ to maintain causal consistency across interactions.

Operationally: at each evaluation step, the LES receives an agent's tool call (e.g., discharge(patient_id="P-110", room="Exam Room 2")), consults its system prompt (which defines the simulation rules for that domain), its tool schema (which defines valid function signatures and output formats), its initial state (which defines starting conditions like room inventory or patient queues), and its state description (which annotates how state variables should evolve), and generates a JSON response (e.g., {"status": "success", "message": "Patient P-110 discharged, Exam Room 2 vacated"}). This response is appended to the conversation history, which becomes the new state $s_{t+1}$ .

Why this form: the LES abstracts away the implementation of the environment. A real emergency department management system would require a database, authentication, business logic, and a UI. The LES replaces all of this with an LLM prompt that encodes the same operational logic declaratively. The functional form $f_\theta(s_t, a_t; c)$ emphasizes that the LES is deterministic given its configuration and history—it is not learning or adapting during evaluation, only simulating. The separation of $c$ from the function body is critical because it is $c$ that encodes domain-specific behavior; the same LLM $f_\theta$ can simulate an emergency department or a customs office or a greenhouse simply by changing $c$ .

Environment Configuration Structure

Each LES environment is fully specified by four components, which together define the simulation contract that the LLM must follow (Section 3.2):

System Prompt: This encodes the environment's behavioral rules, simulation logic, error handling protocols, and output format constraints. It tells the LLM how to behave as the environment. For example, a hotel revenue management environment's system prompt specifies pricing rules, occupancy calculations, and the relationship between ADR (Average Daily Rate), CPOR (Cost Per Occupied Room), and revenue metrics. The system prompt is the most critical component because it constrains the LLM's vast but unstructured domain knowledge into a specific, consistent simulation.

Tool Schema: This defines the agent's action space as a set of callable functions with typed parameters and example outputs. Each environment contains 2–10 tools (median 5) reflecting realistic operational interfaces. The tool schema specifies the function names the agent can call, the expected parameter types, and example output formats (which the LES uses as templates for generating responses). It also indirectly serves as the task specification for the agent—the agent knows what tools are available and must decide which to call, in what order, with what arguments.

Initial State: This is a structured JSON object specifying the environment's starting conditions. For example, in an emergency department scenario, the initial state might specify which patients are in which rooms, their vital signs, which rooms are available, and what protocols are active. In a network operations scenario, it might specify the network topology, which devices are up/down, and the current alert queue. The initial state grounds the simulation in concrete, specific conditions rather than generic domain knowledge.

State Description: These are semantic annotations for each field in the initial state, guiding the LLM to maintain causal consistency across multi-turn interactions. For example, for a room inventory field, the state description might specify: "remaining inventory decreases after each booking; when inventory reaches zero, no more bookings can be made for that room type." Without state descriptions, the LLM might generate plausible but causally inconsistent responses (e.g., booking a room that should be full, forgetting a patient was discharged, or allowing duplicate assignments). State descriptions are the mechanism that converts the LLM's generic knowledge of a domain into a specific, stateful simulation that respects causality.

Why these four components: together they form a complete specification of a stateful interactive environment in purely declarative form. The system prompt provides the behavioral contract; the tool schema provides the action interface; the initial state provides the concrete starting conditions; and the state description provides the causal constraints that maintain consistency. No code is written, no databases are deployed, no servers are configured. This is what makes the LES scalable: each new professional domain requires only a carefully written configuration, not an engineering project.

Why LLMs Can Serve as Language Environment Simulators

Section 3.3 provides a four-part justification for why LESs work, which is important because the entire benchmark methodology rests on this claim:

Format priors: Pre-training on vast amounts of API documentation, code, and tool-call logs gives LLMs strong priors for generating well-formatted, syntactically correct JSON tool responses. The LLM has seen millions of examples of function signatures, return values, error messages, and status codes, so it can generate tool responses that are structurally indistinguishable from real API responses.

Domain knowledge: LLMs encode operational logic for hundreds of professional domains through their pre-training. A sufficiently large LLM has been trained on hospital triage protocols, financial auditing procedures, network firewall rules, customs regulations, manufacturing scheduling algorithms, and agricultural monitoring practices. This knowledge is latent in the model weights and can be activated by the system prompt and initial state. The paper's claim is not that the LLM has memorized specific databases or patient records, but rather that it understands the operational grammar of these domains—what information is needed, in what order, with what constraints, and what constitutes a valid state transition.

State maintenance: The combination of system prompt constraints (which define the simulation rules) and in-context state tracking (through the accumulated conversation history) enables coherent multi-turn simulation. Each new tool response is generated by conditioning on the full conversation history, which encodes all previous actions and observations. This is analogous to how a human operator consulting a log would determine the current state of a system, but the LLM does it in a single forward pass.

Edge case handling: LLMs handle unexpected inputs more gracefully than rule-based simulators, generating reasonable error responses for out-of-bounds parameters. A rule-based simulator for a hospital system might crash or return an unhelpful generic error when an agent requests a non-existent patient ID; an LLM can generate a realistic "patient not found" response with appropriate context. This is important for robustness evaluation: the LES needs to be able to generate both valid responses and realistic error responses without breaking.

Why this matters: the LES is not claiming to perfectly replicate a real production system. It is claiming to be a sufficiently realistic simulator that an agent's decision-making process (what it checks, in what order, how it handles errors, whether it verifies its work) can be evaluated meaningfully. The paper explicitly acknowledges (Section 8.1) that the LES models "domain logic rather than domain data"—it knows a drug interaction check should return contraindications, but the specific values are generated rather than retrieved from a real database. The evaluation therefore measures decision-making quality (process) rather than data precision (exact numerical accuracy).

The Multi-Agent Synthesis Pipeline

The synthesis pipeline is the mechanism that converts the 100 scenario designs into 382 evaluation instances, each guaranteed to be solvable, verifiable, discriminative, and diverse. Section 4 describes this pipeline in detail.

Scenario design and reference documents: For each of the 100 professional task scenarios, the authors design 16 non-overlapping sub-topics and construct a professional reference document for each. These documents cover domain terminology, workflows, state variables, edge cases, and constraints. They ground all subsequent generation, ensuring that instances differ structurally rather than superficially—two instances from the same scenario might involve different patients with different conditions, different inventory configurations, or different time constraints, but they share the same underlying domain logic encoded in the reference document.

Generation agents: The pipeline is powered by Gemini-3-Flash-Preview as the LES and synthesis engine. Multiple specialized generation agents (the paper uses the term "multi-agent synthesis pipeline") produce:

Environment configurations: system prompt, tool schema, initial state, and state description for each instance.
Task instructions: the natural language instruction given to the agent under test (e.g., "Discharge P-110 from Exam Room 2, transfer P-552 into the vacated room, execute Phase 1 Data Acquisition then Sepsis Clinical Bundle").
Tool definitions: the specific function signatures with typed parameters that the agent will have access to (drawn from the tool schema, instantiated for the specific instance).
Solution plans: step-by-step reference solutions that can solve the task, used for solvability verification and difficulty calibration (testing whether an LLM can execute the plan autonomously).
Verification rubrics: automated success criteria, typically checking that specific tool calls were made in specific orders with valid arguments, that state constraints were maintained, and that the final state matches expectations.

Solvability verification and difficulty calibration: Each generated task is executed multiple times—both with and without a reference plan—to verify solvability and calibrate difficulty. The reference plan execution confirms that a solution exists (the plan can be followed to completion). The autonomous execution (without the reference plan) measures the difficulty: if the task is solved 100% of the time autonomously, it is trivially easy and filtered out; if it is solved 0% of the time, it is unsolvable (even with the reference plan, an error occurred) and filtered out.

Majority-vote verification: A majority-vote verifier assesses trajectories against rubrics. Because the LES can generate slightly different observations on different runs even for the same action (due to the stochastic nature of LLM generation), the verifier runs multiple times and takes the majority decision to reduce noise in solvability judgments.

Repair module: When a generated task fails the solvability check (the reference plan cannot be executed to completion), a repair module diagnoses the failure (e.g., tool schema mismatch, impossible state constraint, ambiguous instruction) and modifies the configuration before re-execution. Tasks that cannot be repaired are discarded.

Quality filtering criteria: The final filters remove:

Instances where all difficulty levels are trivially solved (100% autonomous success rate)
Instances that are unsolvable even with the reference plan (0% success rate)
Instances with invalid tool schemas (e.g., tools that cannot be called due to parameter mismatches)

Final instance selection: For each of the 100 scenarios, the pipeline selects the difficulty level with the lowest autonomous success rate (excluding 0% and 100%) to maximize discriminative power. This means the 382 evaluation instances are deliberately the hardest solvable instances from each scenario, designed to distinguish between capable and incapable agents.

Dataset statistics: After synthesis and filtering, the evaluation set contains 382 solvable task instances spanning all 100 scenarios. Tasks average 5.5 tools (median 5, range 2–10) and 16.2 tool calls per task, reflecting realistic multi-step professional workflows.

Why this pipeline structure: a naive approach—asking an LLM to generate a task description and verifying it manually—would not guarantee solvability, would not calibrate difficulty, and would not filter out degenerate instances. The multi-step pipeline with execution-based verification and repair ensures that every instance in the benchmark is known to be solvable (a reference plan exists and has been successfully executed in the LES), is neither trivially easy nor impossible, and is validated against a rubric. This is essential for a benchmark that claims to evaluate agent capability: if some instances were accidentally unsolvable, models would be unfairly penalized; if some were trivially easy, they would provide no discriminative signal.

The Evaluation Loop

At evaluation time, the interaction between the agent under test and the LES follows a structured loop, illustrated in Figure 1 of the paper (Section 3.3):

Step-by-step interaction:

Initialization: The agent receives the task instruction (natural language describing the professional job to perform) and the tool schema (the set of callable functions with their parameter types). The LES is initialized with the environment configuration $c$ and an empty conversation history.
Agent action: The agent decides which tool to call next, with what arguments, and issues a structured tool call $a_t$ . This decision is based on the task instruction, the tool schema, and the accumulated conversation history $H_{t-1} = \{a_1, o_1, ..., a_{t-1}, o_{t-1}\}$ .
LES response: The LES processes the tool call conditioned on its configuration $c$ and the full conversation history $H_{t-1}$ , and generates an observation $o_t$ as a structured JSON response. The LES's internal state $s_t$ is maintained implicitly through the conversation history.
History update: The new action-observation pair $(a_t, o_t)$ is appended to the conversation history.
Repeat: Steps 2–4 continue until the agent signals task completion (either through a specific finalization tool call or by reaching a maximum step limit).

This is the standard agent-environment loop from reinforcement learning, adapted to language-based interaction. The key difference from real environments is that the observation $o_t$ is generated by an LLM rather than returned by a real system.

Automated verification: After the agent completes its execution, the complete trajectory is scored by a rubric-based verifier against pre-defined success criteria. The verification is not based on a single final answer but on the entire trajectory—checking that specific tool calls were made in valid orders, that constraints were maintained (e.g., battery above 15%, permits active before work began, chemical verification after mixing), and that the final state satisfies the task requirements. This trajectory-aware grading is important for professional tasks where the correctness of the result depends on the correctness of the process (a patient might be discharged, but if the agent skipped a safety check, the discharge is invalid regardless of the outcome).

Why this evaluation structure: it mirrors the real-world deployment scenario for AI agents in professional contexts—an agent is given a task description, a set of available tools, and must iteratively interact with the environment to complete the task. The evaluation measures both task completion (did the agent achieve the goal?) and process correctness (did the agent follow required procedures, check constraints, verify results?). The rubric-based verification provides automated, reproducible scoring that does not require human judgment for each trajectory.

Environmental Fault Injection Framework

OCCUBENCH evaluates agent robustness through controlled fault injection at evaluation time (Section 5.2). This is a distinct module that modifies the LES's behavior without changing the task instances.

Fault injection mechanism: All task instances are synthesized in clean environments (E0) with no faults. At evaluation time, faults are injected by appending fault rules to the LES's system prompt. This means the same task instance can be evaluated under multiple fault conditions without re-synthesizing the task, and the fault rules are declarative (natural language instructions to the LES) rather than programmatic.

Four fault conditions:

E0 (Clean): No faults injected. This is the baseline, measuring agent performance under ideal conditions. The LES follows its standard configuration exactly.
E1 (Explicit Faults): The LES randomly injects clearly visible error responses: HTTP 500 Internal Server Error, TimeoutError, ConnectionRefused, ServiceUnavailable. These faults have clear error signals—the agent receives an error message and knows the call failed. The correct behavior is to retry the failed call. This measures the agent's ability to recognize and recover from obvious failures.
E2 (Implicit Faults): The LES returns degraded responses with no error signal: truncated data (only the first 1–2 items when more should be returned), incomplete lists, empty/null fields, or stale cached values. The response appears superficially correct—valid JSON, appropriate structure, no error message—but the data is incomplete or degraded. The correct behavior is to detect the quality issue (e.g., by noticing that only 2 of 15 expected items were returned) and re-query. This measures the agent's ability to independently assess response quality, which is a fundamentally different capability from recognizing explicit errors.
E3 (Mixed): Approximately half explicit faults and half implicit faults, randomly interleaved. This represents the most realistic scenario where both types of environmental degradation occur simultaneously.

Fault parameterization: All faults are:

Transient: retrying the same tool call after the fault event expires recovers normal results. This ensures faults are recoverable in principle—the question is whether the agent's behavior enables recovery.
Spaced across the interaction: faults are not concentrated at the start or end of the interaction, but distributed across the task execution.
Controlled by two independent parameters:
- Fault count (fc): the number of distinct fault events injected during the task. Default is 2.
- Fault duration (fd): the number of consecutive tool calls affected per fault event. Default is 2.

This parameterization allows systematic ablation: varying fault count while holding duration fixed (and vice versa) reveals how agents degrade as faults become more frequent or more persistent (Section 6.2, Figure 4).

Why this fault taxonomy: the distinction between explicit (E1) and implicit (E2) faults captures a fundamental asymmetry in real-world robustness. Explicit failures are annoying but manageable—the agent knows something went wrong and can retry. Implicit failures are dangerous—the agent may proceed with degraded data, making decisions based on incomplete or stale information without realizing it. The paper's central finding that E2 is harder than E1 (average 53.4% vs. 62.6%, a 9.2-point gap) validates this distinction and reveals that most current agents lack the capability to detect data degradation. The mixed condition (E3) tests whether agents can handle both types simultaneously, which is the norm in real environments.

Robustness score: The robustness metric is defined as:

$R = \min(\text{CR}_{E1}, \text{CR}_{E2}, \text{CR}_{E3}) / \text{CR}_{E0}$

where $\text{CR}_{E_i}$ is the completion rate under fault condition $E_i$ across all 382 tasks.

What it computes: the worst-case degradation under any single fault type, normalized by clean performance. Taking the minimum across the three fault conditions (rather than the average) means a model is penalized for its weakest fault type—if it handles explicit faults perfectly but collapses under implicit faults, its robustness score is low.

Why this form: using the minimum captures the idea that a robust agent must be resilient to all types of environmental degradation, not just some. A high average robustness across fault types would be misleading if one type causes catastrophic failure. The normalization by $\text{CR}_{E0}$ makes robustness comparable across models with different baseline capabilities—a model with 50% clean performance that degrades to 45% under faults (R = 0.9) is more robust than a model with 80% clean performance that degrades to 60% under faults (R = 0.75), even though the latter still achieves higher absolute performance under faults.

Scenario Design Principles and Dataset Composition

The 100 scenarios in OCCUBENCH are not randomly generated—they follow explicit design principles to ensure coverage, realism, and practical relevance (Section 5.1):

Real job mapping: Each scenario corresponds to an actual professional role (emergency triage nurse, customs officer, production scheduler, network security analyst), not an abstract task. This ensures that evaluation results have direct practical interpretation—a model's score on a scenario reflects its fitness for that specific occupational role, not just its general reasoning ability.

Domain balance: No single domain contributes more than 3 scenarios. This prevents any one professional area from dominating the aggregate score and ensures that cross-industry capability profiles are based on meaningful per-industry sample sizes. The 65 domains are distributed across 10 industry categories (Table 1), with Business & Enterprise having the most scenarios (19) and Science & Research having the fewest (4).

Irreplaceability: The paper explicitly states that the majority of scenarios (nuclear safety monitoring, drug interaction screening, emergency evacuation coordination, customs processing) are untestable by any existing benchmark. This is not a convenience—it is the core motivation. The scenarios were selected precisely because they represent the "untestable majority" of professional work.

Multi-step interaction: All scenarios require multi-turn state transitions, not single-step function calls. This distinguishes OCCUBENCH from simple function-calling benchmarks: an agent must track state across multiple interactions, make decisions based on accumulated information, and execute procedures that unfold over time. The average of 16.2 tool calls per task confirms this multi-step nature.

Industry categories and coverage (from Table 1):

Business & Enterprise: 19 scenarios (resume screening, expense auditing, AML review)
Technology & IT: 16 scenarios (Linux ops, CI/CD recovery, intrusion response)
Industrial & Engineering: 12 scenarios (production scheduling, mine ventilation)
Transportation & Logistics: 11 scenarios (last-mile delivery, train dispatch)
Commerce & Consumer: 9 scenarios (dynamic pricing, hotel revenue management)
Education & Culture: 8 scenarios (adaptive curriculum, fact-checking)
Healthcare & Life Sciences: 7 scenarios (emergency triage, drug interaction screening)
Public Service & Governance: 7 scenarios (permit processing, wildfire evacuation)
Agriculture & Environment: 7 scenarios (irrigation control, crop disease diagnosis)
Science & Research: 4 scenarios (telescope scheduling, excavation planning)

Why this design: the combination of real job mapping, domain balance, irreplaceability, and multi-step interaction ensures that OCCUBENCH evaluates what it claims to evaluate—professional AI agent capability—rather than a proxy like web navigation or code generation skill. The distribution across industries is deliberately broad to enable the paper's central finding that no model dominates all industries, which would be invisible in a benchmark covering only 1–2 domains.

4. Key Insights and Innovations

Innovation 1: Language Environment Simulation Reframes Environment Construction as a Configuration Problem

The dominant assumption across all prior agent benchmarking has been that evaluating an agent requires access to the real environment—deployed web applications (WebArena), configured virtual machines (OSWorld), real GitHub repositories (SWE-bench), or live APIs (TAU-bench). This assumption creates an inescapable tradeoff: you can evaluate agents in the narrow set of domains with public environments, or you can attempt the massive engineering effort of building environments for new domains, but you cannot have both breadth and realism. Every existing benchmark implicitly accepts this tradeoff, which is why the field's collective coverage spans only a handful of domains despite the existence of hundreds of professional occupations where agent deployment is economically valuable.

OCCUBENCH's core intellectual move is to reject this tradeoff entirely by redefining what counts as an "environment." Rather than treating an environment as a software system that must be built, deployed, and maintained, the paper treats it as a behavioral specification that can be described declaratively and executed by an LLM. The Language Environment Simulator formalization is not an engineering trick—it is a fundamental reframing of the evaluation problem from an infrastructure challenge to a specification challenge.

What makes this more than a practical convenience is the argument in Section 3.3 that LLMs are already environment simulators by virtue of their pre-training. The model has been trained on the operational grammar of professional domains—hospital triage protocols, customs regulations, network management procedures—and the configuration $c$ merely constrains this latent knowledge into a specific, stateful simulation. The innovation is not that LLMs can generate plausible text (this has been known since GPT-2), but that the combination of pre-trained domain knowledge, in-context state maintenance, and declarative constraints is sufficient to produce evaluable interactive environments—not training data, not planning rollouts, but the actual environment against which agent capability is measured.

This reframing has a concrete consequence that distinguishes it from prior work on LLMs as world models. Gu et al. (2024) showed LLMs can serve as world models for planning web agent actions; WebWorld (Xiao et al., 2026) trained simulators for agent training; ViMo (Luo et al., 2025) built visual simulators for GUI agents. All of these use simulation as a means to an end—better training data, better planning, better policy learning. OCCUBENCH uses simulation as the end itself: the simulator is the evaluation apparatus. This is a distinct role with distinct requirements. A training simulator can be imperfect because the agent learns to compensate for simulation artifacts during training; an evaluation simulator must be reliable because any error in the simulator becomes an error in the measurement of the agent. The paper's cross-simulator analysis (Section 6.6, Table 4) directly acknowledges and investigates this requirement, finding that simulator quality critically affects evaluation reliability—a negative result that prior world-model work, focused on training, never needed to confront.

The significance of this innovation extends beyond OCCUBENCH itself. The paper demonstrates a methodology—define the behavioral contract, let the LLM execute it, verify solvability through execution, calibrate difficulty, filter degenerate instances—that can be applied to any domain an LLM understands. This means the "untestable majority" of professional work becomes testable not through the accumulation of engineering effort but through the accumulation of domain specifications. The scaling is qualitatively different: adding a new domain to WebArena requires deploying and configuring web applications (engineering effort per domain); adding a new domain to OCCUBENCH requires writing a configuration (specification effort per domain). The paper does not claim this is costless—writing good configurations requires domain expertise and careful prompt engineering—but it argues that the cost structure is fundamentally different and far more scalable.

Evidence: the benchmark covers 100 scenarios across 65 domains and 10 industry categories (Table 1), a breadth that would be practically impossible under the traditional environment-construction paradigm. The cross-simulator analysis (Table 4, Figure 8) provides the caveat that simulator capability matters—GPT-5.2 as a simulator produces degraded evaluations—while confirming that with a capable simulator, agent rankings achieve 85.7% pairwise agreement.

Innovation 2: Cross-Industry Capability Profiling Reveals That Aggregate Benchmarks Obscure Occupational Specialization

Prior agent benchmarks, by construction, produce a single aggregate ranking. WebArena tells you which model is best at web browsing tasks across its deployed websites; SWE-bench tells you which model is best at resolving GitHub issues; TAU-bench tells you which model is best at customer service in retail and airline domains. The implicit assumption in how these rankings are used is that model capability is approximately domain-invariant—a model that ranks first on web tasks will also rank first on desktop tasks, on code tasks, on tool-use tasks. The research community and industry routinely use aggregate leaderboard positions to make claims about which models are "best" as agents.

OCCUBENCH demonstrates that this assumption is empirically false across professional domains, with consequences that are both scientific and practical. The radar chart in Figure 2 and the full industry breakdown in Table 2 show that every model has a distinct occupational capability profile—a unique pattern of strengths and weaknesses across industries that is invisible to any single-domain aggregate score. GPT-5.2 achieves the highest overall completion rate (79.6%) but ranks 5th in Commerce (67%), where Qwen 3.5 Plus dominates (81%). Gemini 3.1 Pro leads in Education (84%) and Science (81%) but drops to 62% in Healthcare, where Qwen 3.5 Plus again leads (81%). Claude Opus 4.6 leads in Transportation (77%) but falls to 53% in Commerce. These are not small variations around a stable mean; they are qualitative reversals of capability ordering between domains.

The intellectual contribution here is not the observation that models differ across tasks—this is trivially true. It is the systematic characterization of occupational specialization as a first-class property of AI agents, analogous to how human professionals specialize. A human emergency physician and a human customs officer have overlapping but distinct cognitive skill profiles; OCCUBENCH shows that AI agents exhibit the same phenomenon, and that a benchmark covering only emergency medicine would rank models entirely differently from one covering only customs processing. The default assumption of domain-invariant agent capability that underlies single-domain benchmarking is not just incomplete—it is actively misleading for deployment decisions.

This finding has a direct practical implication that the paper does not state explicitly but is implied by the data: model selection for professional deployment should be industry-specific, not benchmark-leaderboard-driven. An organization deploying an agent for healthcare triage should look at a model's Healthcare score specifically, not its aggregate OCCUBENCH score or its SWE-bench ranking. The cross-industry profiling capability that OCCUBENCH enables is not a nice-to-have feature; it is the minimal information required to make informed deployment decisions.

Evidence: Table 2 provides the per-industry breakdown for all 15 models across 10 categories. Figure 2 visualizes the capability profiles as a radar chart showing qualitatively different shapes for each model. The text in Section 7.2 interprets these profiles, identifying that Gemini 3.1 Pro excels in knowledge-intensive domains (Education, Science, Technology), Claude Opus 4.6 excels in operational domains (Transportation, Business, Industrial), and Qwen 3.5 Plus excels in consumer-facing domains (Commerce, Healthcare, Agriculture)—patterns that would be invisible in a single aggregate ranking.

Innovation 3: Implicit Fault Vulnerability as a New Diagnostic Dimension for Agent Robustness

The paper's environmental robustness framework (E0–E3) introduces a distinction that has no precedent in agent benchmarking: the difference between explicit failures (the agent knows something went wrong) and implicit degradation (the agent must independently detect that the data is incomplete, truncated, or stale). This distinction is not just a taxonomy—it is a diagnostic instrument that reveals a specific, previously unmeasured capability gap in current agents.

Prior benchmarks evaluate agents exclusively on clean environments. A handful of works in the broader ML robustness literature evaluate models under adversarial perturbations or distribution shift, but no agent benchmark has systematically injected environmental faults and measured recovery behavior. The absence of robustness evaluation is not an oversight—it follows directly from the environment-construction paradigm. Real environments (websites, VMs, APIs) are difficult enough to build and maintain in their clean form; building separate "faulty" versions of each environment for robustness testing has been practically impossible. OCCUBENCH's LES architecture makes fault injection trivial (append fault rules to the system prompt), enabling a systematic evaluation that was previously infeasible.

The central empirical finding—that implicit faults (E2, average 53.4%) are harder than explicit faults (E1, average 62.6%), a 9.2-point gap across all models (Table 3)—is counterintuitive enough to be genuinely informative. The intuition might be that explicit errors are harder because they disrupt execution; an agent must recognize the error, retry, and potentially replan. Implicit faults, by contrast, produce valid-looking responses that the agent can proceed with. But the data shows the opposite: the absence of an error signal is more damaging than the presence of one, because most agents lack the metacognitive capability to independently assess response quality. They trust the environment by default, and when the environment silently degrades, they proceed with bad data.

This finding is not just a benchmark result; it is a diagnosis of a specific architectural weakness in current agent designs. The case studies in Figures 16 and 17 make the mechanism concrete. Under E1, Kimi K2.5 encounters a single HTTP 500 error and abandons the entire task after 2 tool calls, while Claude Opus 4.6 encounters 4 errors across 12 calls and retries each one, completing all required actions—a difference in error recovery policy, not in underlying reasoning capability. Under E2, Kimi K2.5 receives truncated data (2 units instead of 15) with no error signal, retries, gets truncated data again (fault duration = 2), and then assumes the truncated data is complete, producing a financial assessment (DSCR 1.72x, pass) that is catastrophically wrong (actual DSCR 1.19x, fail). Claude Opus 4.6 notices the discrepancy ("only 2 of 15 units") and re-fetches, obtaining complete data after the fault expires—a difference in data quality monitoring, not in calculation skill.

The robustness score metric $R = \min(\text{CR}_{E1}, \text{CR}_{E2}, \text{CR}_{E3}) / \text{CR}_{E0}$ operationalizes this diagnostic. By taking the minimum across fault types rather than the average, it penalizes models for their weakest fault type—a model that is perfectly resilient to explicit errors but fragile to implicit degradation gets a low robustness score, accurately reflecting its deployment risk. This is a more honest metric than an average would be, because a real deployment encounters all fault types.

The practical significance is that robustness evaluation should be a standard part of agent benchmarking, not an optional add-on. The paper shows that models with similar clean-environment performance can have dramatically different fault resilience (Gemini 3.1 Pro and MiniMax M2.7 both have robustness scores of 0.87 despite a 18.4-point gap in E0 performance), and models with high clean performance can have low robustness (Kimi K2.5: 64.1% E0, robustness 0.63). For production deployment decisions, robustness under faults is arguably more informative than clean performance, since real environments are never clean.

Evidence: Table 3 provides the full E0–E3 breakdown for 9 flagship models. Figure 3 visualizes the E0–E3 degradation. Figure 4 ablates fault count and duration, showing continued degradation beyond the default parameters. The case studies in Figures 16 and 17 illustrate the behavioral mechanisms behind the aggregate statistics.

Innovation 4: The Simulator-Agent Capability Gap as a Fundamental Constraint on Simulation-Based Evaluation

The cross-simulator experiment in Section 6.6 produces what might initially appear to be a negative or cautionary result: using GPT-5.2 as the LES causes all agents' scores to collapse (average 29.3% vs. 67.9% under Gemini Flash), including GPT-5.2's own score dropping from 79.6% to 42.4%. This is not framed as a limitation of OCCUBENCH but as a fundamental finding about the nature of LES-based evaluation: the quality of the simulator is a first-order determinant of evaluation validity, and strong agents are not necessarily strong simulators.

The reason this constitutes an innovation rather than just a limitation is that it identifies a previously invisible constraint on the entire paradigm of using LLMs as evaluators. The field has explored LLMs as judges (scoring outputs), LLMs as teachers (generating training data), and LLMs as world models (simulating environments for planning). In each case, the implicit assumption is that a sufficiently capable LLM is adequate for the role. OCCUBENCH's cross-simulator analysis demonstrates that this assumption fails in a specific, diagnosable way for environment simulation. GPT-5.2, the strongest agent model tested, fabricates environment state (inventing empty rooms, Figure 9), omits critical entities (dropping agents from rosters, Figure 10), and invents business rules (enforcing return-window constraints not in the specification, Figure 11). These are not random errors or noise—they are systematic violations of the environment contract that make the task unsolvable regardless of the agent's capability.

The three failure modes identified in the case studies are particularly revealing because they correspond to distinct ways an LLM's generative capabilities can interfere with its simulation role:

State fabrication (Figure 9): The simulator, perhaps drawing on generic knowledge that hospital rooms usually number more than two, invents additional rooms not in the initial state. The agent uses an invented room, which satisfies the simulator's own fabrication but violates the rubric's expectation. This is a case where the simulator's domain knowledge (hospitals have many rooms) overrides the configuration's explicit constraints (there are exactly two rooms).
Entity omission (Figure 10): The simulator drops a specifically named and configured agent ("Raj NYC," Tier 2 Database specialist) from a roster query result, returning only a Tier 3 manager. The agent assigns the only available candidate, which is the wrong specialization. This is a case where the simulator's generation process loses information that is present in the configuration but not sufficiently salient to survive the generation.
Rule invention (Figure 11): The simulator independently computes a date difference and rejects a return as outside the 30-day window, even though the task specification does not include this constraint. This is a case where the simulator's general reasoning capability (calculating elapsed time and applying common business rules) generates a constraint that the environment contract does not impose.

These failure modes are not unique to GPT-5.2—they are intrinsic risks of using any generative model as an environment simulator. The paper's contribution is to characterize these risks systematically and to establish that simulator quality must be verified before drawing conclusions from LES-based evaluations. The pairwise ranking agreement matrix (Figure 8) shows that when both simulators are capable (Gemini Flash vs. Qwen 3.5 Plus), agreement is 85.7%, with the top-3 agents matching exactly. When one simulator is degraded (GPT-5.2), agreement drops to 75.0–82.1%. This quantifies the reliability boundary: LES-based evaluation produces trustworthy rankings when the simulator is sufficiently capable, but a weak simulator invalidates the evaluation.

This finding has direct methodological implications for future work using LES-based evaluation. The paper's recommended practices—verify simulator quality before drawing conclusions, re-verify task solvability when switching simulators—are not boilerplate limitations but necessary validity checks derived from empirical evidence. Any future benchmark that uses LLMs as environment simulators must contend with the simulator-agent capability gap that OCCUBENCH identifies and quantifies.

Evidence: Table 4 shows the cross-simulator completion rates and rankings for 8 agents under 3 simulators. Figures 9–11 provide detailed case studies of the three failure modes. Figure 8 provides the pairwise ranking agreement matrix.

5. Experimental Analysis

Evaluation Methodology

Dataset. OCCUBENCH consists of 100 professional task scenarios across 10 industry categories and 65 specialized domains, yielding 382 solvable evaluation instances after synthesis and quality filtering. The scenarios are fully synthetic—generated by a multi-agent synthesis pipeline powered by Gemini-3-Flash-Preview—with no pre-existing dataset split. Each task maps to a real human job role, and the final evaluation set contains tasks selected at the difficulty level with the lowest autonomous (non-trivial) success rate to maximize discriminative power. Tasks average 5.5 tools and 16.2 tool calls.
Base model(s). The paper evaluates 15 frontier models spanning 8 model families: OpenAI (GPT-5.2), Anthropic (Claude Opus 4, 4.5, 4.6; Claude Sonnet 4, 4.5, 4.6), Google (Gemini 3.1 Pro, Flash-Lite), DeepSeek (V3.2), Moonshot (Kimi K2.5), MiniMax (M2.7), Zhipu (GLM-5), and Alibaba (Qwen 3.5 Plus, Flash). All models are evaluated using thinking/reasoning mode where available, with the reasoning effort set to "high" for models with configurable thinking depth (unless otherwise ablated). The diversity of model families, sizes, and generations is deliberate: it enables within-family size scaling analysis, cross-generational comparison, and open-source vs. closed-source comparison—all of which the paper exploits in Sections 6.3–6.5.
Metrics. The primary metric is Completion Rate (CR): the fraction of the 382 evaluation tasks where the agent's trajectory passes automated rubric-based verification against pre-defined success criteria. Verification is trajectory-aware—it checks that specific tool calls were made in valid orders, that constraints were maintained throughout execution, and that the final state satisfies task requirements—not just that a single final answer matches. The secondary metric is Robustness Score (R), defined as R = min(CR_E1, CR_E2, CR_E3) / CR_E0, measuring worst-case resilience across all fault types normalized by clean performance. All rates are reported over the full 382-task denominator. No confidence intervals or statistical significance tests are reported, which is a notable omission given the relatively small number of instances per industry category (ranging from 4 scenarios for Science to 19 for Business).
Baselines. The paper does not compare against prior benchmarks or existing evaluation methods—it is constructing a new evaluation paradigm, so there is no established baseline to compare against. Within the benchmark, the primary baselines are: E0 (Clean) performance as the capability baseline, and the different model variants as de facto baselines against each other. For robustness, E0 serves as the clean baseline against which E1–E3 degradation is measured. For the reasoning effort ablation, "none" (or equivalent lowest setting) serves as the baseline. The cross-simulator analysis uses Gemini-3-Flash-Preview as the default simulator baseline against which Qwen 3.5 Plus and GPT-5.2 as simulators are compared.
Generation budget / compute accounting. The paper does not use a generation budget in the traditional sense—it does not compare methods at different sampling budgets or report FLOPs. Instead, the "budget" is the completion of the task instance: each model gets one attempt per task (a single trajectory), and success or failure is determined by the rubric-based verifier. The reasoning effort ablation in Section 6.5 varies the inference-time compute allocated to reasoning (none, low, medium, high, max/xhigh) for Claude Opus 4.6 and GPT-5.2, but this is a model-level configuration, not a benchmark-level budget sweep. The fault injection framework parameterizes fault count (default 2) and fault duration (default 2) as independent controls on environmental degradation, which is a form of difficulty modulation, but again not a compute budget in the FLOPs or sample-count sense that prior work on test-time compute scaling uses.
Cross-validation / statistical protocol. No cross-validation, bootstrapping, or statistical significance testing is reported. The 382 evaluation instances are a single fixed test set, and all results are point estimates. The paper does not describe any held-out validation set or any procedure for controlling overfitting to the specific instances. This is a limitation: with only 382 instances and 10 industry categories (some with as few as 4 scenarios, e.g., Science), per-industry completion rates have substantial sampling variance that is not quantified. The cross-simulator analysis (Section 6.6) provides a form of robustness check on rankings by comparing agent ordering across different LES models, but this is a methodological investigation rather than a statistical protocol applied to the primary results.

Main Quantitative Results

Cross-Industry Task Completion (E0)

Table 2 presents the completion rates across all 10 industry categories for all 15 models under E0 (clean environments, thinking mode, high reasoning effort for configurable models). The headline findings are:

No model dominates all industries. GPT-5.2 leads overall at 79.6%, with the highest scores in Agriculture (84%), Business (86%), Industrial (85%), and Science (94%). However, its Commerce score (67%) is substantially below Qwen 3.5 Plus (81%), and its Transportation score (72%) trails Claude Opus 4.6 (77%). Gemini 3.1 Pro ranks second at 72.3%, leading in Education (84%) and Science (81%) but falling to 62% in Healthcare—a 22-point drop from its best to worst industry. Claude Opus 4.6 ranks third at 71.5%, showing the opposite pattern: strongest in Transportation (77%) and Business (78%) but weakest in Commerce (53%) and Science (62%). These are not minor variations; they are qualitative reversals of model ordering depending on which industry is measured.

Open-source models are competitive. Qwen 3.5 Plus (69.9%, rank 4) and DeepSeek V3.2 (69.6%, rank 5) outperform all Claude Sonnet variants and Claude Opus 4 and 4.5. Qwen 3.5 Plus leads Healthcare (81%) and Commerce (81%) outright, beating GPT-5.2 by 5 points in Healthcare and 14 points in Commerce, while DeepSeek V3.2 achieves the second-highest Business score (78%, tied with Claude Opus 4.6). This challenges the assumption that closed-source models uniformly outperform open-source alternatives on professional tasks—the open-source models have different strengths, not uniformly weaker ones.

Substantial variation across models in aggregate. The spread from best (GPT-5.2, 79.6%) to worst (Claude Sonnet 4, 53.4%) is 26.2 points, demonstrating that OCCUBENCH produces meaningful differentiation rather than ceiling effects. Even among top models, the gap from 1st to 5th is 10 points—substantial enough to matter for deployment decisions.

Industry difficulty varies systematically (Figure 12, Section 7.1). Aggregating across all 15 models, the easiest industries are Business & Enterprise (avg 70.1%) and Public Service & Governance (avg 69.4%), while the hardest are Transportation & Logistics (avg 56.2%) and Education & Culture (avg 57.6%). The paper interprets this as reflecting domain complexity: business and public service tasks tend to follow well-documented procedures with clear decision paths, while transportation involves complex multi-constraint optimization (routing, scheduling, load balancing) and education requires nuanced pedagogical judgment and multi-step curriculum reasoning.

Distinct occupational capability profiles (Figure 2, Section 7.2). The radar chart visualization of the top 6 models shows qualitatively different shapes—each model has a unique fingerprint of strengths across industries. Gemini 3.1 Pro excels in knowledge-intensive domains (Education 84%, Science 81%, Technology 78%); Claude Opus 4.6 excels in operational domains (Transportation 77%, Business 78%, Industrial 73%); Qwen 3.5 Plus excels in consumer-facing domains (Commerce 81%, Healthcare 81%, Agriculture 78%). Kimi K2.5 shows balanced but moderate performance, with notable weaknesses in Commerce (56%) and Transportation (57%). These profiles are invisible to single-domain benchmarks and are the paper's central empirical contribution.

Environmental Robustness

Table 3 presents completion rates under fault injection for 9 flagship models (one per family), and Figure 3 visualizes the E0–E3 degradation. The headline findings:

Substantial degradation under faults across all models. Average completion rate falls from 67.5% (E0) to 62.6% (E1) to 53.4% (E2) to 54.4% (E3)—a 14.1-point drop from clean to the worst fault condition. Every model degrades: GPT-5.2 drops from 79.6% to 67.0% under E3 (12.6 points); Claude Opus 4.6 drops from 71.5% to 53.9% under E2 (17.6 points); Qwen 3.5 Plus drops from 69.9% to 51.6% under E2 (18.3 points). This demonstrates that clean-environment performance substantially overestimates real-world deployment readiness.

Implicit faults (E2) are harder than explicit (E1) and mixed (E3) faults—for most models. The average E2 score (53.4%) is 9.2 points below E1 (62.6%) and 1.0 points below E3 (54.4%). Counterintuitively, 4 out of 9 models perform worse under E2 (pure implicit) than E3 (mixed explicit + implicit): Claude Opus 4.6 (53.9% vs. 63.9%), DeepSeek V3.2 (56.0% vs. 51.6%—an exception where E3 is worse), Qwen 3.5 Plus (51.6% vs. 54.2%), and Kimi K2.5 (40.6% vs. 40.1%). The paper's explanation is that explicit errors provide unambiguous failure signals that prompt retry behavior, while implicit faults require the agent to independently assess response quality—a capability most models lack. E3's explicit error component partially compensates for its implicit component by triggering retry behavior that also catches the implicit degradation.

Model ranking shifts under robustness evaluation. Gemini 3.1 Pro, which ranks second in E0 (72.3%), ties for the highest robustness score (0.87) alongside MiniMax M2.7—the lowest-performing model in E0 (53.9%). This means Gemini 3.1 Pro is exceptionally resilient to environmental noise, while MiniMax M2.7 is exceptionally consistent (it degrades proportionally very little because it starts from a lower baseline). GPT-5.2, the E0 leader (79.6%), has a robustness score of 0.84 (rank 3), meaning its absolute performance under faults remains high even though it degrades more than the most robust models on a proportional basis. Kimi K2.5 has the lowest robustness (0.63), dropping from 64.1% to as low as 40.1%, indicating high sensitivity to environmental noise despite decent clean performance.

Increasing fault severity deepens the challenge (Figure 4). Ablating fault count (with duration fixed at 2) and fault duration (with count fixed at 2) on Claude Opus 4.6 and Qwen 3.5 Plus under E3 mixed faults shows continued degradation beyond the default parameters. Claude Opus 4.6 drops from 71.5% at fc=1 to 60.2% at fc=4 (count ablation) and from 67.8% at fd=1 to 57.9% at fd=4 (duration ablation). Qwen 3.5 Plus degrades from 61.3% to 49.7% (count) and 59.7% to 49.2% (duration). The paper highlights that these results reveal "an increasingly severe challenge for deploying agents in real-world environments, where faults are not only inevitable but may be frequent and persistent" (Section 6.2).

Model Scaling Analysis

Figure 5 compares large vs. small model variants within each family under E0. The paper evaluates five within-family pairs: Gemini 3.1 Pro vs. Flash-Lite, Qwen 3.5 Plus vs. Flash, Claude Opus vs. Sonnet (4.6 generation), Claude Opus vs. Sonnet (4.5 generation), and Claude Opus vs. Sonnet (4 generation). The headline findings:

Larger models consistently outperform smaller counterparts, with gaps of 7–11 points. Gemini Pro (72.3%) outperforms Flash-Lite (61.3%) by 11.0 points; Qwen Plus (69.9%) outperforms Flash (59.7%) by 10.2 points; Claude Opus 4.6 (71.5%) outperforms Sonnet 4.6 (64.4%) by 7.1 points; Claude Opus 4 (61.3%) outperforms Sonnet 4 (53.4%) by 7.9 points. The notable exception is the Claude 4.5 generation, where Opus (65.2%) and Sonnet (64.9%) perform nearly identically (gap of 0.3 points). The paper suggests this may reflect that "the 4.5 generation's architectural improvements benefited both model sizes equally" (Section 6.3), though this is speculation—the cause could also be ceiling effects, task-specific saturation, or statistical noise given the relatively small gap.

Scaling benefits are model-family-dependent. The gap size varies from 0.3 to 11.0 points, suggesting that some model families derive more benefit from increased scale on professional tasks than others. However, with only one or two within-family comparisons per architecture, it is impossible to distinguish whether this variation reflects genuine architectural differences in scaling efficiency or simply differences in how the small and large variants were trained (data, compute, optimization).

Generational Progress

Figure 6 tracks Claude's performance evolution across three generations (v4, v4.5, v4.6) for both Opus and Sonnet tiers. The headline findings:

Claude Opus shows consistent generational improvement: 61.3% → 65.2% → 71.5%, a +10.2% total gain over two generations. Each step is a meaningful increment (~4 points), suggesting that the architectural and training improvements between generations transfer to professional task execution.

Claude Sonnet shows a large jump from v4 to v4.5 (+11.5%, from 53.4% to 64.9%) but a slight regression from v4.5 to v4.6 (−0.5%, from 64.9% to 64.4%). The paper speculates this regression may "reflect a trade-off between reasoning depth and execution efficiency in the 4.6 adaptive thinking architecture" (Section 6.4). This is a plausible but unverified hypothesis—the 0.5-point difference could also be within the range of sampling variance given the 382-instance test set. Without confidence intervals, whether this is a genuine regression or noise cannot be determined.

The generational gap between Opus and Sonnet narrows then disappears. At v4, the gap was 7.9 points (61.3% vs. 53.4%). At v4.5, it narrowed to 0.3 points (65.2% vs. 64.9%). At v4.6, it widened again to 7.1 points (71.5% vs. 64.4%). This non-monotonic pattern is interesting: the v4.5 Sonnet gains were so large that it nearly caught up to v4.5 Opus, but v4.6 Opus pulled ahead again. This could reflect the specific improvements in each generation (v4.5 may have included architecture changes that disproportionately benefited smaller models; v4.6 may have included scale-dependent reasoning improvements that larger models exploit better), but without architectural details, these remain interpretations rather than explanations.

Reasoning Effort Ablation

Figure 7 evaluates the effect of reasoning effort (thinking depth) on Claude Opus 4.6 (five levels: none, low, medium, high, max) and GPT-5.2 (five levels: none, low, medium, high, xhigh). The headline findings:

Higher reasoning effort consistently improves performance, with a 27.5-point gap between extremes for GPT-5.2. GPT-5.2 improves monotonically: 54.7% (none) → 59.7% (low) → 66.8% (medium) → 76.2% (high) → 82.2% (xhigh). Each step provides a meaningful gain, with the largest jumps occurring between medium and high (+9.4 points) and between high and xhigh (+6.0 points). Claude Opus 4.6 shows a similar but more compressed trend: 70.2% (low) → 71.2% (medium) → 72.3% (high) → 73.8% (max), a total gain of 3.6 points. The "none" setting for Claude Opus 4.6 is not reported—the x-axis for Claude starts at "low."

GPT-5.2 benefits substantially more from reasoning effort scaling than Claude Opus 4.6. The 27.5-point gap for GPT-5.2 is nearly 8× larger than Claude's 3.6-point gap, despite GPT-5.2 starting from a lower floor (54.7% at none vs. 70.2% at low for Claude). This suggests that GPT-5.2's reasoning architecture derives disproportionate benefit from deeper thinking on professional tasks—or, alternatively, that its "none" setting is genuinely crippled for multi-step tool use while Claude's "low" setting retains substantial capability. The difference has practical implications: an organization deploying GPT-5.2 should invest heavily in inference-time reasoning compute, while an organization deploying Claude Opus 4.6 may see diminishing returns beyond "high" for professional tasks.

The monotonicity of the trend is notable. For both models, no intermediate setting underperforms a lower setting—higher reasoning effort is strictly better. This is important because it rules out the possibility that "overthinking" degrades performance on these tasks, a concern that has been raised in other contexts. The paper states: "These results suggest that allocating more compute to reasoning at inference time is a reliable strategy for improving agent performance on complex professional tasks" (Section 6.5). However, the paper only tests up to the maximum available setting; it does not test whether performance would plateau or degrade at even higher (hypothetical) effort levels.

Cross-Simulator Evaluation

Table 4 presents completion rates and rankings for 8 agents under three different LES models: the default Gemini-3-Flash-Preview, Qwen 3.5 Plus, and GPT-5.2. Figure 8 shows the pairwise ranking agreement matrix. The headline findings:

Using GPT-5.2 as the simulator catastrophically degrades evaluation quality. Under Gemini Flash, the 8 agents average 67.9% completion rate; under Qwen 3.5 Plus, they average 63.4%; under GPT-5.2, they average 29.3%. GPT-5.2's own score drops from 79.6% (as agent evaluated by Gemini Flash) to 42.4% (as agent evaluated by itself as simulator)—a 37.2-point collapse. Every agent's absolute score drops substantially under the GPT-5.2 simulator, but the relative ranking remains partially preserved: GPT-5.2 still ranks 1st (42.4%), Claude Opus 4.6 ranks 2nd (33.5%), DeepSeek V3.2 ranks 3rd (29.6%). This is consistent with the simulator degrading all agents proportionally, though not identically—some drop more than others (Gemini Pro drops from 2nd to tied 4th).

With a capable simulator (Qwen 3.5 Plus), agent rankings show high agreement with Gemini Flash. The pairwise ranking agreement is 85.7% (24 out of 28 model pairs preserve their relative ordering). The top-3 agents (GPT-5.2, Gemini Pro, Opus 4.6) match exactly, though Opus 4.6 and Qwen 3.5 Plus swap positions 3 and 4. The four disagreements all involve mid-ranked models with small performance gaps: Qwen 3.5 Plus (rank 4 under Gemini, rank 6 under Qwen), DeepSeek V3.2 (rank 5 vs. 4), GLM-5 (rank 7 vs. 5), and Kimi K2.5 (rank 6 vs. 8). GLM-5 shows the largest rank shift (from 7th to 5th), gaining 1.5 points (62.6% → 64.1%).

The GPT-5.2 simulator's failures are systematic and diagnosable, not random noise. The case studies in Figures 9, 10, and 11 identify three distinct failure modes: state fabrication (inventing empty rooms that don't exist in the environment specification, Figure 9), entity omission (dropping a specifically named agent from a roster query, Figure 10), and rule invention (independently computing a return-window expiry and rejecting a return, even though the task specification does not include this constraint, Figure 11). In all cases, the agent's strategy is correct—it makes the right decisions given the information it receives—but it fails because the simulator violates the environment contract. These failures demonstrate that GPT-5.2's general reasoning capability and domain knowledge, while beneficial for being an agent, actively interfere with its role as a simulator.

Simulator quality matters more for absolute scores than for relative rankings. While absolute scores collapse under GPT-5.2, the pairwise ranking agreement between Gemini Flash and GPT-5.2 is still 75.0% (21/28 pairs preserved), rising to 82.1% between Qwen 3.5 Plus and GPT-5.2. This suggests that LES-based evaluation is more reliable for comparing models than for estimating their absolute deployment capability, and that ranking-based conclusions (Model A > Model B) are more robust to simulator choice than threshold-based conclusions (Model A is deployment-ready because it exceeds X%).

GPT-5.2 as a simulator fails to distinguish between models effectively. The score range under GPT-5.2 is extremely compressed: from 23.0% (Kimi K2.5) to 42.4% (GPT-5.2), a spread of only 19.4 points, compared to 25.7 points under Gemini Flash (53.9% to 79.6%) and 21.9 points under Qwen 3.5 Plus (52.4% to 74.3%). This compression reduces the benchmark's discriminative power—small real differences between models become harder to detect.

Ablation Studies and Robustness Checks

Fault count ablation (Figure 4a): Varying fault count from 1 to 4 with fault duration fixed at 2 under E3 mixed faults, tested on Claude Opus 4.6 and Qwen 3.5 Plus. Claude Opus 4.6 degrades from 71.5% (fc=1) to 66.0% (fc=2) to 61.3% (fc=3) to 60.2% (fc=4)—a total drop of 11.3 points. Qwen 3.5 Plus degrades from 61.3% (fc=1) to 54.2% (fc=2) to 51.8% (fc=3) to 49.7% (fc=4)—a total drop of 11.6 points. The degradation is roughly linear for both models, with no evidence of a threshold beyond which performance collapses. This suggests that fault frequency imposes a cumulative cognitive load rather than triggering catastrophic failure at a specific frequency.

Fault duration ablation (Figure 4b): Varying fault duration from 1 to 4 with fault count fixed at 2 under E3 mixed faults. Claude Opus 4.6 degrades from 67.8% (fd=1) to 63.9% (fd=2) to 62.0% (fd=3) to 57.9% (fd=4)—a total drop of 9.9 points. Qwen 3.5 Plus degrades from 59.7% (fd=1) to 54.2% (fd=2) to 54.2% (fd=3) to 49.2% (fd=4)—a total drop of 10.5 points, though with a plateau between fd=2 and fd=3. The fact that duration increases beyond the default (fd=2) continue to degrade performance indicates that agents do not simply retry once and succeed—they may retry multiple times and fail repeatedly when the fault persists, consuming their available steps or abandoning the approach.

Reasoning effort monotonicity: The reasoning effort ablation (Figure 7) serves as an implicit robustness check on the benchmark's ability to capture compute-dependent improvements. The strict monotonicity for GPT-5.2 (54.7% → 59.7% → 66.8% → 76.2% → 82.2%) and near-monotonicity for Claude Opus 4.6 (70.2% → 71.2% → 72.3% → 73.8%) suggest that the benchmark does not saturate at the high end—there is still room for improvement even at the maximum reasoning effort levels—and that the difficulty calibration (selecting instances with the lowest autonomous success rate) successfully avoids ceiling effects for strong models.

Model family scaling consistency (Figure 5): The within-family comparisons serve as a validation that the benchmark captures model scaling effects. Larger models outperform smaller variants in 4 of 5 pairs, with gaps consistent with real capability differences rather than noise. The exception (Claude 4.5 Opus vs. Sonnet, 0.3 points) provides a useful counterpoint: when two models are genuinely close in capability, the benchmark does not artificially inflate the gap.

Cross-simulator ranking robustness (Figure 8, Table 4): This is the paper's most important robustness check, because it directly tests whether the core evaluation methodology (LES-based evaluation) produces results that are stable across simulator choices. The 85.7% pairwise agreement between Gemini Flash and Qwen 3.5 Plus, with exact matching of the top-3 agents, provides evidence that LES-based rankings are reliable when the simulator is sufficiently capable. The degraded agreement with GPT-5.2 (75.0%) and the collapsed absolute scores confirm that simulator quality is a genuine validity constraint, not a hypothetical concern.

Case study trajectory analysis (Figures 13–17): While not ablation studies in the traditional sense, the five detailed case studies provide qualitative validation that the benchmark's automated verification captures meaningful behavioral differences. Figure 13 (last-mile delivery) shows the verifier penalizing DeepSeek V3.2 for violating a battery constraint, even though it completed the delivery—a trajectory-aware judgment that a simple outcome-based grader would miss. Figure 14 (fish farm) shows Qwen 3.5 Plus failing verification because it skipped a post-mixing chemistry check, even though it correctly configured the equipment—the verifier catches the skipped verification step. Figure 15 (building inspection) shows DeepSeek V3.2 failing for procedural ordering errors (brazing before permit renewal, not restoring a gas valve), not for task non-completion. These cases confirm that the rubric-based verifier is making nuanced, trajectory-aware judgments, not just checking final answers.

Critical Assessment

The central claims of OCCUBENCH are: (1) it is the first benchmark systematically evaluating AI agents on real-world professional tasks across 100 scenarios, 10 industries, and 65 specialized domains, enabled by LESs; (2) no single model dominates all industries—each exhibits a distinct occupational capability profile invisible to single-domain benchmarks; (3) implicit environmental faults are harder than explicit and mixed faults; (4) larger models, newer generations, and higher reasoning effort consistently improve performance; and (5) strong agents are not necessarily strong simulators, but with a capable simulator, agent rankings are highly consistent (85.7% pairwise agreement). I assess each claim against the presented evidence.

Claim 1 (benchmark scope and LES enablement): The paper clearly demonstrates coverage of 100 scenarios across 10 industry categories and 65 domains, with 382 evaluation instances. The scenario descriptions in Table 1 and the case studies (Figures 13–17) provide concrete evidence that these are genuine professional task scenarios (emergency triage, fish farm water quality control, building inspection compliance, etc.), not generic reasoning problems disguised as professional tasks. The LES formalization (Section 3) and the synthesis pipeline (Section 4) describe how these instances were generated, and the solvability verification (executing each instance with a reference plan and filtering unsolvable ones) provides a meaningful quality guarantee. However, the paper does not provide human validation of the generated instances. No domain expert evaluated whether the emergency department triage scenario actually reflects real triage protocols, whether the customs declaration task reflects real customs procedures, or whether the nuclear safety scenario captures genuine safety-critical workflows. The claim that these scenarios represent "real-world professional tasks" rests entirely on: (a) the paper's assertion that LLMs encode domain knowledge, and (b) the solvability verification, which only confirms that a solution exists within the simulated environment, not that the solution or the environment reflects real professional practice. This is a significant gap. A small-scale expert validation—having, say, 5 ER physicians review the triage scenario and confirm it captures realistic decision-making—would substantially strengthen this claim. Without it, we must accept on faith that Gemini-3-Flash-Preview's synthesized environment configurations accurately represent professional domains, which is exactly the assumption the paper itself problematizes in Section 6.6 (showing that GPT-5.2 fabricates environment state). The simulator that synthesizes the benchmark may have similar but undetected failure modes.

Claim 2 (no model dominates all industries, distinct occupational profiles): This claim is strongly supported by the data. Table 2 and Figure 2 provide unambiguous evidence of industry-specific capability variation. GPT-5.2's Commerce score (67%) is 14 points below Qwen 3.5 Plus (81%); Gemini 3.1 Pro's Healthcare score (62%) is 19 points below Qwen 3.5 Plus (81%); Claude Opus 4.6's Commerce score (53%) is the lowest among top models. These are large gaps, not sampling noise at the margins. The case study in Figure 13 (last-mile delivery) illustrates the mechanism: a constraint-checking failure (battery management) that is specific to transportation/logistics tasks, not a general reasoning failure. However, the per-industry sample sizes are small and imbalanced. Business & Enterprise has 19 scenarios; Science & Research has 4. A model's Industry score is an average over its constituent scenarios, and with only 4 Science scenarios, a single anomalous scenario can substantially shift the score. The paper reports no confidence intervals on per-industry completion rates, so we cannot assess whether, for example, Gemini 3.1 Pro's Science lead (81% vs. GPT-5.2's 94%) is statistically reliable or within noise given the small denominator. This matters particularly for the claim that models have "distinct occupational capability profiles"—if the per-industry estimates have wide confidence intervals due to small scenario counts, the apparent profile shapes may be partially noise artifacts. A bootstrap analysis or Clopper-Pearson confidence intervals would clarify what degree of profile differentiation is actually distinguishable.

Claim 3 (implicit faults are harder than explicit and mixed): This claim is supported by the aggregate statistics (Table 3: E2 average 53.4% vs. E1 62.6%), but with important caveats. The gap direction is consistent across 7 of 9 models (Claude Opus 4.6, Gemini Pro, GPT-5.2, GLM-5, Qwen 3.5 Plus, Claude Sonnet 4.6, Kimi K2.5). Two models show deviation: MiniMax M2.7 has E1 (52.9%) < E2 (47.1%)—consistent—but DeepSeek V3.2 has E2 (56.0%) > E3 (51.6%), meaning E3 is harder than E2 for this model, contrary to the paper's framing that E2 is the hardest overall. The paper's explanation for why E2 is harder than E3 (E3's explicit errors trigger retry behavior that also helps with implicit issues) is an interpretation, not a demonstrated mechanism—no experiment isolates whether retry behavior actually accounts for the E3 advantage. A targeted ablation could test this: run E3 but suppress the retry behavior, or run E2 with an explicit instruction to re-query uncertain responses, and measure the effect.

More critically, the fault injection is synthetic and controlled—2 fault events of 2 rounds each by default—which is a far cry from the unpredictable, correlated, context-dependent faults of real production environments. The paper's fault taxonomy (explicit vs. implicit) is conceptually useful but leaves out important real-world failure modes: adversarial faults (an attacker deliberately corrupting specific data fields), correlated cascading faults (a network partition causing multiple simultaneous timeouts), persistent faults (not transient—the system is genuinely down), and semantically misleading responses (the API returns valid JSON with correct schema but subtly wrong values, e.g., returning a patient's weight in pounds labeled as kilograms). The implicit faults tested (truncated data, missing fields) are a narrow subset of real-world data degradation. The paper's claim that this is a "systematic assessment of how agents handle environmental faults" (Section 1) is true but narrower than the phrase suggests to a practitioner.

Claim 4 (scaling consistently improves performance): This claim breaks into three sub-claims: model size scaling (larger > smaller), generational scaling (newer > older), and reasoning effort scaling (more > less). The model size scaling evidence (Figure 5) is the strongest: 4 of 5 within-family comparisons show gaps of 7–11 points, with one exception (Claude 4.5, 0.3 points) that is plausibly within noise. The generational scaling evidence (Figure 6) shows consistent improvement for Claude Opus (+10.2% over two generations) but a slight regression for Claude Sonnet v4.5 → v4.6 (−0.5%), which is probably noise but cannot be confirmed without confidence intervals. The reasoning effort scaling evidence (Figure 7) is the most dramatic: GPT-5.2 improves 27.5 points, Claude Opus 4.6 improves 3.6 points. The monotonicity of both curves supports the "consistently improves" framing. However, the reasoning effort ablation only tests two models—GPT-5.2 and Claude Opus 4.6—so the claim that "higher reasoning effort consistently improves performance" is demonstrated for these two specific models, not for the general class of reasoning-enabled LLMs. It would be strengthened by including at least one more model with configurable reasoning effort (e.g., Gemini 3.1 Pro or DeepSeek V3.2 if they support it). Additionally, the paper does not control for or measure the actual compute cost of increased reasoning effort—it treats reasoning effort as an ordinal setting (none, low, medium, high, max/xhigh) without reporting the associated token cost or latency. Without this information, a practitioner cannot make cost-benefit tradeoffs: is the 27.5-point gain from GPT-5.2 "none" to "xhigh" worth a 2×, 5×, or 10× increase in inference cost? The paper cannot say, because it does not measure the cost.

Claim 5 (strong agents are not necessarily strong simulators, but capable simulators yield consistent rankings): This claim is well-supported by the cross-simulator experiment (Table 4, Figures 8–11). The evidence that GPT-5.2 is a poor simulator is convincing—the collapsed absolute scores, the specific failure modes in the case studies, and the 37.2-point drop in GPT-5.2's own score when evaluated under itself all point in the same direction. The evidence for ranking consistency with capable simulators is moderate: 85.7% pairwise agreement between Gemini Flash and Qwen 3.5 Plus, with top-3 match. However, 85.7% agreement means 4 of 28 model pairs are misordered between the two simulators. These disagreements involve mid-ranked models (Qwen 3.5 Plus, DeepSeek V3.2, GLM-5, Kimi K2.5), and the paper does not analyze which specific scenarios or industries drive the disagreements. If the disagreements cluster in particular domains where one simulator is systematically less reliable, that would be important information for interpreting results in those domains. The paper also does not test whether the Qwen simulator, like GPT-5.2, exhibits state fabrication, entity omission, or rule invention—it assumes Qwen is reliable because its absolute scores are close to Gemini's and its rankings agree well, but this is circular: agreement with Gemini Flash is evidence of reliability only if we already trust Gemini Flash. A human evaluation of a sample of Qwen-simulated trajectories would provide independent validation.

Missing experiments that would strengthen the paper:

Human expert validation of scenario realism. Even a small-scale study (10–20 scenarios reviewed by domain experts) would substantiate the claim that these represent genuine professional tasks.
Confidence intervals on all reported completion rates. With 382 instances, per-industry rates have substantial uncertainty, especially for categories with 4–7 scenarios. Without confidence intervals, readers cannot assess whether industry-to-industry differences are statistically distinguishable.
Cost reporting for reasoning effort. The paper presents reasoning effort as a free knob, but it costs tokens and latency. Reporting the cost would make the scaling curves actionable.
Fault type ablation beyond count and duration. Testing different distributions of faults (clustered vs. uniformly spaced), persistent faults (non-transient), or adversarially targeted faults (injected at specific critical tool calls) would provide a richer robustness picture.
Multiple simulators for benchmark synthesis. The paper uses only Gemini-3-Flash-Preview to synthesize the benchmark. If synthesis were repeated with a different simulator (e.g., Qwen 3.5 Plus), would the same instances be generated? Would the difficulty ordering be preserved? This would test whether the benchmark itself is simulator-dependent, not just the evaluation.
Direct comparison to an existing benchmark in an overlapping domain. OCCUBENCH claims to cover domains no other benchmark covers, but a small-scale head-to-head on, say, a business operations scenario against a TAU-bench retail task would help calibrate what OCCUBENCH's scores mean relative to the established evaluation landscape.

Overall assessment: The experimental results provide credible evidence for the paper's main qualitative claims—cross-industry capability variation, implicit fault vulnerability, scaling benefits, and the simulator-agent capability gap. The case studies are particularly effective at illustrating the mechanisms behind the aggregate statistics. However, the absence of statistical rigor (no confidence intervals, no significance tests), the lack of human validation of scenario realism, and the narrow scope of the fault injection taxonomy mean that the quantitative results (specific completion rates, specific robustness scores, specific ranking orders) should be treated as point estimates with unknown uncertainty rather than precise measurements. The benchmark's unique value—cross-industry professional task coverage enabled by LESs—is genuine, but its reliability for drawing fine-grained conclusions about model capability ordering (beyond broad tiers) is not yet established.

6. Limitations and Trade-offs

6.1 Simulation Fidelity: Domain Logic Without Domain Data

The assumption or constraint. The LES architecture models domain logic rather than domain data. As the paper explicitly acknowledges in Section 8.1:

"Language Environment Simulators model domain logic rather than domain data. An LES understands that a drug interaction check should return contraindications, but the specific values it returns are generated rather than retrieved from a real database. This means OCCUBENCH evaluates an agent's decision-making process (whether it checks the right things in the right order) rather than its ability to handle exact real-world data values."

This is a fundamental scope limitation: the benchmark measures process correctness, not data precision.

The consequence. For professional domains where precise numerical correctness is critical—financial calculations to the cent, drug dosages to the milligram, engineering tolerances to the micron—the LES may return plausible but fictitious values. An agent that makes correct decisions based on incorrect data would pass verification (the rubric checks process), while an agent that catches the data anomaly and refuses to proceed would fail (the rubric expects completion). The paper acknowledges this directly: "For domains where precise numerical correctness is critical (e.g., financial calculations to the cent), LES-based evaluation should be complemented with real-environment testing" (Section 8.1). However, this complement is not provided, and several OCCUBENCH scenarios fall squarely in these precision-critical domains—hotel revenue management (calculating ADR, CPOR, NOI), property valuation assessment (computing DSCR to two decimal places), and building inspection (verifying pressure test results). In these cases, the benchmark is evaluating decision-making around numbers that may be arbitrary.

What evidence exists in the paper. The paper provides no quantitative measurement of data fidelity—no comparison between LES-generated values and ground-truth values from real professional systems, no analysis of whether the numerical values in generated observations are internally consistent or realistic. The cross-simulator analysis (Section 6.6) provides indirect evidence: GPT-5.2's rule invention (Figure 11) involves independently computing a date difference and enforcing a return-window constraint not in the specification, demonstrating that LLMs can generate numerically precise but logically incorrect constraints. This failure mode (overprecise fabrication) is the mirror image of the more common concern (vague plausibility): the simulator can be too precise in enforcing rules it invented.

Mitigation status. The paper is transparent about this limitation but treats it as a scope boundary rather than a problem to solve. The recommendation is to complement LES-based evaluation with real-environment testing for precision-critical domains, but no methodology for doing so is provided. Future work could explore calibrating LES-generated values against real distributions, injecting known ground-truth data into configurations, or flagging precision-critical task steps for human verification.

6.2 The Untested Assumption of Scenario Realism: No Human Expert Validation

The assumption or constraint. The entire benchmark rests on the claim that the 100 scenarios represent "real-world professional tasks" and that each "maps to a real human job role" (Section 5.1). This claim is supported by the paper's argument that LLMs encode domain knowledge (Section 3.3), and by the solvability verification in the synthesis pipeline (Section 4), which confirms that a solution exists within the simulation but says nothing about whether the simulation reflects real professional practice.

The consequence. Without human expert validation, we cannot distinguish between a scenario that genuinely captures professional decision-making (e.g., an emergency triage nurse actually follows a process resembling what the LES simulates) and a scenario that captures a plausible but incorrect caricature of the profession (e.g., the LES invents a plausible-sounding triage protocol that no real hospital follows). This matters because the paper draws strong conclusions about model "fitness for professional work" (Section 1) and makes deployment-relevant claims about which models excel in which industries (Section 7.2). If the Healthcare scenarios do not reflect real healthcare workflows, then the finding that "Qwen 3.5 Plus leads Healthcare at 81%" (Table 2) is measuring capability on a proxy task, not on genuine healthcare professional work.

The cross-simulator analysis (Section 6.6) compounds this concern. The paper demonstrates that GPT-5.2, when acting as a simulator, fabricates environment state, drops entities, and invents business rules. The benchmark itself was synthesized by Gemini-3-Flash-Preview—a different model, but one whose simulator-side behavior has not been validated against human expert judgment. The paper has shown that LLM-based simulators can be unreliable; it has not shown that Gemini-3-Flash-Preview is reliable for the specific task of benchmark synthesis, only that the instances it generates are solvable. Solvability within an LLM-generated simulation does not imply professional realism.

What evidence exists in the paper. Zero. The paper reports no human expert review of any scenario. No domain expert—no emergency physician, no customs officer, no nuclear safety engineer—is cited as having reviewed or validated the benchmark content. The 382 evaluation instances are accepted as professionally representative based solely on: (a) the LLM's pre-trained knowledge, (b) the professional reference documents the authors wrote to guide generation, and (c) the solvability and difficulty calibration filters.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation—the closest it comes is the data fidelity discussion in Section 8.1, which is about the values returned by the LES during evaluation, not about the validity of the scenarios themselves. A small-scale expert validation study (5–10 scenarios reviewed by practicing professionals in the relevant domains) would substantially strengthen the benchmark's credibility and is a clear priority for future work. Without it, the "real-world" in "real-world professional tasks" should be understood as "plausibly real-world to an LLM," which is a weaker and less actionable claim.

6.3 Statistical Uncertainty Is Unquantified: No Confidence Intervals on Key Metrics

The assumption. All reported completion rates, robustness scores, and per-industry averages are point estimates with no quantification of uncertainty. The 382 evaluation instances are treated as a fixed set, and all percentages are reported to one decimal place, but no confidence intervals, bootstrap intervals, or significance tests accompany any result.

The consequence. The paper draws fine-grained conclusions that require statistical reliability: model A outperforms model B in Industry X, implicit faults are harder than explicit faults, the Claude Sonnet v4.5 → v4.6 regression (−0.5 points) is a "slight regression" reflecting an architectural trade-off, and the 85.7% pairwise simulator agreement indicates "highly consistent" rankings. Without uncertainty quantification, none of these conclusions can be assessed for robustness to instance sampling.

The problem is severe for per-industry metrics because the scenario counts are small and imbalanced: Science & Research has 4 scenarios, Education & Culture has 8, Healthcare has 7, Public Service has 7, Agriculture has 7. With only 4 Science scenarios, the 95% Clopper-Pearson confidence interval for a completion rate of 94% (GPT-5.2's Science score) might be approximately 70–100%—wide enough that the claim "GPT-5.2 leads in Science (94%)" could be consistent with a true capability anywhere from barely-competent to near-perfect. Similarly, with 7 Healthcare scenarios, Qwen 3.5 Plus's 81% could have a confidence interval of roughly 48–98%, making its apparent 5-point lead over GPT-5.2 (76%) statistically indistinguishable from noise.

The cross-simulator agreement metric (85.7%) is based on 28 model pairs. The probability of observing 24/28 agreements under random ordering is extremely low, so the qualitative finding of above-chance agreement is robust, but the specific 85.7% figure has its own uncertainty. A bootstrap of the 28 pairs would reveal whether the four disagreements are concentrated in a small number of unstable comparisons.

What evidence exists in the paper. The paper provides no confidence intervals, no bootstrap analyses, no significance tests, and no discussion of statistical power anywhere in the manuscript. The cross-simulator experiment (Table 4) reports exact completion rates and rankings but does not test whether specific pairwise comparisons (e.g., Qwen 3.5 Plus vs. DeepSeek V3.2) are statistically distinguishable under either simulator. Figure 2 (radar chart) visually suggests distinct profiles, but without error bars, a reader cannot assess whether the visual differences exceed sampling noise.

Mitigation status. Not addressed. The paper's conclusions are presented with the implicitness of statistical certainty, but the sample sizes (382 total instances, as few as 4 per industry category) do not support the precision implied by the single-decimal-point reporting. Future work should include bootstrap-based confidence intervals on all per-industry and per-fault-condition completion rates, and should indicate when between-model or between-industry differences are not statistically significant. For the current paper, readers should treat all specific numerical comparisons—especially per-industry comparisons and the Claude Sonnet v4.5 → v4.6 regression—as suggestive rather than definitive.

6.4 Simulator Coupling: The Benchmark Is Tied to a Specific LES Model

The assumption. All 382 evaluation instances were synthesized, verified, and difficulty-calibrated using a single simulator model: Gemini-3-Flash-Preview. The solvability guarantee (Section 4) means each instance has been executed successfully with a reference plan under this specific LES. The instance selection—choosing the difficulty level with the lowest autonomous success rate—was also performed under this LES.

The consequence. If a different LES were used for evaluation—say, Qwen 3.5 Plus or a future model—tasks that are solvable under Gemini-3-Flash may become unsolvable (as the cross-simulator experiment demonstrates), and the selected difficulty levels may no longer be optimal. More subtly, the difficulty ordering of instances may change: a task that Gemini-3-Flash renders moderately difficult might be trivially easy under a different simulator, or vice versa, because different simulator models have different failure modes (state fabrication, entity omission, rule invention) that affect different scenarios differently.

This matters because the benchmark is intended to be a lasting evaluation resource. As LLMs improve and the default simulator is updated (or as researchers choose different simulators for cost, speed, or reproducibility reasons), the benchmark's difficulty calibration, solvability guarantees, and even the validity of individual instances may drift. The paper's own cross-simulator experiment (Section 6.6) demonstrates this drift quantitatively: absolute scores shift dramatically between simulators (average 67.9% → 63.4% → 29.3%), and while rankings are partially preserved, they are not invariant. This means OCCUBENCH scores are not absolute measurements of agent capability—they are measurements of agent capability under a specific simulator, and the distinction matters.

What evidence exists in the paper. The cross-simulator analysis (Table 4, Figure 8) directly demonstrates this limitation for evaluation. The paper does not, however, test the analogous question for benchmark synthesis: if the entire synthesis pipeline were re-run with a different simulator model (e.g., Qwen 3.5 Plus), would the same scenarios produce the same instances? Would the solvability filtering select the same tasks? Would the difficulty calibration select the same difficulty levels? The paper provides no evidence that the benchmark itself is stable across simulator choices for synthesis.

Mitigation status. The paper acknowledges simulator dependence as a limitation of evaluation in Section 8.1: "evaluation results are tied to the specific simulator used during data synthesis. Tasks verified as solvable under Gemini-3-Flash may become unsolvable under a different LES." However, it frames this as an inherent constraint rather than a problem with a solution: "This is an inherent limitation of any LES-based evaluation: the simulator is part of the evaluation apparatus, not a neutral observer." The paper recommends verifying simulator quality and re-verifying solvability when switching simulators, but provides no methodology for doing so systematically. The coupling between benchmark synthesis and simulator model means that the benchmark's validity is only as strong as Gemini-3-Flash-Preview's reliability as a professional domain simulator—and as Section 6.6 demonstrates, even frontier models can be unreliable simulators.

6.5 Fault Injection Taxonomy Is Synthetic and Narrow: Gaps in Real-World Robustness Coverage

The assumption. The environmental robustness evaluation (Section 5.2) is built around a specific, controlled fault taxonomy: explicit faults (HTTP 500, TimeoutError, ConnectionRefused, ServiceUnavailable) and implicit faults (truncated data, missing fields, incomplete lists, empty/null values, stale cached values). All faults are transient (retrying eventually recovers normal results), spaced across the interaction, and parameterized by a single fault count and fault duration.

The consequence. This taxonomy, while systematic, excludes several fault categories that are common in real production environments and that would stress-test different agent capabilities:

Persistent faults: A service that is actually down, not transiently failing. The correct behavior is not to retry indefinitely but to escalate, use a fallback, or gracefully degrade. OCCUBENCH's transient-only design means retry-always is always the correct strategy, which rewards persistence rather than discriminating between appropriate retry and appropriate abandonment.
Correlated cascading faults: A network partition causing multiple simultaneous timeouts across different services, requiring the agent to recognize the systemic nature of the failure rather than treating each timeout independently. The current injection model treats faults as independent events.
Adversarially targeted faults: Faults injected at specifically critical tool calls—the one call that checks a safety constraint, the final verification step, the data fetch that the entire subsequent calculation depends on. The current random spacing may or may not hit critical calls, meaning the measured robustness may underestimate or overestimate real-world vulnerability depending on whether the random faults happen to land on critical actions.
Semantically misleading responses: The LES returns valid JSON with correct schema but subtly wrong values—e.g., returning a patient's weight in pounds labeled as kilograms, or returning yesterday's readings labeled as today's. This is a more insidious form of implicit fault than truncation because there is no structural anomaly to detect; only domain knowledge can catch the error.
Timeout-like delays without errors: The response arrives correctly but 30 seconds late, causing the agent to exceed a time budget. This is a common real-world fault (slow downstream services) that affects latency-sensitive deployments but is not modeled.

The paper's implicit fault finding—that E2 is harder than E1 and E3—may be partially an artifact of this narrow fault taxonomy. If additional implicit fault types were included (semantically misleading responses, stale-but-structurally-valid data), the E2 performance might drop further, or the mechanism of failure might shift from "failed to detect truncation" to "detected problem but could not determine correct alternative."

What evidence exists in the paper. The fault parameter ablation (Figure 4) tests fault count and duration but does not vary fault type beyond the explicit/implicit/mixed taxonomy. The case study for E2 (Figure 17) shows truncated data (2 of 15 units returned), which is a clean example of structural degradation, but no example is provided of a semantically misleading response. The paper's claim that this is "a systematic assessment of how agents handle environmental faults" (Section 1) is accurate relative to prior benchmarks (which do zero fault injection) but narrow relative to the full space of production faults.

Mitigation status. The paper does not acknowledge this as a limitation—the fault taxonomy is presented as comprehensive within its intended scope. The robustness score R is defined over only the three fault types (E1, E2, E3), meaning it measures resilience to a specific controlled fault distribution, not general environmental robustness. Future work could expand the fault taxonomy to include persistent faults, correlated failures, adversarial targeting, and semantically misleading responses, and could measure whether the implicit-explicit difficulty gap (E2 > E1) generalizes across a broader set of fault types or is specific to structural data degradation.

6.6 The Hardest Professional Tasks Remain Effectively Unsolvable Across All Models

The assumption. The benchmark is designed to measure agent capability across a wide range of professional task difficulties. The instance selection deliberately chooses the difficulty level with the lowest autonomous success rate (Section 4) to maximize discriminative power. The implicit assumption is that current models are capable enough that the benchmark can differentiate them.

The consequence. While the paper does not report a formal "unsolvable" category as the prior example paper does with its "bin 5" hardest problems, the substantial performance gaps and industry-specific failures reveal that for certain domains and models, performance is near floor. Claude Opus 4.6 achieves 53% in Commerce, meaning it fails roughly half of commerce tasks. Claude Sonnet 4 achieves 31% in Education and 35% in Agriculture. MiniMax M2.7 averages 53.9% overall with a low of 31% in Education. Even GPT-5.2's 67% in Commerce means it fails one-third of commerce tasks. These are not ceiling effects—they are evidence that for substantial subsets of professional work, even the strongest current models are not reliably competent.

This matters for deployment decisions because the benchmark's discriminating power at the low end is coarse. The difference between 31% and 38% in Education tells a practitioner that both models are inadequate for autonomous deployment in educational professional tasks, but does not tell them whether the 7-point gap reflects a meaningful capability difference or whether both models share a common failure mode (e.g., inability to reason about curriculum sequencing) that no current model overcomes. The paper's cross-industry capability profiling (Section 7.2) interprets industry-level strengths and weaknesses as occupational specializations, but for domains where even the best model scores poorly (Commerce best is 81% for Qwen 3.5 Plus, but Claude Opus 4.6 scores 53%; Transportation best is 77% for Claude Opus 4.6, but Qwen 3.5 Plus scores 55%), the "specialization" may reflect some models being merely less bad rather than genuinely capable.

What evidence exists in the paper. The full completion rate table (Table 2) reveals substantial variation from 31% to 94% depending on model and industry. Figure 12 shows that Transportation (56.2%) and Education (57.6%) are the hardest industries on average across all 15 models—meaning the average model fails nearly half of transportation and education tasks. The paper does not analyze whether failures in these hard industries share common causes across models (suggesting a fundamental capability gap that scaling might not address) or whether different models fail for different reasons (suggesting specialization that could be combined through model routing or ensembles).

Mitigation status. The paper does not frame low performance in hard industries as a limitation—it presents industry difficulty variation (Figure 12) as a feature of the benchmark that reveals domain complexity. However, from a practitioner's perspective, a benchmark that shows your best option fails 33–47% of the time on important tasks is a signal that the technology is not ready for autonomous deployment in those domains, regardless of how well it ranks relative to other models. The paper does not provide guidance on what completion rate threshold constitutes "deployment-ready" for a given professional domain, which limits the actionable interpretation of the results for organizations making build-vs-buy or deploy-vs-wait decisions. This is a fundamental limitation of the benchmark's framing as a comparative evaluation tool: it tells you which model is best, but not whether any model is good enough.

7. Implications and Future Directions

How This Work Changes the Landscape

OCCUBENCH changes the evaluation landscape by making a previously impossible claim—"we can systematically evaluate AI agents on professional occupational tasks"—into a demonstrable engineering reality. The shift is methodological, not merely incremental: the paper demonstrates that environment construction can be reframed from an engineering problem to a configuration problem, replacing deployed software infrastructure with declarative specifications executed by an LLM. This is not a new model architecture or a better training objective—it is a different way of thinking about what an evaluation environment is, and the consequences ripple through how the field can allocate its measurement effort.

The magnitude of the shift is best understood by what becomes newly tractable. Before OCCUBENCH, the set of evaluable professional domains was bounded by the availability of public environments—web applications for WebArena, VMs for OSWorld, live APIs for TAU-bench. This meant the research community could measure agent capability on perhaps a dozen domains total, concentrated in web browsing, desktop operations, code editing, and customer service. After OCCUBENCH's methodology, the evaluable set expands to any domain an LLM can understand—which, as the paper demonstrates with 65 specialized domains across 10 industries, is vast. The scaling economics are qualitatively different: adding a domain to WebArena requires deploying and configuring web applications (engineering effort per domain); adding a domain to an LES-based benchmark requires writing a configuration (specification effort per domain). The paper's coverage of 100 professional scenarios at once—from emergency triage to customs processing to nuclear safety monitoring—is not a larger version of what came before; it is a different category of coverage, enabled by a different category of methodology.

This reframing reconciles a contradiction that has been latent in agent evaluation rather than actively debated: the tension between domain breadth and evaluation realism. Prior benchmarks implicitly accepted a tradeoff—you could have realism (real web applications, real VMs, real APIs) but narrow coverage, or you could broaden coverage through synthetic tasks but sacrifice the interactive, stateful, multi-step nature of real professional work. OCCUBENCH shows this tradeoff is not fundamental; it is an artifact of treating "environment" as "software that must be built." By redefining "environment" as "behavioral specification executed by an LLM," the paper achieves breadth (65 domains) without sacrificing the interactive, stateful, multi-step structure that distinguishes agent evaluation from static QA or function-calling benchmarks. The average of 16.2 tool calls per task across 382 instances confirms that these are genuine multi-turn interactions, not single-step queries dressed up as professional tasks.

The paper also reorients research priorities in agent robustness. Before OCCUBENCH, robustness to environmental faults was simply not evaluated—no benchmark had the architectural flexibility to inject controlled faults across diverse environments. The finding that implicit faults (E2, average 53.4%) are substantially harder than explicit faults (E1, average 62.6%)—a 9.2-point gap across all models—establishes a new diagnostic dimension that no prior evaluation framework could measure. This redirects attention from the question of "which model is best?" to "which model is most resilient, and to what?" The implication for research is that improving clean-environment task completion and improving fault resilience may require different technical interventions—better reasoning improves both, but the implicit fault vulnerability (where agents accept truncated data as complete, Figure 17) suggests a specific gap in metacognitive monitoring (checking whether received data is sufficient for the task) that better reasoning alone may not close. The robustness score metric R = min(CR_E1, CR_E2, CR_E3) / CR_E0 provides a concrete optimization target for future work on resilient agents.

The cross-simulator finding—that GPT-5.2, the strongest agent model, produces the worst environment simulation (causing all scores to collapse to an average of 29.3%, Table 4)—introduces a new constraint on simulation-based evaluation that the field had not previously articulated. The implicit assumption in prior work using LLMs as world models or simulators was that a capable LLM is adequate for the role. OCCUBENCH demonstrates this assumption fails in specific, diagnosable ways (state fabrication, entity omission, rule invention), and that simulator quality is a first-order determinant of evaluation validity, not a secondary concern. This finding makes certain research directions more attractive—verifying simulator reliability before drawing conclusions, developing simulator quality metrics, understanding why some models make good agents but poor simulators—and makes one research direction less attractive: the naive approach of using the strongest available model as the simulator without verification. The 85.7% pairwise ranking agreement between capable simulators provides an empirical reliability target that future simulation-based benchmarks can aim to match or exceed.

Finally, the paper's cross-industry capability profiling—showing that every model has a distinct occupational capability shape (Figure 2), with GPT-5.2 leading overall but trailing Qwen 3.5 Plus by 14 points in Commerce and Gemini 3.1 Pro leading Education by 22 points over its own Healthcare score—changes how model selection should be done for professional deployment. The dominant practice of using aggregate leaderboard rankings to choose a model implicitly assumes domain-invariant capability. OCCUBENCH provides systematic evidence that this assumption is false for professional tasks, and that the information loss from aggregation is severe enough to reverse capability orderings between industries. This is not a finding that shifts the research frontier of model architecture; it shifts the evaluation practice that the entire field uses to measure progress.

Follow-Up Research This Work Enables

Human expert validation of scenario realism. The paper's most important unvalidated claim is that the 100 scenarios represent genuine professional work. A targeted study would recruit 10–15 domain experts (emergency physicians, customs officers, network security analysts, production schedulers, etc.), present each with 2–3 OCCUBENCH scenarios in their domain of expertise, and ask them to rate on a Likert scale: (a) whether the task reflects real professional decision-making, (b) whether the tool schema captures realistic operational interfaces, and (c) whether the verification rubric correctly identifies competent vs. incompetent performance. Expert feedback would calibrate the benchmark's claim to professional realism—providing per-industry realism scores—and identify specific scenarios where the LLM-synthesized environment deviates from real practice. This study is newly tractable because OCCUBENCH provides the concrete artifact to validate; before it existed, there were no professional-domain agent scenarios to show experts.

Diagnostic analysis of industry-specific failure modes. The paper shows that different models have different occupational capability profiles, but does not analyze why Model A fails on Commerce tasks that Model B passes. A follow-up study would take the 382 evaluation trajectories from the top 5 models and categorize failures by type: skipped verification steps (as in the fish farm case, Figure 14), procedural ordering errors (as in the building inspection case, Figure 15), constraint violations (as in the battery management case, Figure 13), premature task abandonment, tool misuse (calling the right tool with wrong arguments), and state tracking errors (forgetting information acquired earlier). This would produce a failure taxonomy that maps model architectures to characteristic weaknesses—do Claude models fail differently from GPT models? Do reasoning-heavy models make different types of errors than reasoning-light variants? The paper's trajectory-aware verification (scoring the full tool-call sequence, not just final answers) makes this analysis tractable because failures are already associated with specific steps in the execution trace. The result would inform model selection for practitioners (choose Model A for Commerce not just because it scores higher, but because it specifically doesn't skip verification steps) and architecture development (if all models fail on constraint-checking, that capability needs targeted improvement).

Expanding the fault taxonomy to persistent, correlated, and adversarial faults. The paper's finding that implicit faults (E2) are harder than explicit faults (E1) is based on a narrow fault taxonomy: transient structural degradation (truncated data, missing fields). A follow-up would inject three new fault types not tested in the paper: (a) persistent faults: the service is genuinely down for the remainder of the task, requiring the agent to abandon the failing approach and find an alternative strategy; (b) correlated cascading faults: a single root cause (network partition) causes simultaneous timeouts across multiple independent tool calls, testing whether the agent recognizes systemic failure vs. treating each timeout independently; and (c) semantically misleading responses: the LES returns valid JSON with correct structure but subtly wrong values (e.g., weight in pounds labeled as kilograms, yesterday's readings labeled as today's), testing whether agents can catch errors that require domain knowledge rather than structural anomaly detection. These would test whether the implicit-explicit difficulty gap generalizes: do agents that survive structural truncation also survive semantic poisoning? The paper's fault injection framework (appending fault rules to the LES system prompt) makes adding new fault types a configuration change rather than a re-engineering effort, making this follow-up practically straightforward.

Cross-simulator benchmark synthesis: measuring the stability of the benchmark itself. The paper's cross-simulator experiment (Section 6.6) shows that evaluation results depend on the simulator, but does not test whether the benchmark itself (the 382 instances, their difficulty calibration, and the selected difficulty levels) would change if synthesized by a different model. A direct replication would re-run the entire synthesis pipeline (Section 4) using Qwen 3.5 Plus instead of Gemini-3-Flash-Preview as the generation engine, producing a second set of instances from the same 100 scenario specifications. The key measurements: (a) what fraction of Gemini-synthesized instances are reproduced (same task structure, same tool schema, similar initial state) by Qwen? (b) do the difficulty calibrations agree—does the same difficulty level get selected as "lowest autonomous success rate"? (c) if the same agent models are evaluated on both benchmark versions, do the industry-level capability profiles and overall rankings remain stable? This would quantify how much of OCCUBENCH's signal comes from the scenario specifications (which are human-designed) vs. from the specific LLM used to instantiate them, establishing whether LES-based benchmarks have genuine test-retest reliability across simulators or whether they are intrinsically coupled to their synthesis engine.

Combining OCCUBENCH-style professional task evaluation with real-environment testing for precision-critical domains. The paper acknowledges that LES-based evaluation should be complemented with real-environment testing when numerical precision matters (Section 8.1), but provides no bridge between the two. A follow-up would select 3–5 OCCUBENCH scenarios that have natural counterparts in existing benchmarks or deployable test environments: for example, a hotel revenue management scenario (Commerce) could be implemented against TAU-bench's airline API with adapted booking and pricing tools; a network intrusion response scenario (Technology & IT) could be deployed on a containerized network simulation with actual firewall and monitoring tools; a financial auditing scenario (Business) could use a small real accounting database with ground-truth transactions. The same agent models would be evaluated on the LES version and the real-environment version of each task, measuring correlation between LES scores and real-environment scores. This would establish the critical calibration: for what types of professional tasks does LES-based evaluation accurately predict real-environment performance, and for what types does it diverge? The paper's existing methodology provides the LES half; the contribution would be closing the loop with ground truth.

Training agents specifically for implicit fault resilience using OCCUBENCH as a training signal. The paper's E2 results show that even strong models fail to detect truncated data, with Kimi K2.5 producing a catastrophic financial miscalculation (DSCR 1.72x vs. actual 1.19x, Figure 17) because it assumed 2 returned units represented all 15. A training intervention would fine-tune a base agent model on trajectories that include explicit demonstrations of data quality checking: before using the results of any data-fetching tool call, the agent is trained to verify completeness (e.g., "Received 2 units. Expected 15 based on property metadata. Re-querying for complete data.") and to cross-check quantities against metadata. The training data could be generated by running the LES with E2 fault injection, capturing trajectories where the reference plan (which includes verification steps) succeeds, and using those as supervised fine-tuning examples. The paper's synthesis pipeline already generates reference plans with solution steps; modifying those plans to include explicit verification sub-steps and training on them would test whether implicit fault resilience can be taught rather than requiring general reasoning improvements. The evaluation would measure whether the fine-tuned model's E2 completion rate approaches its E0 rate, closing the implicit fault gap.

Practical Applications and Downstream Use Cases

Model selection for industry-specific agent deployment. The paper's finding that no model dominates all industries, with capability reversals like Qwen 3.5 Plus leading GPT-5.2 by 14 points in Commerce (81% vs. 67%) and Claude Opus 4.6 leading Gemini 3.1 Pro by 15 points in Transportation (77% vs. 62%), means that organizations deploying AI agents for professional work should select models on a per-industry basis, not on aggregate leaderboard position. A retail company deploying an agent for dynamic pricing and order management (Commerce) would be better served by Qwen 3.5 Plus than by GPT-5.2 despite GPT-5.2's superior aggregate score (79.6% vs. 69.9%), because the 14-point Commerce gap translates to a substantially higher task completion rate on the specific type of work the agent will perform. Conversely, a logistics company deploying an agent for last-mile delivery routing (Transportation) would prefer Claude Opus 4.6 (77%) over Qwen 3.5 Plus (55%), a 22-point gap that would be invisible if selection were based on overall ranking alone. The cross-industry capability profiling in Table 2 and Figure 2 provides the specific per-industry scores that make this informed selection possible.

Robustness-aware deployment gating. The environmental robustness results (Table 3) enable organizations to set deployment readiness thresholds that account for fault resilience, not just clean-environment performance. A model that scores 79.6% in E0 but drops to 67.0% under mixed faults (GPT-5.2, robustness 0.84) may still be deployment-ready for fault-tolerant applications where occasional failures are acceptable, but a model that drops from 64.1% to 40.1% under implicit faults (Kimi K2.5, robustness 0.63) poses a material risk for any application where data quality issues are possible—which is most real-world settings. The specific finding that implicit faults (truncated data, missing fields) cause larger degradation than explicit errors (timeouts, 500s) means that organizations should specifically test their candidate models on E2-like conditions before deployment, and should invest in agent architectures that include explicit data quality verification steps. The case study in Figure 17 provides a concrete example of the business impact: an agent that accepts truncated data without verification can produce financially catastrophic errors (reporting DSCR 1.72x for a property that actually scores 1.19x, reversing a covenant compliance decision). A deployment process gated on E2 performance above a domain-specific threshold would catch this vulnerability before production.

Benchmark-driven iteration on agent architecture for professional tasks. The failure modes revealed by the case studies—skipped verification steps (Figure 14, fish farm), procedural ordering errors (Figure 15, building inspection), and constraint-checking failures (Figure 13, last-mile delivery)—provide concrete targets for architectural improvements to agent design. A team building an agent for professional work could use OCCUBENCH as a development benchmark, running their agent on the 382 instances, categorizing failures using the trajectory-aware verification output (which identifies the specific tool call where the failure occurred and what constraint was violated), and iterating on their prompting strategy, tool-use framework, or fine-tuning data to address the most common failure categories. For example, if constraint-checking failures (like the battery management error in Figure 13) account for a large fraction of Transportation failures, the team could add explicit "check constraints before acting" instructions to their agent's system prompt, or could fine-tune the agent on trajectories that demonstrate proactive constraint verification before executing actions that modify state. The paper's average of 16.2 tool calls per task provides enough interaction depth that these kinds of process-level improvements would be measurable rather than lost in noise.

Cost-benefit analysis of reasoning effort allocation. The reasoning effort ablation (Figure 7) shows that GPT-5.2 improves 27.5 points from none to xhigh (54.7% to 82.2%), while Claude Opus 4.6 improves only 3.6 points from low to max (70.2% to 73.8%). Without knowing the associated cost, this is interesting but not actionable. A practical application would instrument the evaluation to measure the actual token cost and wall-clock latency of each reasoning effort setting, producing a cost-performance curve: for GPT-5.2, what is the cost per percentage point of improvement at each step from none to xhigh? Is the jump from none to low (54.7% to 59.7%, a 5.0-point gain) cheaper or more expensive per point than the jump from high to xhigh (76.2% to 82.2%, a 6.0-point gain)? For Claude Opus 4.6, does the 0.6-point gain from medium to high (71.2% to 72.3%) justify the additional reasoning tokens, or is it within measurement noise? Organizations operating at scale could use these curves to set per-task reasoning effort dynamically: easy tasks get low reasoning effort (saving cost), while tasks that initial assessment flags as difficult get escalated to higher effort levels. The paper's difficulty calibration framework (selecting instances with the lowest autonomous success rate) already provides a difficulty signal; combining it with cost-aware reasoning effort selection would operationalize the "compute-optimal test-time scaling" concept that the prior reference paper explored for math reasoning, but applied to professional agent tasks.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Large Language Model Agent: A Survey on Methodology, Applications and Challenges2025· agents← LLM-judged similar
- PokéChamp: an Expert-level Minimax Language Agent2025· agents← LLM-judged similar
- Why Do Multi-Agent LLM Systems Fail?2025· agents← in-domain predecessor
DiningBench: A Hierarchical Multi-view Benchmark for Perception and Reasoning in the Dietary Domain2026· evaluation← in-domain predecessor
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies2026· evaluation← in-domain predecessor
ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks2025· evaluation← similar topic
- PaperBench: Evaluating AI's Ability to Replicate AI Research2025· agents← LLM-judged similar
- LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries2025· evaluation← in-domain predecessor
VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications2025· evaluation← similar topic
- ReviewScore: Misinformed Peer Review Detection with Large Language Models2025· evaluation← in-domain predecessor
- FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol2026· evaluation← similar topic
- DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle2025· evaluation← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

Do Thought Streams Matter? Evaluating Reasoning in Gemini Vision-Language Models for Video Scene Understanding2026· evaluation→ in-domain predecessor
- LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignment2026· evaluation→ in-domain predecessor
- MolmoWeb: Open Visual Web Agent and Open Data for the Open Web2026· llm-systems→ topic thread
- MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark2023· evaluation→ similar topic
- Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Models2023· evaluation→ similar topic
- VideoPhy: Evaluating Physical Commonsense for Video Generation2024· evaluation→ similar topic
Fiber: Distributed Computing for AI Made Simple2012· evaluation→ similar topic
- LAMBADA: Word Prediction Requiring a Broad Discourse Context2016· evaluation→ in-domain predecessor
- FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech2022· evaluation→ similar topic
- GenAI Arena: An Open Evaluation Platform for Generative Models2024· evaluation→ similar topic
- EcomBench: Towards Holistic Evaluation of Foundation Agents in E-commerce2025· evaluation→ similar topic
ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks2025· evaluation→ similar topic
- Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?2025· evaluation→ in-domain predecessor
- VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications2025· evaluation→ similar topic