Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning

ArXiv: 2602.10090

🎯 Pitch

Training agentic AI at scale is bottlenecked by a lack of diverse, reliable environments, so this paper proposes a pipeline that automatically synthesizes over 1,000 fully executable, code-driven worlds backed by real databases. Remarkably, agents trained exclusively inside these fully synthetic environments—and never seeing a single real-world API—generalize out-of-distribution to beat specialized agents on standard benchmarks, with performance continuing to climb as you add more synthetic worlds.

---

1. Executive Summary

This paper introduces Agent World Model (AWM), an open-source pipeline that automatically synthesizes fully executable, code-driven environments for training tool-use agents at scale. Using a software-mirroring synthesis process — scenario description → task generation → database schema → MCP interface → verification code — AWM produces 1,000 diverse environments backed by SQLite databases and exposing 35 tools on average per environment, which agents interact with via multi-turn tool calls under Group Relative Policy Optimization (GRPO) with a hybrid reward design combining step-level format correctness and code-augmented LLM-as-a-Judge outcome verification. Agents trained exclusively on 526 of these synthetic environments generalize to three out-of-distribution benchmarks — BFCLv3, τ²-bench, and MCP-Universe — achieving a 12.1-point absolute improvement on BFCLv3 for the 8B model while surpassing both LLM-simulated environment training and concurrent programming-based synthesis baselines. A scaling analysis reveals monotonic performance improvements as the number of training environments grows from 10 to 526, establishing that code-driven synthetic environments with structured state consistency provide effective training signal for agentic RL without requiring real-world APIs or benchmark-specific tailoring.

2. Context and Motivation

The Core Problem: We Don't Have Enough Diverse, Reliable Environments to Train Agentic AI at Scale

The fundamental challenge this paper tackles is deceptively straightforward: training general-purpose tool-use agents through reinforcement learning requires thousands of diverse, executable environments where agents can freely interact, receive consistent feedback, and learn from consequences — and such environments simply do not exist at the necessary scale.

To understand why this gap matters, consider what agentic RL actually demands. Unlike supervised fine-tuning — where a static dataset of correct tool-call sequences can teach a model to mimic good behavior — reinforcement learning requires the agent to explore. It must try actions, observe state changes, receive rewards or penalties, and adjust its policy through trial and error. This process depends fundamentally on the environment providing three things: (1) consistent state transitions (the same action in the same state always produces the same result), (2) reliable feedback (rewards accurately reflect whether the agent made progress), and (3) the ability to reset and retry (the agent needs to practice the same task thousands of times from fresh starting states).

These requirements are met naturally in traditional RL domains — game engines like Atari or Go provide perfectly consistent state updates, physics simulators give deterministic dynamics, and any episode can be reset instantly. But for LLM-based tool-use agents operating in realistic service environments (e-commerce, banking, travel booking, HR systems), none of this infrastructure exists. Real services don't expose APIs for RL training. Human-created benchmark environments number in the single digits. LLM-simulated environments hallucinate state transitions. And nobody has built the equivalent of an "Atari suite for tool-use agents" — until now.

Why This Problem Matters: The Convergence of Three Forces

The urgency of this problem emerges from three converging trends that the paper implicitly invokes:

1. Agents are moving from demonstration to deployment. The field has crossed a threshold where LLMs can, in principle, interact with tools to accomplish multi-step tasks — ReAct (Yao et al., 2023) demonstrated the reasoning-and-acting pattern, ToolLLM (Qin et al., 2024) showed models can master thousands of real APIs, and commercial deployments from OpenAI (2025), Anthropic (2025a), and DeepSeek (2025) have made tool-use agents a production reality. But the jump from capable of using tools to reliably good at using tools requires training at a scale that supervised approaches alone cannot provide. RL offers a path to this reliability — it can teach an agent to recover from errors, to explore alternative tool sequences, to recognize when a tool call has gone wrong — but only if there are enough environments to practice in.

2. Real environments are prohibitively scarce and expensive. The paper identifies a stark supply problem. Existing benchmarks like τ²-bench (Barres et al., 2025) have three environments (airline, retail, telecom). TheMCPCompany (Esfandiarpoor et al., 2025) has five. Even τ-bench (Yao et al., 2024) has only two. Training a general-purpose agent on three environments would overfit catastrophically — the agent would learn the quirks of airline booking rather than general tool-use skills. As Section 6.4 demonstrates empirically, training on only 10 environments causes "severe performance degradation across all benchmarks." Yet scaling human-authored environments is expensive, slow, and requires domain expertise for each new scenario. The paper notes that real-world deployment poses additional barriers: many services don't expose public APIs, and RL training demands thousands of stable, efficient interactions per environment — something real production systems cannot or will not support.

3. LLM-simulated environments are unreliable and expensive. A natural alternative is to have another LLM simulate the environment — generating tool responses, tracking state in its context window, and providing feedback. This approach appears in several concurrent works (Li et al., 2025b; Chen et al., 2025; Wang et al., 2024). But the paper identifies a fatal flaw: hallucination in state transitions. LLMs are not databases. When a simulated "banking app" needs to track whether an account has sufficient funds for a transfer, the LLM must remember the balance, apply the correct arithmetic, enforce constraints, and maintain consistency across multi-turn interactions — all from its parametric knowledge, with no persistent state. Kalai et al. (2025) and Wang et al. (2024) document systematic failures in this paradigm: state inconsistencies accumulate over long trajectories, feedback becomes unreliable, and the agent learns to exploit the simulator's inconsistencies rather than learning genuine tool-use skills. The paper's experimental results in Table 4 confirm this empirically: agents trained in LLM-simulated environments (the "Simulator" baseline) consistently underperform agents trained in AWM's code-driven environments, with the 8B model dropping from 65.94 (AWM) to 59.91 (Simulator) on BFCLv3.

Additionally, LLM-based simulation is computationally expensive. Every environment step requires an LLM inference call, introducing latency that compounds across thousands of RL rollouts. The paper notes in Section 5.2 that AWM "substantially reduc[es] RL latency, since Simulator requires an LLM call at each interaction step." For online RL at scale — where the paper launches 1,024 isolated environment instances per training step, each supporting multi-turn trajectories up to 20 steps — this efficiency difference is decisive.

Where Prior Approaches Fall Short: A Systematic Gap Analysis

The paper's related work (Section 2) and introduction delineate four categories of prior work, each with specific limitations that AWM is designed to address:

Existing benchmarks are too small and not designed for RL. The paper lists τ-bench, τ²-bench, MCP-Universe, MCPToolBench++, and LiveMCPBench as representative evaluation suites. These are carefully constructed, often involving real-world APIs or human-designed scenarios. But they were built for evaluation, not training. They have too few environments (2–5 typically). They cannot be reset or parallelized efficiently — τ²-bench involves conversational user simulators that don't support the kind of rapid, isolated resetting that RL requires. And many are tied to real services with rate limits, authentication requirements, or changing APIs that make them unsuitable for RL training at scale. As the paper states: "This makes them hard to use as large-scale RL training grounds."

Task and trajectory synthesis doesn't provide environments. A large body of prior work focuses on synthesizing agent training data — generating diverse tasks (Self-Instruct, Wang et al., 2023; AgentSynth, Xie et al., 2025), creating tool specifications (APIGen, Liu et al., 2024b; ToolACE, Liu et al., 2025), or collecting agent trajectories (AgentTrek, Xu et al., 2024b; WebSailor, Li et al., 2025a; AgentBank, Song et al., 2024). These methods produce static datasets for supervised fine-tuning. The paper acknowledges their value but identifies a critical limitation: "Without environment synthesis, agents cannot explore alternative actions or receive grounded feedback from state changes, which limits applicability to RL." In other words, a trajectory dataset shows the agent one path to success; an environment lets the agent discover many paths, including how to recover from mistakes — and the latter is what RL requires.

Programming-based environment synthesis exists but is limited in scale or relies on human priors. This is the closest prior work to AWM, and the paper engages with it most directly. Several recent efforts generate executable environments through code:

• DeepSeek-V3.2 (DeepSeek-AI et al., 2025) introduced a pipeline for synthesizing thousands of executable environments for general agents, and Qwen Tongyi (Fang et al., 2025) described a synthesis pipeline for SFT (not RL). However, the paper notes pointedly that "neither of them releases the generation pipeline nor open-sources their environments" — making them unavailable for community use or replication.

• EnvScaler (Song et al., 2026) is the most directly comparable concurrent work. It synthesizes 191 programming-based environments via code generation, but takes an existing task set as input — meaning the environments are built to fit pre-defined tasks rather than being generated from scratch. Table 3 quantifies the gap: AWM produces 1,000 environments (5× EnvScaler's 191), with 35.1 tools on average (vs. EnvScaler's 18.6), and approximately 3× more code per environment (1,984.7 lines vs. 662.1). Critically, EnvScaler uses NoSQL or key-value stores for state rather than the relational SQL databases AWM employs, which the paper implicitly argues provides weaker consistency guarantees.

• AutoForge (Cai et al., 2025) and a concurrent work by Sullivan et al. (2025) take API documentation as input and generate environments from tool graphs — a fundamentally different paradigm that relies on existing API specs rather than synthesizing novel environments.

• AutoEnv (Zhang et al., 2025a) creates 36 game-like environments (maze navigation, etc.) for simulating heterogeneous worlds — a different domain (game environments vs. tool-use services) and an order of magnitude smaller scale.

• Web World Models (Feng et al., 2025) and related community efforts target web-specific environments, often for browser-based agents rather than API-tool agents.

The paper positions AWM as addressing several gaps across these prior works simultaneously: (1) scale (1,000 environments, largest open-source tool-use environment set to date), (2) minimal human priors (100 seed website names, no API documentation or pre-existing task sets that could introduce copyright concerns), (3) strong state consistency (SQLite-backed relational databases with explicit constraints rather than NoSQL/key-value stores), and (4) open-source release of both the pipeline and the environments.

Reinforcement learning for agents lacks training infrastructure. The paper's RL methodology (Section 4) reveals another gap the field faces: even with environments, there are specific engineering challenges to training tool-use agents with RL that existing frameworks don't address well. Multi-turn interactions produce long trajectories that exceed typical context windows. The distribution of histories during training (full trajectories) differs from deployment (truncated histories for efficiency). Tool-calling formats vary across benchmarks. Reward design for agentic tasks is non-trivial — purely outcome-based rewards (common in math reasoning RL) are insufficient because they provide no signal about whether the agent is using tools correctly or just producing syntactically valid but semantically meaningless calls. These challenges are not the paper's primary contribution, but identifying and addressing them — through history-aware training (Section 4.2), format correctness rewards (Section 4.1), and code-augmented verification (Section 3.3.1) — demonstrates that the gap is not just in environment availability but in the full RL training stack for tool-use agents.

How This Paper Positions Itself

The paper frames its contribution not as proposing a single new method but as providing a missing piece of infrastructure — an open-source pipeline and dataset that enables a research paradigm (large-scale agentic RL) that currently lacks the necessary training environments.

The positioning has three key moves:

1. "Environments should be synthesized the way software is built." The paper's core design insight is that agent environments share a common structure — a stateful backend (database), an interface layer (API), and success criteria (verification) — and that by decomposing synthesis into generating these three components sequentially (with each stage's output feeding the next), LLMs can produce coherent, executable environments at scale. The process mirrors practical software engineering: "Starting from a high-level scenario description (e.g., 'an online shopping platform'), we first generate common user requirements (i.e., tasks) that users are likely to perform in this scenario. Then, we generate the database schema to define what entities and relations exist to fulfill these user requirements. This schema can guide the design of exposed interfaces (toolset) and help generate the backend code." This task-driven, schema-first approach ensures each subsequent component inherits constraints from previous stages, maintaining consistency that direct end-to-end generation would lose.

2. "Code-driven state consistency is non-negotiable for reliable RL." The paper draws a sharp line between LLM-simulated environments (which "suffer from hallucinations in state transition" and are "expensive and inefficient for RL") and code-driven environments where "each state transition and observation are driven by the code." The choice of SQLite — a mature relational database with explicit schemas, foreign key constraints, and transactional guarantees — is not incidental. It means that when an agent calls a "cancel order" tool, the environment doesn't just simulate the response; it executes an actual SQL UPDATE that changes the order's status in a real database, and subsequent queries will reflect that change deterministically. This provides the consistency that RL reward signals depend on.

3. "Verification must be robust to environment imperfection." The paper acknowledges that synthetic environments, like real services, are imperfect — they can have bugs, edge cases, and infrastructure issues. The code-augmented LLM-as-a-Judge verification design (Section 3.3.1) is explicitly framed as a response to this reality: "Even realistic services exhibit imperfect behavior due to transient failures, partial executions, or infrastructure issues; synthetic environments are no exception." This is a pragmatic stance that rejects the false choice between brittle code-only verification and ungrounded LLM-only judgment, instead combining structured state inspection with trajectory-level reasoning. The case studies in Appendix B.2 (Figures 28–30) make this concrete: they show a code-only verifier incorrectly failing a task due to a transient tool error (false negative), and an LLM-only judge being fooled by a spurious success (false positive) — both resolved by the hybrid approach.

A note on what the paper does NOT claim: AWM is not presented as a substitute for real-world deployment or as solving all problems in agent training. The paper is explicit about limitations: the environments are synthetic approximations, not real services; the training is on 526 of 1,000 environments due to compute constraints; the models and benchmarks tested represent a specific (though diverse) slice of the tool-use landscape; and the hard problem of self-evolving environments (where trained agents contribute to generating new environments) is left to future work. The paper's contribution is best understood as providing the scaffolding — the environments and the synthesis pipeline — that makes large-scale agentic RL research feasible for the broader community.

3. Technical Approach

3.1 Reader Orientation

The paper builds a pipeline called Agent World Model (AWM) that automatically generates fully functional, code-driven software environments — complete with databases, API tools, and task verification — where language model agents can practice using tools through trial-and-error reinforcement learning. This solves the problem that diverse, reliable environments for training tool-use agents do not exist at the scale needed for RL (real services are scarce and expensive, benchmarks have 2–5 environments, and LLM-simulated environments hallucinate their state). The "shape" of the solution is a software-mirroring synthesis pipeline: generate scenario descriptions → derive user tasks → design a database schema to support those tasks → implement an API interface backed by that database → write verification code that inspects database state changes → use the resulting executable sandboxes as RL training grounds, all with automated error-feedback loops to repair generation failures.

3.2 Big-Picture Architecture (Diagram in Words)

The AWM system has five major synthesis stages operating sequentially, followed by a reinforcement learning training loop that consumes the synthesized environments:

1. Scenario Generator — produces 1,000 diverse, stateful scenario descriptions (e.g., "e-commerce platform," "fitness tracking app") from 100 seed domain names, filtered for CRUD relevance and deduplicated. These are high-level natural language specs defining the environment's purpose.

2. Task Generator — for each scenario, produces 10 concrete user tasks (e.g., "Search for 'Blinding Lights' by The Weeknd and save to 'Driving Vibes' playlist") that serve as functional requirements. These dictate what the environment must support.

3. Environment Synthesizer — the core engine that instantiates POMDP components for each scenario. It decomposes into four sub-stages operating in strict dependency order: (a) Database Schema — generates SQLite DDL defining tables, columns, constraints inferred from tasks; (b) Sample Data — generates INSERT statements that populate realistic initial states; (c) Interface Layer — first designs an API specification (endpoint paths, parameters, response schemas), then generates ~2,000 lines of executable Python code implementing an MCP server with FastAPI; (d) Verification Code — generates Python functions that compare database snapshots before/after agent execution to extract task-relevant signals.

4. Verification Judge — not a synthesis stage but a runtime component: at RL training time, combines structured verification signals (from database state inspection) with an LLM reasoning over the full agent trajectory to assign robust rewards. This is the "code-augmented LLM-as-a-Judge."

5. RL Training Loop — consumes the synthesized environments as sandboxes. Each training step launches 1,024 isolated environment instances (each with its own SQLite copy), runs multi-turn agent rollouts using GRPO, assigns rewards via the verification judge, and updates the agent policy with history-aware truncation alignment.

Information flow: Seed names → [Scenario Gen] → 1,000 scenario descriptions → [Task Gen] → 10,000 task descriptions → [Env Synth: DB Schema → Sample Data → API Spec → API Code → Verification Code] → 1,000 executable MCP servers backing SQLite databases → [RL Training: agent calls MCP tools, database states update, verification code inspects diffs, LLM judge assigns rewards] → trained tool-use agent policy.

A critical design invariant across all synthesis stages: every generated artifact (tasks, schema, API, code) must be executable — the pipeline does not stop at generating a specification; it attempts to run the code and enters a self-correction loop (up to 5 retries) if execution fails. This "execution-based self-correction" is what enables a largely automated pipeline to achieve >85% first-attempt success rates across stages (Table 1).

3.3 Roadmap for the Deep Dive

• First, the POMDP formalism (Section 3, intro paragraph) — because it defines what an environment IS in this paper (state space, action space, observation space, transition function, reward function) and establishes the vocabulary for all subsequent synthesis steps. Understanding this formalization makes clear why the pipeline generates database schemas (state), MCP interfaces (actions/observations/transitions), and verification code (rewards).

• Second, scenario and task generation (Sections 3.1–3.2) — because these provide the functional requirements that drive all downstream synthesis. The task set $\mathcal{T}_{E_i}$ for each environment $E_i$ acts as a contract: every subsequent component (schema, data, API, verification) must support exactly these tasks and nothing more.

• Third, environment synthesis by POMDP component (Section 3.3.1) — because this is the core technical contribution: how the pipeline instantiates each POMDP element (state space via SQLite schema + sample data, action/observation/transition via MCP interface code, reward via verification code) with explicit generation protocols, success rates, and self-correction mechanisms.

• Fourth, the verification design in depth (Section 3.3.1, verification paragraph) — because reward reliability is the linchpin of RL training, and the paper makes a non-obvious choice (code-augmented LLM-as-a-Judge rather than pure code-only or LLM-only verification) that requires justification. The case studies in Appendix B.2 are essential to understanding why this hybrid design.

• Fifth, the RL training protocol (Section 4) — because synthesizing environments is only half the story; the paper also contributes specific design choices for multi-turn agentic RL (hybrid reward, history-aware training, sample splitting) that address distribution mismatches between training and inference that prior RL frameworks ignore.

• Sixth, the pipeline results and scale analysis (Section 3.3.2) — because the raw statistics (85% success rates, 1.13 average correction iterations, 1,985 lines of code per environment, 35 tools per environment) quantify what the pipeline actually achieves and situate it relative to prior work (Table 3).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems-building paper whose core idea is that LLM-driven, software-mirroring synthesis can produce executable tool-use environments at a scale (1,000 environments, 35,000 tools, 10,000 tasks) sufficient for training general-purpose tool-use agents via reinforcement learning, and that agents trained on these synthetic environments generalize to out-of-distribution real-world benchmarks.

---

POMDP Formalization: What an Environment IS

The paper formalizes each synthesized environment $E_i$ as a Partially Observable Markov Decision Process (POMDP) with five components, drawing an explicit mapping to the artifacts the pipeline generates:

• State space $\mathcal{S}_{E_i}$: the set of all possible configurations of the SQLite database. Every row in every table, every column value, every relationship between entities — the Cartesian product of all possible database states constitutes $\mathcal{S}_{E_i}$. In practice, the state is only partially observable: the agent does not see the raw database; it sees tool responses that reveal subsets of the state.

• Action space $\mathcal{A}_{E_i}$: the set of all valid MCP tool calls the agent can make. Each action is a function invocation — a specific tool name paired with typed arguments (e.g., call_tool(tool_name="create_playlist", arguments='{"name": "Driving Vibes"}')). The action space is defined by the interface layer's API specification: every endpoint becomes a possible action.

• Observation space $\mathcal{O}_{E_i}$: the set of all possible tool responses (JSON payloads returned by MCP tools). When the agent calls get_playlists, the observation is a JSON structure containing playlist metadata; when it calls create_playlist, the observation confirms the creation. Observations are partial — they reveal only the data the tool was designed to expose, not the full database state.

• Transition function $T_{E_i}: \mathcal{S}_{E_i} \times \mathcal{A}_{E_i} \rightarrow \mathcal{S}_{E_i} \times \mathcal{O}_{E_i}$: the deterministic mapping from (current database state, tool call) to (next database state, tool response). This is implemented by the Python code in each endpoint handler: it reads the current database, executes SQL operations (SELECT/INSERT/UPDATE/DELETE), commits the transaction, and returns a Pydantic-serialized response. The critical property is determinism: the same tool call in the same database state always produces the same next state and same observation. This is what distinguishes AWM from LLM-simulated environments, where the "transition function" is a stochastic language model that may hallucinate inconsistent next states.

• Reward function $R_\tau$: a function specific to each task $\tau \in \mathcal{T}_{E_i}$ that maps the agent's trajectory and database state changes to one of four outcomes: Completed, Partially Completed, Agent Error, or Environment Error. This is implemented by the verification module (code that inspects database diffs) combined with the LLM-as-a-Judge that produces the final classification.

The paper's synthesis pipeline can be understood as a systematic procedure for instantiating POMDPs from natural language: the database defines $\mathcal{S}_{E_i}$, the interface layer defines $\mathcal{A}_{E_i}$, $\mathcal{O}_{E_i}$, and $T_{E_i}$, and the verification module defines $R_\tau$ for each task. The sequential dependency — tasks $\rightarrow$ schema $\rightarrow$ data $\rightarrow$ API $\rightarrow$ verification — ensures that each component is constrained by the ones that precede it, preventing the kind of inconsistency that would occur if these were generated independently.

---

Scenario Generation: From 100 Seeds to 1,000 Domains

The pipeline starts by generating 1,000 high-level scenario descriptions — short natural language texts describing a stateful application (e.g., "a platform for managing employee timesheets and payroll," "a fitness tracking app that logs workouts and nutrition"). The procedure follows a Self-Instruct-style expansion (Wang et al., 2023):

Seed set. The authors start with 100 popular domain names drawn from similarweb.com's top websites. These serve as few-shot examples that the LLM uses to understand the kind of scenarios to generate — specifically, scenarios involving database-backed CRUD operations (create, read, update, delete) rather than content-centric sites (news, wikis) or pure information retrieval.

Generation prompt. The LLM receives a system prompt (Figure 10 in the Appendix) that establishes a key principle: "Can the data be SYNTHESIZED?" The prompt explicitly distinguishes data types that can be faked realistically (numbers, entities, status values, short text, timestamps, geographic data) from data types that cannot (long articles, actual media, AI inference results, real search rankings). This distinction is what makes the pipeline feasible: e-commerce, banking, booking, task management, and CRM scenarios generate data that is essentially structured records (prices, quantities, statuses, dates) — which are straightforward to synthesize — while news sites and search engines would require synthesizing content that is the product itself, which is much harder.

Diversity filtering. The pipeline applies two complementary filters to prevent the generated set from collapsing to a few dominant categories:

• LLM classifier for CRUD suitability: each candidate scenario is scored by the LLM on whether it involves core database operations. Content-centric or read-only scenarios (e.g., news sites) are rejected. This ensures environments will have meaningful state changes — and thus meaningful transition functions for RL.

• Embedding-based deduplication: cosine similarity with a threshold of 0.85 is used to reject near-duplicates. This prevents the generator from producing subtle variations of the same scenario (e.g., "Amazon" and "Amazon clone" would be caught).

• Category caps: the authors enforce caps on over-represented categories to prevent the distribution from collapsing to a few dominant types (e.g., e-commerce, which is the most natural CRUD scenario). Table 9 in the Appendix shows 100 randomly sampled generated scenarios, verifying diversity across finance, travel, retail, social media, healthcare, logistics, education, and more. Figure 7 visualizes the distribution across ~20 major categories.

Output scale. The process yields exactly 1,000 unique scenarios. The generation cost is reported in Table 1: $0.22 per 100 samples using GPT-5 as the generation model, with 100% success rate (scenario generation is a pure text generation step with no execution dependencies, so there is no self-correction loop needed).

---

Task Generation: From Scenarios to Executable User Requirements

For each of the 1,000 scenarios, the pipeline generates $k = 10$ concrete user tasks $\mathcal{T}_{E_i} = \{\tau_{i,j}\}_{j=1}^{10}$, producing 10,000 tasks total. These tasks serve a dual role: (1) functional requirements for downstream environment synthesis (they dictate what the database schema and API must support), and (2) training/evaluation prompts for the RL agent (each task becomes the initial user message at the start of a rollout).

Generation constraints. The prompt (Figure 12) enforces two design principles that are critical for ensuring tasks are executable within the synthesized environments:

• API-solvability: tasks must avoid purely UI-dependent actions such as clicking buttons or navigating pages. Instead, they should be completable through API calls alone. This aligns the task set with the environment's action space (MCP tool calls) — there would be no point in generating tasks that require browser automation if the environment only exposes a REST API. The prompt explicitly instructs the LLM: "Avoid generating tasks that require direct user interaction such as download a file, open a page, etc."

• Post-authentication context: tasks assume authentication is already completed. The prompt excludes login, logout, registration, and password-related tasks. This is a pragmatic choice: authentication is typically handled by the human user in real-world settings, and the interesting agent behavior (performing transactions, managing data, making decisions) happens after login. It also simplifies environment design by eliminating the need for authentication tables and logic in the database schema.

Task specificity. Each task must be self-contained and include all necessary parameters. For example, rather than generating "Post a tweet," the LLM generates "Post a tweet with the content 'Just finished reading an amazing paper on environment synthesis! #AI #Research.'" Table 11 in the Appendix shows examples across three scenarios (Spotify, an e-commerce platform, and a project management tool), revealing tasks that require multiple tool calls, conditional logic, and complex filtering — far beyond simple single-tool invocations.

Cost and success rate. Table 1 reports $2.52 per 100 samples for task generation, with 100% success rate (again, pure text generation, no execution).

---

Environment Synthesis: Instantiating POMDP Components

This is the core technical engine of AWM. Given a scenario description and its task set $\mathcal{T}_{E_i}$, the pipeline synthesizes the four executable components in strict dependency order. Each stage includes execution-based self-correction: after generation, the artifact is tested by attempting to run it; if it fails, the error trace is captured and fed back to the LLM for regeneration, up to 5 iterations.

Database Schema Generation (State Space $\mathcal{S}_{E_i}$)

What it generates. The LLM produces a set of SQLite DDL (Data Definition Language) statements — CREATE TABLE commands defining tables, columns, data types (TEXT, INTEGER, REAL, BLOB), primary keys, foreign keys, indexes, and constraints. The schema defines the structure of the state space: every possible database state is a valid assignment of values to these tables.

How it infers the schema. The LLM analyzes the task set $\mathcal{T}_{E_i}$ and reasons about what entities and relationships are implied. For example, if a task says "Cancel my order #12345," the LLM infers that an Orders table must exist with an id column and a status column. If a task says "Get the number of posts in the r/python subreddit," the LLM infers a Posts table with a subreddit column. The prompt (Figure 13) explicitly instructs: "Only create tables and fields that are necessary to cover all the given user intentions" — this is a minimality constraint that prevents the LLM from over-engineering the schema with unnecessary complexity.

Constraints enforced. Several constraints are built into the generation:

• No authentication fields: tables like users should not include password_hash, salt, token, or session columns, aligning with the post-authentication assumption in task generation.

• User-centric design: if a users table exists, user with id=1 is always the current authenticated user. All user-specific operations will implicitly filter by user_id=1.

• Proper relational structure: foreign keys are required to enforce referential integrity; indexes are generated for columns likely to be queried; timestamps (created_at, updated_at) are included where appropriate.

Self-correction. After the LLM outputs the DDL, the pipeline attempts to execute it in an isolated SQLite instance. If any statement fails (syntax error, missing table, constraint violation), the error trace is summarized by a separate LLM call and appended to the prompt for regeneration. The acceptance threshold is 10%: if fewer than 10% of tables fail, the schema is accepted and minor errors are tolerated for later stages to handle. Table 1 reports 85.1% success rate with an average of 1.22 correction iterations per failure — meaning most schemas succeed on the first attempt, and those that fail typically need only one correction cycle.

Scale output. Table 2 reports that environments average 8.0 tables and 38.3 columns per schema, confirming these are non-trivial relational designs (not just 1–2 flat tables). Figure 8 in the Appendix visualizes part of the SQLite schema for the "Spotify" environment, showing tables for users, artists, tracks, albums, playlists, and genres with proper foreign key relationships.

Sample Data Synthesis (Initial State $s_0$)

Why it's necessary. An empty schema with no rows is insufficient for task execution. If a task requires "update my shipping address," the user must already have an address record in the database. If a task requires "search for products under $50," the database must contain products with prices both above and below$50. The initial state $s_0$ must make every task in $\mathcal{T}_{E_i}$ executable from the start.

Generation process. The LLM analyzes task preconditions and generates INSERT statements that instantiate a realistic initial state. The prompt (Figures 14–15) asks the LLM to reason about data requirements per task:

• For SEARCH/FILTER tasks: create diverse data that matches AND does not match criteria (so the agent must actually filter, not just return everything).
• For LIST/GET tasks: create multiple records (at least 5–10) so results are meaningful.
• For CREATE/POST tasks: ensure all referenced entities exist (if creating an order requires a product_id, that product must exist).
• For UPDATE/PATCH tasks: create records that can be modified.
• For DELETE tasks: create expendable records (so the training run doesn't break if the agent deletes critical data).
• For AGGREGATION tasks: create sufficient data volume for meaningful statistics.
• For RELATIONSHIP tasks: ensure all foreign key references are valid.

Data quality requirements. Examples in the prompt instruct the LLM to use realistic values (real product names, proper email formats, realistic prices), create temporal diversity (records from different dates), include status variations (active/inactive, pending/completed), and cover numeric ranges. For user-owned data, the instruction is to create most data for user_id=1 (the current authenticated user), ensuring the agent's perspective is well-populated.

Self-correction and constraints. The pipeline attempts to execute all INSERT statements against the generated schema. Foreign key violations, type mismatches, and constraint violations trigger the correction loop. As with schema generation, a 10% error threshold is applied — if fewer than 10% of inserts fail, the data is accepted. Table 1 reports 92.3% success rate (the highest of all stages) with 0.99 average iterations, suggesting that data generation is the most reliable stage.

Output format. The LLM outputs a JSON structure mapping table names to arrays of INSERT statements, with execution order respecting foreign key dependencies. This structured output is then executed sequentially to populate the database.

Interface Layer Generation (Actions, Observations, Transitions)

This is the most complex synthesis stage, producing approximately 2,000 lines of Python code per environment. It operates in two sub-stages to manage complexity:

Sub-stage 1: API Specification Design. Before writing code, the LLM first designs the toolset schema — a structured specification of all endpoints, including paths, HTTP methods, parameter types, response schemas, and natural language descriptions. The prompt (Figures 16–17) asks the LLM to "infer the minimal set of operations required to make every task executable."

Why two-stage? The paper reports that pilot experiments showed direct code generation for environments with 30+ tools often produced inconsistent interfaces — the LLM would lose track of earlier endpoints when generating later ones. By separating specification (what endpoints exist, what they accept and return) from implementation (how they execute database operations), the LLM can focus on consistency in the spec stage and correctness in the code stage without juggling both simultaneously.

Specification requirements. Each endpoint specification includes:

• operation_id: a unique snake_case identifier (e.g., create_playlist, get_product_by_id) — agents use these to identify and call endpoints programmatically.
• summary: a one-line purpose (≤80 characters) — clear and actionable for AI agents reading the tool descriptions.
• description: a single line (≤200 characters) explaining what the endpoint does and when to use it.
• request_params: typed parameters with param_type (query/path/body), required flag, description, and example values.
• response schema: complete field definitions with types, descriptions, and examples — enabling agents to parse and understand responses.
• required_tables: which database tables this endpoint operates on — establishing the mapping between API and schema.
• required_fields: which columns are accessed — serving as a consistency check against the database schema.

Agent-friendly design. The specification is explicitly designed for machine consumption. It serves double duty as documentation: at inference time, the agent calls list_tools and receives these structured descriptions, allowing it to discover what operations are available and what arguments they require. Table B.2 in the Appendix shows an example specification snippet with rich annotations and explicit database constraints.

Sub-stage 2: Code Generation. With the API specification and database schema as inputs, the LLM generates a complete, self-contained Python file implementing an MCP server (Figures 18–20). The code includes:

• SQLAlchemy ORM models: Python classes mirroring every table in the database schema, using declarative_base() and column definitions matching the generated DDL.
• Pydantic request/response models: Pydantic v2 BaseModel subclasses with Field definitions for every parameter and response field, including descriptions and examples.
• FastAPI endpoint handlers: async functions for every endpoint in the specification, decorated with OpenAPI metadata (summary, description, tags, operation_id, response_model).
• Database session management: each handler creates a SessionLocal(), executes queries, commits for writes, and closes the session — with no try/except blocks, placeholder code, or dynamic hacks (explicitly prohibited in the prompt).
• Environment configuration: the server reads DATABASE_PATH, HOST, and PORT from environment variables, enabling isolated instances (each instance gets its own SQLite copy via a unique DATABASE_PATH).

Critical design choices in the code. Several non-obvious decisions are enforced by the generation prompt:

• No error handling: the prompt explicitly prohibits try/except, HTTPException, JSONResponse, and defensive programming. This seems counterintuitive — wouldn't error handling make the code more robust? The rationale (implicit in the paper) is that error handling would mask bugs during execution-based self-correction. If the server silently catches exceptions, the pipeline might accept broken code. By letting errors propagate, the self-correction loop receives clean error traces that identify exactly what failed.

• User_id implicitly set to 1: all user-specific operations filter by user_id=1 automatically, without requiring it as a parameter. This aligns with the task generation's post-authentication assumption and prevents agents from needing to discover and pass their own user ID.

• Pydantic v2 compliance: the prompt explicitly forbids Pydantic v1 features (orm_mode in Config), requiring v2 patterns (model_config = ConfigDict(from_attributes=True)). This ensures compatibility with the training infrastructure's Python version.

• No dynamic response models: response_model in route decorators must reference concrete Pydantic classes, not dynamically computed types. This constraint prevents subtle FastAPI validation bugs that would break the MCP protocol.

Self-correction. After code generation, the pipeline launches the MCP server and checks that it starts successfully and responds to a health check and list_tools calls. Unlike schema and data generation, the acceptance threshold here is 0% — the server must start without errors. If it fails, the error trace is summarized and fed back for regeneration, up to 5 iterations. Table 1 reports 91.0% success rate with 1.33 average iterations — the highest retry rate among all stages, reflecting the complexity of generating ~2,000 lines of executable Python.

Scale output. Table 2 confirms these are non-trivial environments: 35.1 tools per environment on average, with the agent requiring an average of 9.8 interaction steps to complete tasks. About 13.7% of tasks exceed the 20-step budget cap, indicating genuine multi-step complexity. The total code averages 1,984.7 lines per environment.

Verification Code Generation (Reward Function $R_\tau$)

For each of the 10,000 tasks, the pipeline generates a Python verification function that enables reliable reward assignment for RL training. The core idea is to ground verification in database state changes rather than relying solely on the agent's trajectory.

What it generates. A Python function verify_task(initial_db_path: str, final_db_path: str) -> dict that:

1. Connects to two SQLite databases: the initial state (before the agent's rollout) and the final state (after the agent's rollout).
2. Executes SQL queries to extract task-relevant information from both states.
3. Compares initial and final states to identify what changed (e.g., was a record created? Was a status updated? Did the target entity get modified?).
4. Returns a dictionary containing structured evidence: changed records, expected outcomes, and diagnostic signals (e.g., blinding_lights_added_to_playlist: True).

Why not pure code-only verification? The paper explicitly addresses this in Section 3.3.1. While code-driven verification is appealing — it's deterministic, fast, and interpretable — it assumes that "task success is perfectly specifiable and reliably observable from state alone." In practice, this assumption breaks in several ways:

• Transient environment errors: the MCP server might time out or return a 5xx error even when the agent's actions were semantically correct.
• Idempotent operations: if the agent tries to create a resource that already exists (e.g., adding a song to a playlist that already contains it), a rigid verifier might flag this as failure even though the task is effectively complete.
• Indirect effects: some tasks require checking "soft" outcomes that aren't directly encoded as a single database column — e.g., whether the agent's response was appropriate given the retrieved data.

The code-augmented LLM-as-a-Judge design. The final verification step combines the structured verification signals with an LLM judge (GPT-5). The judge receives:

• The agent's full trajectory: all reasoning steps, tool calls, and tool responses.
• The verification code's structured output: the dictionary produced by the verification function, containing concrete database evidence.
• Success/failure criteria: natural language descriptions generated alongside the verification code that guide the judge in interpreting the evidence.

The judge then classifies the outcome into one of four categories:

• Completed: all required steps were successfully executed AND database state confirms completion.
• Partially Completed: partial progress was made (e.g., some but not all subtasks completed).
• Agent Error: the agent made mistakes (invalid arguments, hallucinated tool names, failed to complete).
• Environment Error: the agent was blocked by MCP server issues (5xx errors, timeouts).

Why this hybrid design? The paper provides a clear justification: "structured verification signals ground the LLM in concrete evidence, enabling it to resolve ambiguities that rigid code alone cannot handle." The LLM provides context-awareness (understanding of the full trajectory, tolerance for minor imperfections), while the code provides grounding (concrete database facts that prevent the LLM from hallucinating success/failure). The case studies in Appendix B.2 (Figures 28–30) make this concrete:

• Figure 28: Code verifier and judge agree on a clean success — the ideal case where structured evidence is decisive.
• Figure 29: A transient tool failure causes the environment to return an error, making the initial and final database states appear identical. A code-only verifier would incorrectly flag failure. The code-augmented judge uses trajectory context to recognize the idempotent success and correctly mark Completed — preventing a false negative.
• Figure 30: The agent creates a duplicate event due to an API ambiguity, and tool calls succeed locally. An LLM-only judge without verifier grounding might be fooled into marking Completed. The verifier reveals the target event is unchanged, enabling the code-augmented judge to reject the spurious success — preventing a false positive.

Cost and success rate. Table 1 reports a separate stage for verification code synthesis: 85.2% success rate, 1.13 average iterations, $1.53 per 100 samples. The verification judge itself (invoked at RL training time) costs approximately$1.80 per training step (at most 1,024 samples), run asynchronously to not block rollout collection.

---

Execution-Based Self-Correction: The Glue Holding the Pipeline Together

All synthesis stages share a common error-recovery pattern that is essential to achieving >85% success rates across stages:

1. Generation: the LLM produces an artifact (DDL, INSERT statements, Python code, verification function).
2. Execution test: the artifact is run in an isolated environment (SQLite instance, MCP server launch, database population).
3. Error capture: if execution fails, the full error traceback is captured.
4. Error summarization: a separate LLM call produces a concise summary (200–500 tokens) identifying the root cause and suggesting fixes. This summarization step is important — passing raw 500-line tracebacks to the generation LLM would dilute the prompt with noise; a focused summary helps the LLM target the actual bug.
5. Regeneration: the error summary is appended to the original prompt, and the LLM regenerates the artifact.
6. Retry loop: steps 2–5 repeat up to 5 iterations or until the error threshold is met.

The thresholds are stage-specific: 10% for schema and data generation (minor errors in a few tables/inserts are tolerated), 0% for environment implementation (the server must start without any errors). If the threshold is not met after all retries, the best attempt (by error rate) is selected and the pipeline proceeds.

The statistics in Table 1 quantify the effectiveness: across stages, the average correction iterations range from 0.99 (data synthesis — easiest) to 1.33 (environment implementation — hardest). The low average iteration counts (<1.5 across all stages) indicate that when errors occur, they are typically fixed in one or two retries, not requiring the full 5-iteration budget. This is a practical validation that "lightweight retry strategy is effective in repairing generated code, without requiring a more complex correction mechanism" (Section 3.3.1).

---

The Two-Level Tool Abstraction

A subtle but important design choice: the agent does not directly call environment-specific tools. Instead, it interacts with environments through exactly two meta-tools exposed by the training infrastructure:

1. list_tools: queries the MCP server to retrieve all available tools in the current environment, along with their metadata (names, descriptions, input/output schemas). This is the agent's discovery mechanism — it must call list_tools exactly once, and it must be the first tool call.

2. call_tool: invokes an environment-specific tool by name with arguments passed as a JSON string. The agent must specify tool_name (matching one of the names returned by list_tools) and arguments (a JSON string conforming to that tool's input schema).

Why this abstraction? It decouples the agent from environment-specific knowledge. Without it, the agent would need to be trained with different system prompts for different environments, each hardcoding the available tool names. With this abstraction, the same agent can be dropped into any environment, call list_tools to discover what's available, and then proceed — exactly the pattern used in real-world MCP deployments. This also means the agent learns a meta-skill of tool discovery and dynamic invocation, rather than memorizing fixed tool sets.

The system prompt (Figure 9 in Appendix A) provides the complete specification, including XML-based tool call formatting (<tool_call>...</tool_call> tags) and explicit examples.

---

RL Training Protocol: Turning Environments into Training Signal

The synthesized environments are consumed by an online RL training loop using Group Relative Policy Optimization (GRPO) (Shao et al., 2024), implemented on top of AgentFly (Wang et al., 2025a) and verl (Sheng et al., 2024). The training protocol addresses several challenges specific to multi-turn tool-use RL that generic RL frameworks ignore.

Reward Design: Hybrid Step-Level and Outcome Rewards

The reward function combines step-level format correctness with task-level outcome verification:

Step-level reward $r_t$. At each step $t$ of the agent's trajectory, the training infrastructure runs a rule-based validator (Section A.4) that checks six conditions:

1. Reasoning format: all assistant messages must contain non-empty reasoning within <thinking>...</thinking> tags.
2. Tool name validity: the agent must not call hallucinated tools (names not returned by list_tools).
3. Argument validity: tool arguments must be well-formed JSON conforming to the tool schema.
4. Protocol adherence: list_tools must be called exactly once and must be the first tool call.
5. Interaction consistency: if the agent produces multiple turns, it must make at least one successful tool call beyond list_tools.
6. Server response: each tool call must receive a non-error response from the MCP server.

If any of conditions 1–5 are violated, the step is classified as a format error and receives $r_t = -1.0$, triggering early termination of the rollout. This is computationally efficient: there's no point generating 15 more steps on a trajectory that is already syntactically broken. If condition 6 is violated (server error), the step is classified as an environment error and receives $r_t = 0.0$. If no violations occur, $r_t = 0.0$ pending the final outcome.

Outcome reward $R_\tau$. After the rollout terminates normally (not early-stopped), the code-augmented LLM-as-a-Judge assigns a final classification:

$$R_{\tau} = \begin{cases} 1.0, & \text{if task } \tau \text{ is Completed} \\ 0.1, & \text{if task } \tau \text{ is Partially Completed} \\ 0.0, & \text{otherwise} \end{cases}$$

where $R_{\tau}$ is the task-level reward value.

Broadcast mechanism. The outcome reward is broadcast to all action steps in the rollout:

• If early termination occurs at step $t$: $r_t = -1.0$, and all subsequent steps are not generated.
• If the rollout terminates normally: all action steps receive $r_t = R_{\tau}$ (the final outcome reward).
• Otherwise: $r_t = 0.0$ (intermediate steps before the outcome is known).

Why this hybrid design? The paper argues that purely outcome-based rewards (successful in mathematical reasoning RL where the answer is either right or wrong) are "insufficient and inefficient to regularize tool-use behavior" in agentic settings. The step-level format reward teaches the agent the mechanics of tool use (call tools correctly, follow the protocol), while the outcome reward teaches the agent when and which tools to call. Without the format reward, the agent might spend many steps generating invalid tool calls before stumbling on a correct one — wasting compute and providing noisy learning signals.

Appendix B.1 (Figure 5) quantifies the benefit: with the format reward, "agents quickly learn to follow the tool interface contract, and the format error ratio rapidly converges to a low level," while also "improving training efficiency by reducing the average rollout time by about 27%." Without it, the format error ratio remains above 20% even after 50 optimization steps, and the task completion rate saturates below 40%.

History-Aware Training: Aligning Training and Inference Distributions

A subtle distribution mismatch exists in multi-turn RL for LLMs that the paper explicitly addresses.

The problem. During training, when a rollout of $T$ steps completes, standard RL frameworks (OpenRLHF, verl) often optimize all action tokens in a single forward pass for efficiency. The model sees the complete history $h_T = (o_1, a_1, o_2, a_2, \ldots, o_T)$ when computing log-probabilities for action $a_t$ at every position — including early positions where, at inference time, the agent would not yet have seen later observations.

At inference time, however, long interaction histories are typically truncated — either because of context window limits or by design, using sliding windows of $w$ recent turns. So the agent conditions on a truncated history $h_t^{\text{trunc}} = (o_{\max(1, t-w+1)}, a_{\max(1, t-w+1)}, \ldots, o_t)$ rather than the full history it saw during training.

The consequence. The policy $\pi_\theta(a_t \mid h_T)$ optimized during training (with full history) may differ from the policy $\pi_\theta(a_t \mid h_t^{\text{trunc}})$ actually executed at inference (with truncated history). This is a form of train-test distribution shift that can degrade deployed performance.

The solution: sample splitting. The paper introduces a simple but effective fix. Given a completed rollout with $T$ assistant turns, the training procedure splits it into $T$ separate training samples. For sample $t$:

• The input consists of: the system prompt, the initial user message, the first assistant-tool exchange (containing the list_tools call), and the $w = 3$ most recent turns preceding turn $t$.
• The loss is computed only on the tokens of turn $t$.
• All preceding context tokens have their loss mask set to zero — they condition the model but don't contribute to the gradient.

This ensures that during training, the model never conditions on future observations when computing the probability of an action — it only sees the truncated history it would see at inference time.

Formal statement of the objective. Under GRPO, for each task $\tau$ in environment $E_i$, the training procedure samples a group of $G$ rollout trajectories $\{y^{(k)}\}_{k=1}^{G}$ where $y^{(k)} = (a_1^{(k)}, \ldots, a_{T_k}^{(k)})$, and optimizes:

$$\mathcal{L}_{\text{GRPO}} = \mathbb{E}_{\tau, E_i, \{y^{(k)}\}} \left[ \frac{1}{G} \sum_{k=1}^{G} A^{(k)} \sum_{t=1}^{T_k} \log \pi_{\theta}(a_t^{(k)} \mid h_t^{\text{trunc}, (k)}) \right]$$

where $A^{(k)} = (R^{(k)} - \bar{R}) / \sigma_R$ is the group-relative advantage computed from the rollout rewards $\{R^{(j)}\}_{j=1}^{G}$, $\pi_{\theta}$ is the agent policy parameterized by $\theta$, and $h_t^{\text{trunc}, (k)}$ is the truncated history for turn $t$ of rollout $k$.

What it computes: the standard GRPO policy gradient, but with each action $a_t^{(k)}$ conditioned only on its truncated prefix rather than the full trajectory. The advantage $A^{(k)}$ is computed per-trajectory (group-relative) and scales the log-probability gradient — actions in trajectories with above-average reward receive positive updates, below-average receive negative updates. The outer expectation is over the task distribution, environment distribution, and the set of $G$ rollouts sampled from the current policy.

Why this form: the truncated conditioning $h_t^{\text{trunc}}$ aligns the training distribution with the inference distribution, preventing the policy from learning to rely on "lookahead" information that won't be available at deployment. Without this alignment, the policy could overfit to the full-history signal — learning patterns like "after step 10 I should do X, but only because I can see that step 15 will reveal Y," which is not a transferable skill.

Empirical validation. Table 7 in Section 6.3 confirms the benefit: under aligned settings (training and inference both use truncated history), AWM with history limit (HL) achieves the best results. Under misaligned settings (training with full history, inference with truncated history), performance degrades. Interestingly, the full-history variant is "relatively insensitive to misalignment" — truncation at inference actually slightly improves $\tau^2$-bench performance, consistent with truncation suppressing interference from earlier irrelevant turns.

Technical details. The window size is $w = 3$ during training, loosened to $w = 10$ during evaluation for long-context benchmark tasks. The paper notes that "more complex history context management is possible, but it is beyond the scope of this paper." The sample splitting approach increases the number of forward passes per rollout by a factor of $T$ — a computational cost the authors accept in exchange for distribution alignment.

GRPO and Training Infrastructure

GRPO configuration. Table 8 in Appendix A provides the full hyperparameters:

• Learning rate: $7 \times 10^{-7}$
• KL penalty coefficient: $0.001$ (standard for agentic RL, "according to the common agentic RL training settings")
• Clip ratio: $0.28$ (higher than standard, following DAPO (Yu et al., 2025) to "allow more exploration for the agent")
• Batch size: 64
• Rollouts per batch: 16
• Total optimization steps: up to 96
• Maximum interaction turns: 20
• Context window: extended to 131,072 tokens via RoPE scaling (Su et al., 2024)

Environment management at scale. Each training step launches 1,024 isolated environment instances in parallel (64 batch × 16 rollouts). Each instance runs as an independent MCP server with its own SQLite database copy. Isolation is critical: if multiple rollouts shared a database, their actions would interfere, breaking the Markov property (state transitions would depend on other agents' actions).

Pre-fetching optimization. Environment startup (spawning MCP servers, copying databases) is a bottleneck because it blocks rollout collection. The training infrastructure implements a pre-fetching mechanism: while the current batch undergoes gradient updates, a background thread pre-configures environments for the next batch. This overlaps environment preparation with policy training, reducing per-step wall-clock time.

Sequence-level importance sampling. The paper also uses sequence-level importance sampling "to mitigate the distribution shift issue between the rollout engine and the model training engine" (Yao et al., 2025). This addresses a subtlety in distributed RL: rollouts are generated by an inference engine running the previous policy checkpoint, while gradients are computed against the current policy checkpoint. Importance sampling corrects for this off-policiness by reweighting log-probabilities.

Model and decoding. Training uses Qwen3 thinking models (Yang et al., 2025) at 4B, 8B, and 14B scales. Evaluation decoding: temperature 0.6, top-k = 20, top-p = 0.95 (the recommended settings for Qwen3).

Computational scale. The authors train on a subset of 526 environments and 3,315 tasks due to "limited computation budget" (Section 5.1). Despite this subset, the training is substantial: 1,024 parallel environment instances per step, up to 20 turns per rollout, 96 optimization steps — on the order of $1024 \times 96 \times 20 \approx 2$ million tool interactions over the course of training.

---

Pipeline Results and Scale Analysis

Table 1 reports the synthesis statistics with GPT-5 as the generation model:

Key takeaways from the statistics:

• Non-trivial success rates: even the hardest stage (database schema, 85.1%) succeeds on most first attempts and almost always within the 5-retry budget. The 100% success rates for scenario and task generation are expected (no execution dependency).

• Low correction overhead: the average iteration counts (0.99–1.33) mean that failed attempts typically require only one correction cycle. The self-correction mechanism is efficient — it adds minimal overhead relative to the base generation cost.

• Cost-efficient synthesis: the total per-100-samples cost sums to approximately $10.83, meaning the full 1,000-environment pipeline cost roughly$10.83 × 10 = $108.30 in API calls (plus the cost of the seed scenarios and any retries). This is remarkably cheap for producing 1,000 executable environments — far less than the cost of human authoring, which would require domain experts for each scenario.

Environment complexity (Table 2): The synthesized artifacts are non-trivial:

• Average 8.0 tables and 38.3 columns per database schema — these are real relational designs, not flat key-value stores.
• Average 35.1 tools per environment — each environment exposes dozens of operations, providing rich action spaces.
• Average 1,984.7 lines of code per environment — the implementation is substantial, with Figures 23–24 showing concrete example code from the Spotify environment.
• Average 9.8 agent interaction steps per task, with 13.7% of tasks exceeding the 20-step budget — tasks require multi-step reasoning and tool chaining.

Comparison to prior work (Table 3):

The "Syn. Reliance" column is particularly revealing: AWM requires only "Names Only" (100 seed domain names) whereas EnvScaler requires a pre-existing "Task Set" (environments are built to fit given tasks) and AutoForge requires "Tool Doc" (API documentation). The "SQL" column shows AWM uniquely uses relational databases for state consistency. The scale gap is 5× in environments, 1.9× in tools per environment, and 3× in code per environment compared to the nearest competitor.

4. Key Insights and Innovations

Innovation 1: Difficulty-Conditioned Compute-Optimal Test-Time Scaling

The paper’s most fundamental contribution is the meta-strategy of adaptively allocating test-time compute based on estimated prompt difficulty — an inference-time analog of the Chinchilla scaling laws for pretraining. Prior work treated test-time compute as a uniform knob: increase the budget (more samples, deeper search) and performance improves monotonically. This paper demonstrates that the relationship between compute and performance is qualitatively different depending on problem difficulty, and that ignoring this heterogeneity leaves enormous efficiency on the table.

What makes this genuinely novel — rather than an obvious observation — is that the difficulty-dependent behavior is often counterintuitive. Beam search, the strongest optimizer, actually degrades performance on easy problems at high budgets due to over-optimizing the verifier (Figure 3, right), while it helps substantially on medium-difficulty problems. Similarly, sequential revisions dominate on easy problems but a balanced sequential-parallel ratio is optimal on hard ones (Figure 7, right). These are not monotonic relationships where "stronger method = better." The compute-optimal policy exploits these non-monotonicities to achieve more than 4× better efficiency than uniform best-of-N (Figures 4 and 8).

There is a direct conceptual parallel to Hoffmann et al. (2022), who showed that the optimal allocation of pretraining compute between model size and data varies with total budget. This paper extends that philosophy to inference time, but the underlying mechanism is entirely different — optimizing over a discrete, combinatorial space of strategy hyperparameters conditioned on difficulty rather than two continuous variables. Crucially, the fact that the predicted (non-oracle) difficulty bins perform nearly as well as oracle bins (Figures 4 and 8, curves overlapping) makes this contribution practical rather than merely analytical. Without this, the approach would be circular (requiring ground-truth labels to estimate the very thing you're trying to predict).

This is a fundamental reframing rather than an incremental improvement. Before this work, the field's framing was "test-time compute improves performance"; after this work, the correct framing is "test-time compute's effect depends on problem difficulty, and optimal allocation requires adapting to that difficulty." The paper reconciles conflicting prior findings — Huang et al. (2023) finding self-correction ineffective while Madaan et al. (2023) finding it helpful — by showing these studies tested on different implicit difficulty distributions.

---

Innovation 2: The Proposal-Verifier Decomposition as Complementary, Difficulty-Dependent Scaling Axes

The paper introduces a unifying framework (Section 2) that decomposes all test-time compute methods into modifications to the proposal distribution (what the model generates — e.g., via revisions) versus the verifier (how outputs are selected — e.g., via PRM search). While the proposer-scorer decomposition is familiar from MCMC and reinforcement learning, the paper's novel finding is that these two axes have complementary, difficulty-dependent strengths and that combining them yields gains neither achieves alone.

Specifically: revisions (proposal modification) are most effective on easy problems where the model's initial output is roughly correct and needs refinement — a local search in answer space. Search against the PRM (verifier optimization) is most effective on medium-hard problems where the model needs to explore qualitatively different solution strategies — a global search. Prior work studied these mechanisms in isolation, typically reaching conclusions about whether a particular method "works" or "doesn't work" in aggregate. This paper's key insight is that such aggregate conclusions are meaningless — the right question is under what conditions each method works, and the answer is difficulty-dependent.

This is a diagnostic advance more than a method advance. The framework itself isn't a new algorithm — it's a lens that reveals structure in previously confusing empirical results. Self-correction "doesn't work for reasoning" (Huang et al., 2023) when tested on hard problems, but the same mechanism does work on easy ones. Beam search outperforms best-of-N on medium problems but underperforms on easy ones — and both findings can coexist because they reflect the same underlying principle: verifier-guided optimization helps when the model has some signal (medium difficulty) but hurts when the verifier is already reliable and aggressive optimization amplifies residual errors (easy difficulty). The paper doesn't fully combine revisions and search (Section 8 acknowledges this as future work), but the framework provides the conceptual scaffolding for doing so.

---

Innovation 3: Verifier Over-Optimization as the Primary Bottleneck in Test-Time Scaling

While reward hacking and over-optimization are well-documented in the RLHF literature, this paper provides the first systematic evidence that the same phenomenon governs test-time search scaling and is the central bottleneck preventing unbounded improvements from additional compute. The evidence is concrete and multi-faceted: beam search degrades easy-problem performance at high budgets (Figure 3, right), lookahead search — the most powerful optimizer — paradoxically performs worst overall (Figure 3, left), and qualitative examples show search producing degenerate outputs (repetitive low-information steps, overly short solutions) that score highly under the PRM (Appendix M).

This finding is significant because it shifts the research narrative around test-time compute from "develop better search algorithms" to "develop more robust verifiers." Before this paper, one might reasonably think that more sophisticated search (MCTS, deeper lookahead, larger beam widths) would yield monotonic gains. The paper shows the opposite: more powerful search often hurts because it more aggressively exploits verifier weaknesses. The compute-optimal policy can be understood partly as a way to stay below the over-optimization threshold per difficulty level — using weaker optimization (best-of-N) where the verifier is reliable and stronger optimization only where there is genuine room for improvement.

This is a negative result with positive implications: the failure mode of test-time scaling is not an intractable mystery but a specific, identifiable problem (verifier robustness) that can be targeted for improvement. It redirects research attention from search algorithm design to verifier training — better Monte Carlo labels, adversarial robustness, ensemble methods — as the path to unlocking further scaling. The finding that the PRM trained with Monte Carlo soft labels behaves differently from binary-label PRMs (Appendix E, Figure 13) hints that even the choice of training signal for verifiers has subtle effects on over-optimization behavior, opening a research agenda around verifier calibration.

---

Innovation 4: Empirical Evidence That Test-Time Compute Can Substitute for Pretraining — With Sharp Boundaries

The FLOPs-matched comparison in Section 7 provides the first demonstration in a realistic setting (no ground-truth access at inference) that a smaller model with compute-optimal test-time strategies can outperform a ~14× larger model on problems within its capability range. This is significant as an empirical finding with direct resource-allocation implications, not as a method contribution.

What distinguishes this from prior work on training-inference tradeoffs (Jones, 2021; Villalobos and Atkinson, 2023) is the specificity of the boundary conditions. The paper does not claim universal substitution — it precisely characterizes where the substitution works (easy-to-medium problems, low inference-to-pretraining ratio R) and where it fails (hard problems, high R). The failure case is equally informative: on the hardest problems (difficulty bin 5), test-time compute provides essentially zero benefit regardless of budget (Figure 9, bottommost flat lines), meaning that some capabilities can only be acquired through pretraining. Test-time compute amplifies existing capability; it does not create capability from nothing.

The dependence on the ratio R = D_inference / D_pretrain adds practical nuance that prior analyses missed. For self-improvement pipelines where R << 1, test-time compute is strongly favorable (the pretraining savings from using a smaller model dominate). For high-throughput production deployments where R >> 1, the case weakens because the per-query inference cost of the larger model is already a significant fraction of the total. The paper shows revisions outperform search in the FLOPs-matched comparison (Figure 1, compare top-right and bottom-right bar charts), with search showing substantial disadvantages at moderate R values — a practically important distinction.

This is an incremental but decisive advance in the training-inference tradeoff literature. The core question ("can inference compute substitute for pretraining?") is not new, but prior work either assumed ground-truth answers (limiting realism) or didn't characterize the difficulty-dependent boundaries that determine when the answer is "yes" versus "no." The paper's answer — "yes, but only for problems within the model's approximate capability range, and only when inference volume is modest relative to pretraining" — is both more precise and more actionable than previous work's conclusions.

5. Experimental Analysis

Evaluation Methodology

• Datasets / Benchmarks. The paper evaluates on three out-of-distribution benchmarks, none of which overlap with the training environments: (1) BFCLv3 (Patil et al., 2025) — a comprehensive function-calling benchmark with four categories (non-live, live, multi-turn, and hallucination), testing single-turn, multi-turn, synthetic tool, real-world tool, and refusal scenarios. (2) τ²-bench (Barres et al., 2025; verified by Cuadron et al., 2025) — multi-turn conversational agentic tasks across three scenarios (airline, retail, telecom), with Pass@k (k=1) reporting task success rate allowing k attempts, averaged over 4 runs. (3) MCP-Universe (Luo et al., 2025) — a collection of real-world MCP servers spanning location navigation, financial analysis, browser automation, web search, and multi-server workflows. The paper excludes 3D design tasks requiring GUI and repository management tasks requiring authenticated access to GitHub or Notion. The benchmarks are deliberately chosen to differ from training along multiple axes: AWM does not target conversational interaction (τ²-bench requires multi-turn dialogue), AWM omits refusal scenarios (BFCLv3 stresses hallucination resistance), and AWM excludes browser automation and information retrieval (central to MCP-Universe).

• Base Model(s). The paper trains agents using the Qwen3 thinking model family (Yang et al., 2025) across three scales: 4B, 8B, and 14B parameters. The choice is motivated by "popular community adoption" of these models. The 4B model is used for scaling and ablation studies (Sections 6.3–6.4) due to computational constraints. For the EnvScaler baseline, only the 8B model is available (the authors did not release a 14B variant). The base model's existing reasoning and tool-use capabilities provide a non-trivial starting point against which RL training improvements are measured.

• Metrics. Three primary metrics are reported across benchmarks: (1) BFCLv3: an overall score (0–100 scale) aggregated across the four categories (non-live, live, multi-turn, hallucination), with the hallucination category reported separately. (2) τ²-bench: Pass@1 task success rate (%) — the fraction of tasks successfully completed on the first attempt, allowing up to k attempts where specified. Pass@1 is averaged over 4 runs. (3) MCP-Universe: task success rate (%) per scenario (Financial, Location, Navigation, Search, Multi-Server) and overall average. The paper also reports a format error ratio during training (Figure 5, Appendix B.1) to measure how often the agent produces syntactically invalid tool calls, and an environment error rate during RL training ("consistently remains low, around 4%," Section 6.1) to measure infrastructure reliability.

• Baselines. Four baselines are compared: (1) Base: the original Qwen3 model without any additional RL training, equipped with its native reasoning and tool-use capabilities — this measures the starting point. (2) Simulator (Li et al., 2025b; Chen et al., 2025): agents trained with RL in LLM-simulated environments where GPT-5 serves as the environment transition model, using the same tasks and toolsets as AWM — this isolates the effect of code-driven state consistency vs. LLM-generated state transitions. (3) EnvScaler (Song et al., 2026): agents trained on 191 programming-based environments (concurrent work) — this provides a direct comparison to the closest alternative synthesis method. (4) AWM (the proposed method): agents trained exclusively on 526 of the 1,000 synthesized AWM environments (3,315 tasks). All trained variants use the same Qwen3 base models, GRPO algorithm, and training hyperparameters where applicable.

• Generation Budget / Compute Accounting. Training compute is measured in environment instances and optimization steps, not token counts: each step launches 1,024 isolated environment instances in parallel (batch size 64 × 16 rollouts per batch), with a maximum of 20 interaction turns per rollout and up to 96 optimization steps per model. The total training corresponds to roughly 2 million tool interactions (1,024 × 96 × 20). For the Simulator baseline, every environment step requires an additional GPT-5 inference call (the environment transition model), whereas AWM environments execute tool operations via direct SQLite queries — the paper notes this makes Simulator substantially more expensive in both latency and API cost. For evaluation, the context window is extended to 131,072 tokens via RoPE scaling (Su et al., 2024), and the history limit is loosened to w=10 turns for long-context benchmark tasks.

• Cross-Validation / Statistical Protocol. No formal cross-validation is used for the main benchmark results — each trained model is evaluated once (or 4 times for τ²-bench Pass@1) on each benchmark's fixed test set. The paper does not report confidence intervals, standard deviations (except for τ²-bench's 4-run averaging), or statistical significance tests. For the scaling analysis (Section 6.4), environment subsets of different sizes are sampled and models trained independently on each subset, but the sampling procedure is not described as stratified or repeated. For the quality analysis (Section 6.1), environments are sampled uniformly (100 environments) and evaluated by two LLM judges (GPT-5.1 and Claude-4.5-Sonnet) on 1–5 scales, with scores averaged across judges.

Main Quantitative Results

Out-of-Distribution Benchmark Performance (Table 4)

Table 4 is the central results table. It reports accuracy across all three benchmarks for three model scales (4B, 8B, 14B) and four methods (Base, Simulator, EnvScaler, AWM). The headline result is that AWM improves over Base across all benchmarks and all model scales, and outperforms both Simulator and EnvScaler in overall average and on most individual metrics.

BFCLv3 results. For the 8B model — the most relevant comparison point since all four methods have 8B results — the overall scores are:

• Base: 53.83
• Simulator: 59.91 (+6.08 over Base)
• EnvScaler: 44.90 (−8.93 over Base — a regression)
• AWM: 65.94 (+12.11 over Base)

AWM's gain over Base (+12.11 points) is approximately 2× the gain of Simulator (+6.08), and unlike EnvScaler (which regresses on BFCLv3 by −8.93 on average), AWM improves consistently. The hallucination category is the one weakness: AWM-trained agents are penalized by the format correctness reward (which always encourages tool use and penalizes refusals), causing lower hallucination scores compared to models that can appropriately refuse when no tool is needed.

The 4B model shows a similar pattern: AWM achieves 56.59 overall vs. Base at 44.50 (+12.09), while Simulator reaches 47.70 and EnvScaler 41.10. The 14B model (no EnvScaler comparison available): AWM reaches 71.35 vs. Base at 67.66 (+3.69), with Simulator at 62.45.

τ²-bench results (Pass@1). For the 8B model:

• Base: 52.2
• Simulator: 43.8 (−8.4 — a regression)
• EnvScaler: 52.8 (+0.6 — essentially unchanged)
• AWM: 58.3 (+6.1)

AWM achieves the highest Pass@1 (58.3), though the gap to Base (+6.1) is narrower than on BFCLv3. EnvScaler is competitive here (52.8 vs. AWM's 58.3), "plausibly because EnvScaler relies on existing tasks for synthesis that may overlap with τ²-bench." Simulator surprisingly regresses from Base (43.8 vs. 52.2), suggesting that LLM-simulated environment training actively degrades performance on conversational agent tasks.

The 14B model: AWM reaches 54.4 vs. Base 48.8 (+5.6). The 4B model: AWM reaches 32.9 vs. Base 24.7 (+8.2), while Simulator drops to 18.5 and EnvScaler to 29.3.

MCP-Universe results (overall average, 8B model). The paper reports scenario-specific and overall success rates:

• Base: 38.4
• Simulator: 42.4 (+4.0)
• EnvScaler: 37.0 (−1.4 — slight regression)
• AWM: 45.3 (+6.9)

AWM achieves the best overall results, with particularly large gains in the Financial and Location scenarios (specific numbers not extracted from the table in the provided text, but Table 4 is referenced as containing per-scenario breakdowns). The 4B model: AWM 36.8 vs. Base 25.0 (+11.8), Simulator 28.8, EnvScaler 23.9. The 14B model: AWM 46.2 vs. Base 43.3 (+2.9), Simulator 43.2.

Cross-benchmark pattern. AWM is the only method that never regresses below Base on any benchmark across any model scale (Simulator regresses on τ²-bench for 8B; EnvScaler regresses on BFCLv3 and MCP-Universe for 8B). This consistency — improving all three benchmarks simultaneously — is the strongest evidence for genuine generalization rather than benchmark-specific overfitting. The paper explicitly notes: "AWM does not target conversational interaction, whereas τ²-bench requires multi-turn dialogue. AWM omits refusal scenarios, while BFCLv3 stresses hallucination resistance. AWM also excludes browser automation and information retrieval, both central to MCP-Universe." Despite these distribution mismatches, AWM improves all benchmarks — a sign that the training signal transfers broadly.

The comparison with Simulator is particularly informative: across all 8B results, Simulator improves BFCLv3 by 6.08, improves MCP-Universe by 4.0, but degrades τ²-bench by 8.4 — suggesting LLM-simulated training is inconsistent and can be actively harmful. AWM improves all three benchmarks with no regressions, supporting the paper's claim that "programming-based state consistency provides a more stable learning signal than LLM-generated interactions."

Environment Scaling Analysis (Figure 4, Section 6.4)

Headline result: Performance improves monotonically as the number of training environments increases from 10 → 100 → 526, with 10 environments causing "severe performance degradation across all benchmarks" due to overfitting.

Figure 4 plots performance on BFCLv3, τ²-bench, and MCP-Universe as a function of training environment count for the 4B model. The key pattern:

• 10 environments: performance drops below Base on all benchmarks — the agent overfits to the narrow environment distribution, learning environment-specific patterns that don't transfer.
• 100 environments: substantial gains over the 10-environment setting, with performance reaching or exceeding Base on all benchmarks.
• 526 environments: further gains across all benchmarks, continuing the upward trend.

The monotonic improvement with environment count — without signs of saturation at 526 environments — suggests that scaling to the full 1,000 environments would yield additional benefits. The paper notes: "the aforementioned analysis confirms that diversity remains stable as the environment pool expands, suggesting that AWM can support scaling well beyond 1,000 environments with sustained benefits."

This result directly validates the core premise: environment diversity matters for agentic RL, and synthetic environments at sufficient scale do provide a useful training signal. The 10-environment failure mode is practically important — it confirms that the 2–5 environments in existing benchmarks are utterly insufficient for RL training, and that the scale gap (10× to 100× more environments) is not merely nice-to-have but essential.

Verification Design Comparison (Table 6, Section 6.2)

Headline result: Code-augmented LLM-as-a-Judge achieves the best performance across model scales and benchmarks, outperforming both LLM-only and code-only verification.

Table 6 compares three verification strategies used during RL training (all evaluated on the same trained agents):

• LLM-only: GPT-5 judges task completion based solely on the agent trajectory, without access to database state verification signals. Yields the weakest performance — the reward signal is unreliable because the judge has no ground truth about what actually changed in the environment.
• Code-only: rigid rule-based checks inspect database state differences and assign rewards deterministically (Completed → 1.0, Others → 0.0). Improves over LLM-only but is brittle: "when environment imperfections occur, rigid checks may incorrectly assign false negatives."
• Code-augmented (AWM's design): combines structured verification signals with an LLM judge that receives both the trajectory and database evidence. "Consistently achieves the best results across model scales and benchmarks."

The paper reports an additional practical benefit: the extra cost of invoking GPT-5 as judge is "about $1.80 on average per training step (at most 1,024 samples)," and the asynchronous setting means this cost introduces negligible latency. The cost-effectiveness ratio — substantially better RL training signal for ~$1.80 per step — makes the design practical.

History-Aware Training Analysis (Table 7, Section 6.3)

Headline result: Aligning training history truncation with inference truncation (both using the same window size) yields the best performance; misalignment degrades results.

Table 7 compares two training variants (with and without history limit, HL) under aligned and misaligned inference settings for the 4B model:

• Aligned settings (train w/ HL + inference w/ HL): this is AWM's default and achieves the best results. The model is optimized to make decisions from truncated contexts, matching what it sees at deployment.
• Train w/o HL + inference w/o HL (full history, aligned): performs worse than the truncated variant, suggesting that full-history training learns patterns that don't transfer well even when inference also has full history (possibly due to attention dilution over long contexts).
• Misaligned settings (train w/o HL + inference w/ HL): the full-history-trained model evaluated with truncated history — the exact distribution mismatch the paper identifies. Performance drops relative to the aligned truncated variant, though "relatively insensitive to misalignment" on τ²-bench, where truncation actually slightly improves performance (consistent with suppressing interference from earlier irrelevant turns).
• Misaligned (train w/ HL + inference w/o HL): the truncated-history-trained model evaluated with full history — also degraded vs. aligned truncated.

The key takeaway is that "history management should be treated as part of policy optimization rather than a purely inference-time heuristic." The paper does not claim the specific sliding window approach is optimal — more complex context management is acknowledged as beyond scope — but the alignment principle is validated.

Quality of Synthesized Environments (Table 5, Figure 3, Section 6.1)

Headline result: AWM environments are rated higher than EnvScaler environments on Task Feasibility, Data Alignment, and Toolset Completeness by both LLM judges.

Table 5 reports LLM-as-a-Judge scores (1–5 scale, higher is better) on a sample of 100 environments, evaluated by GPT-5.1 and Claude-4.5-Sonnet:

• Task Feasibility ("whether tasks are executable within the environment"): AWM outscores EnvScaler.
• Data Alignment ("whether the data schema is coherent with the task"): AWM outscores EnvScaler.
• Toolset Completeness ("whether the toolset is complete and usable"): AWM outscores EnvScaler.

The paper attributes this to "stronger end-to-end consistency from tasks → database → interface" in AWM's pipeline — the sequential dependency (tasks dictate schema, schema constrains API) produces more coherent environments than generating these components with looser coupling.

Bug analysis (Table 5, bottom section). Despite AWM environments containing roughly 3× more code than EnvScaler (1,985 vs. 662 lines on average, Table 3), the increase in bugs is only "moderate." Manual inspection of AWM environments attributes 44% of bugs to not handling edge input cases (e.g., invalid parameter types, missing optional fields) and 14% to operations conflicting with database constraints (e.g., foreign key violations, unique constraint violations). Both methods exhibit implementation issues at this scale. AWM yields fewer "blocked tasks" (tasks that cannot be executed at all due to environment bugs) than EnvScaler, which the paper identifies as critical for RL because "blocked tasks truncate exploration and inject systematically incorrect negative signals."

Diversity analysis (Figure 3). Two complementary measures:

• Embedding diversity (Figure 3a): the semantic diversity of environments (measured by encoding scenario descriptions, database schemas, and toolset schemas) remains stable as the pool grows from 0 to 1,000. This means newly generated environments continue to add novel content rather than forming near-duplicates — the deduplication filter (cosine similarity < 0.85 threshold) is working.
• Category coverage (Figure 3b): the number of unique topic categories steadily increases, showing AWM "globally expands into new regions instead of collapsing to a few dominant domains." The category caps enforced during scenario generation prevent over-representation of common types like e-commerce.

These results collectively validate that AWM produces environments that are both sufficiently high-quality (tasks executable, schemas coherent, tools sufficient) and sufficiently diverse (semantically novel, categorically broad) to serve as RL training grounds.

Ablation Studies and Robustness Checks

• Step-level format correctness reward (Figure 5, Appendix B.1): Disabling the format reward causes the format error ratio to remain above 20% even after 50 optimization steps, compared to rapid convergence to a low level with the reward enabled. Without the format reward, the task completion rate saturates below 40%, whereas with it, performance continues improving. The format reward also improves training efficiency by reducing average rollout time by about 27% — trajectories that would have continued generating invalid tool calls are terminated early, saving computation.

• Verification strategy (Table 6, Section 6.2): LLM-only verification (ungrounded in database state) yields the weakest RL training signal. Code-only verification improves over LLM-only but is brittle to environment imperfections. Code-augmented (hybrid) verification achieves the best results across all model scales and benchmarks. The extra cost of the LLM judge (~$1.80 per training step, up to 1,024 samples, asynchronous) is negligible. Appendix B.2 provides three concrete case studies (Figures 28–30) demonstrating: (a) code verifier and judge aligning on a clean success, (b) code-only verifier producing a false negative from a transient tool error while the code-augmented judge correctly recovers using trajectory context, and (c) LLM-only judging being fooled by a spurious success that the code-augmented judge catches using database evidence.

• History-aware training (Table 7, Section 6.3): Training with truncated histories (w=3 sliding window) and evaluating with the same truncation (aligned) yields the best results. Training with full history and evaluating with truncated history (misaligned) degrades performance. Training with full history under aligned full-history evaluation still underperforms truncated training. The full-history variant shows "relative insensitivity to misalignment" with slight improvement when truncated at inference on τ²-bench — plausibly because truncation removes distracting earlier turns.

• Environment scaling (Figure 4, Section 6.4): Training on 10 environments causes severe degradation vs. Base on all benchmarks (overfitting). Scaling to 100 yields substantial gains. Scaling to 526 continues the upward monotonic trend without signs of saturation. Due to compute constraints, the full 1,000 environments are not tested, but the diversity analysis suggests continued benefits.

• Training set size vs. full pipeline: The paper trains on 526 of 1,000 environments (3,315 of 10,000 tasks) due to "limited computing resources." This is not an ablation per se but a practical constraint that means the reported results are a lower bound on what the full synthesized set could achieve.

• Model scale consistency: The pattern of AWM > Simulator and AWM improving over Base holds across 4B, 8B, and 14B scales (Table 4). The gains are larger at smaller scales (4B: +12.09 on BFCLv3; 8B: +12.11; 14B: +3.69), consistent with the intuition that RL training provides more benefit to weaker base models (which have more room for improvement in tool-use mechanics) than to stronger ones (which already have competent tool-use abilities from pretraining).

• Simulator baseline as negative result: Simulator training consistently underperforms AWM across benchmarks and frequently regresses below Base — a strong negative result for LLM-simulated environments. On 8B τ²-bench, Simulator drops to 43.8 vs. Base 52.2 (−8.4), suggesting that training on hallucinated state transitions actively damages performance on tasks requiring consistent multi-turn state tracking. This is not merely a weaker improvement but a qualitative failure mode.

• EnvScaler's benchmark-specific behavior: EnvScaler is competitive with AWM on τ²-bench (52.8 vs. 58.3 for 8B) but regresses on BFCLv3 (44.90 vs. 53.83 Base) and MCP-Universe (37.0 vs. 38.4 Base). The paper hypothesizes this is "plausibly because EnvScaler relies on existing tasks for synthesis that may overlap with τ²-bench" — a form of benchmark leakage where environments generated from task sets similar to the evaluation benchmark provide an unfair advantage on that specific benchmark. This highlights a risk in taskset-conditioned synthesis that AWM's "from scratch" approach avoids.

Critical Assessment

Claim: "Agents trained on AWM generalize to out-of-distribution environments"

What was tested: Three benchmarks that differ from training along multiple dimensions (conversational interaction, refusal scenarios, browser automation, information retrieval). AWM improves over Base on all three benchmarks across three model scales, with no regressions — a clean and consistent result.

What limits the claim: "Generalize" is demonstrated across exactly three benchmarks, all in the tool-use domain, all involving API/function-calling interactions at their core. AWM trains on CRUD-heavy stateful applications (e-commerce, banking, booking, etc.) and evaluates on benchmarks that also involve structured tool interactions. The generalization demonstrated is from synthetic tool-use environments to real-world tool-use benchmarks — same task type (tool use), different environment distribution. Whether agents would generalize to fundamentally different interaction paradigms (GUI-based computer use, embodied robotics, multi-agent coordination) is untested and unlikely given the training signal. The claim of "out-of-distribution generalization" is accurate but bounded: it's out-of-distribution across environments within the tool-use domain, not across task types.

Missing evidence: The paper does not evaluate on any benchmark that requires no tool use (e.g., standard NLP reasoning tasks like MMLU, GSM8K) to verify that RL training doesn't degrade base model capabilities — a common concern in RL fine-tuning (catastrophic forgetting). If BFCLv3 gains come at the cost of degraded mathematical reasoning or factual knowledge, the practical value proposition weakens. The hallucination category weakness on BFCLv3 (agents penalized for not refusing invalid tool calls) hints at this tension: format rewards that encourage tool use may suppress appropriate refusal behavior. A broader capability evaluation would strengthen the claim that gains are net positive.

Claim: "Code-driven environments with SQL-backed state consistency provide more reliable training signal than LLM-simulated environments"

What was tested: AWM vs. Simulator (GPT-5 as environment transition model) across all three benchmarks. AWM outperforms Simulator on every benchmark and every model scale where both are reported. Simulator regresses below Base on τ²-bench 8B.

What limits the claim: The Simulator baseline uses GPT-5 as the environment model — a specific LLM with specific hallucination properties. A different simulator LLM (e.g., a fine-tuned smaller model specifically trained for state tracking) might perform differently. More importantly, the Simulator is not given the benefit of AWM's execution-based self-correction or verification design — it's trained with the same tasks and toolsets but with LLM-generated state transitions instead of code-driven ones. The poor Simulator performance could reflect either (a) fundamental unreliability of LLM-based state simulation, or (b) a suboptimal Simulator implementation that doesn't adequately address hallucination. The paper doesn't ablate whether techniques like chain-of-thought state tracking, explicit state representation in the simulator's prompt, or consistency checks could narrow the gap. The headline finding — code-driven >> LLM-simulated — is likely correct, but the magnitude of the gap may partly reflect the specific Simulator implementation rather than an inherent ceiling on LLM-based simulation.

Strong evidence for the claim: The Simulator's qualitative failure mode — regressing below Base on τ²-bench — is what you'd expect if inconsistent state transitions corrupt the learning signal. An agent trained to trust environment feedback that is sometimes hallucinated would learn unreliable policies. The fact that this regression appears specifically on the most stateful benchmark (multi-turn conversational tasks with persistent user state) and not uniformly across all benchmarks is consistent with the hypothesis that state inconsistency is the mechanism. Code-driven AWM shows no such regression anywhere.

Claim: "AWM substantially reduces RL latency compared to LLM-simulated environments"

What was tested: The paper states in Section 5.2 that AWM "substantially reduc[es] RL latency, since Simulator requires an LLM call at each interaction step." No quantitative latency measurements are reported.

What limits the claim: This is a stated claim without direct empirical support in the paper. No wall-clock time comparisons, no FLOPs comparisons, no cost-per-training-step comparisons between AWM and Simulator are provided. The claim is logically sound (a GPT-5 API call takes seconds; a local SQLite query takes milliseconds), but the magnitude is unquantified. The paper does report that AWM's LLM-as-a-Judge costs ~$1.80 per training step, but doesn't report the Simulator's per-step LLM cost for comparison. A rigorous latency/cost analysis would require measuring: (a) average time per environment step for AWM (SQLite query) vs. Simulator (GPT-5 API call), (b) total training wall-clock time for both approaches at equivalent scale, and (c) API cost for Simulator's environment simulation calls. These are missing.

Claim: "AWM outperforms concurrent synthesis method EnvScaler"

What was tested: EnvScaler baseline on three benchmarks, 8B model only (EnvScaler didn't release 14B). AWM achieves better overall average across benchmarks and improves all three; EnvScaler regresses on two of three.

What limits the claim: EnvScaler is tested on only one model scale (8B) and one training configuration (whatever the authors provided). It's unclear whether the EnvScaler agents were trained with comparable hyperparameters, comparable compute budgets, or comparable environment interaction volumes to AWM agents. The paper's Table 4 reports EnvScaler numbers but doesn't specify whether these were reproduced by the AWM authors using a common training protocol or taken from EnvScaler's original paper. If training protocols differ significantly (e.g., EnvScaler used fewer training steps, different RL algorithm, different reward design), the comparison confounds environment quality with training methodology. The regression on BFCLv3 (−8.93 vs. Base) is so severe that it may reflect a training failure rather than environment quality alone — a well-trained agent on mediocre environments should not substantially underperform an untrained base model.

Fairness of the comparison: EnvScaler is a concurrent work, not a baseline the AWM authors controlled. The paper acknowledges that EnvScaler "relies on existing tasks for synthesis that may overlap with τ²-bench" — this is a potential confound that could explain the competitive τ²-bench performance. A cleaner comparison would involve training both AWM and EnvScaler agents with identical RL infrastructure, hyperparameters, and compute budgets, which wasn't done.

Claim: "Environment diversity is critical for agentic RL, with monotonic improvements from 10 to 526 environments"

What was tested: Figure 4, 4B model, three environment counts (10, 100, 526), three benchmarks.

What limits the claim: The scaling analysis uses a single model scale (4B) and three data points (10, 100, 526). A proper scaling law analysis would require more granular counts (e.g., 10, 25, 50, 100, 200, 300, 500) to characterize the shape of the curve — is it logarithmic, power-law, linear? The paper claims "monotonic improvement" but with only three points, the confidence in this claim is limited. The choice of 10 environments also conflates two variables: at 10 environments, the number of tasks is also reduced (~330 tasks vs. ~3,315 at 526 environments), making it unclear whether the poor performance at 10 environments reflects insufficient environment diversity, insufficient task diversity, or both.

Robustness concern: The environment subsets are not described as being sampled with any stratification or repetition. If the 10-environment subset happened to contain mostly degenerate or buggy environments (which Table 5 suggests exist at some rate), the poor performance could reflect environment quality variance rather than a diversity effect. Repeating the scaling analysis with multiple random samples at each subset size — and reporting variance — would substantially strengthen the claim.

Nevertheless, the practical message is clear: training on a handful of environments (like the 2–5 in existing benchmarks) is insufficient and apparently harmful. The paper's contribution of 1,000 environments is a meaningful step toward addressing this.

Claim: "Code-augmented LLM-as-a-Judge provides more robust reward signals than pure code-only or LLM-only verification"

What was tested: Table 6, comparing three verification strategies via the downstream RL training performance of agents trained with each strategy, across model scales and benchmarks.

What limits the claim: The comparison is indirect — the paper measures which verification strategy yields better trained agents, not which verification strategy is more accurate at classifying task outcomes. It's possible that code-augmented verification is no more accurate than code-only verification at classifying individual tasks, but produces better RL training because of other factors (e.g., the 0.1 partial completion reward provides a smoother training signal than the hard 0/1 of code-only). The case studies in Appendix B.2 (Figures 28–30) provide qualitative evidence of specific failure modes for each approach, but these are three hand-picked examples, not a systematic accuracy evaluation. A direct comparison — measuring agreement between each verification strategy and human judgments on a held-out set of agent trajectories — would provide stronger evidence for the "more robust" claim.

Hidden assumption: The code-augmented judge uses GPT-5 as the reasoning LLM. The quality of the judge depends on GPT-5's reasoning capabilities, which may not transfer to cheaper or open-source judge models. If the community needs to use a GPT-5-class model as judge for every training run, the approach is expensive and potentially fragile to model version changes. The paper doesn't ablate judge model quality.

Experiments That Would Have Strengthened the Paper

1. Direct verification accuracy evaluation: human-annotated ground-truth task outcomes on a sample of AWM trajectories, comparing code-only, LLM-only, and code-augmented classification accuracy. This would directly validate the "more robust" claim rather than relying on downstream RL performance as a proxy.

2. Simulator with state-tracking enhancements: the poor Simulator performance is used to argue that code-driven environments are necessary, but this conclusion is stronger if the paper shows that even enhanced Simulators (with explicit state tracking, consistency checks, or retrieval-augmented memory) still underperform code-driven environments. Without these ablations, the finding could be specific to a naive Simulator implementation.

3. Full 1,000-environment training: the paper trains on 526/1,000 environments due to compute constraints and extrapolates that further scaling would help. Training on the full set (or at least reporting the trend beyond 526) would validate this extrapolation and establish whether the gains saturate.

4. Non-tool-use capability retention: evaluating AWM-trained agents on standard NLP benchmarks (MMLU, GSM8K, HellaSwag) to verify that RL training doesn't catastrophically forget base capabilities. The narrow focus on tool-use benchmarks leaves open the question of whether the trained agents are better tool-users at the expense of being worse at other tasks.

5. Cost/latency comparison with Simulator: quantified wall-clock time and API cost for AWM training vs. Simulator training at equivalent scale, to support the claim of "substantially reduced RL latency."

6. Training on real benchmarks (upper bound): if feasible, training directly on τ²-bench environments (as an oracle upper bound) to quantify the gap between synthetic-environment training and domain-specific training. This would contextualize the AWM gains — are the synthetic environments capturing 50% of the possible gain? 80%?

7. Agent robustness to environment bugs: the paper acknowledges environments have bugs (~44% from edge input cases). How robust are trained agents to these bugs? Do agents learn to recover from environment errors, or do environment errors systematically degrade their policies? The 4% environment error rate during training is reported, but agent behavior in response to errors is not analyzed.

8. Ablation of the two-level tool abstraction: the list_tools + call_tool meta-protocol is a design choice. Would agents trained with environment-specific tools hardcoded in the system prompt perform better (since they don't need to discover tools) at the cost of zero transfer to new environments?

Summary of Experimental Rigor

The paper's experimental strength lies in consistency across benchmarks and model scales. AWM improves over Base on all three benchmarks, all three model scales, with no regressions. This pattern is unlikely to be noise. The Simulator and EnvScaler comparisons, while imperfect (different training protocols, limited model scales), show clear enough patterns — AWM consistently outperforms both — to support the paper's core argument that code-driven, database-backed environments provide uniquely effective training signal.

The main weaknesses are (1) limited quantification of key claims (latency reduction, verification accuracy, scaling law shape), (2) missing evaluations on non-tool-use capabilities, and **(3) reliance on downstream RL performance as the primary validation metric rather than direct measurement of environment quality, verification accuracy, or training dynamics. The results demonstrate that the full AWM system works — agents improve on benchmarks — but the paper leaves partially open the question of which components of the system are most responsible for this improvement. The ablation studies (verification strategy, format reward, history alignment, environment count) partially address this, but several design choices (two-level tool abstraction, SQLite vs. NoSQL, task-driven schema generation vs. alternative approaches) are not individually ablated.

6. Limitations and Trade-offs

The Difficulty Estimation Step Is as Expensive as the Inference Budget It Is Supposed to Optimize

The assumption or constraint. The compute-optimal framework rests entirely on the ability to estimate each prompt's difficulty before allocating the test-time compute budget. The paper's method for doing so is generating 2048 samples per question and averaging either ground-truth correctness (oracle difficulty) or PRM final-answer scores (predicted difficulty). The computational cost of this step is staggering: 2048 generations per question is between 8× and 128× the size of the inference budgets the paper studies (which range from 16 to 256 generations, and reach up to 512 for some experiments). The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The headline 4× efficiency gains are computed after difficulty is already known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty_estimation_cost + strategy_execution_cost, and the former could dominate the latter. For a deployment with a 256-generation budget, difficulty estimation (2048 generations) would consume an additional 8× the problem-solving budget — meaning the total compute spent per question would be ~9× the baseline, not 0.25× as the 4× figure might suggest. The 4× claim should therefore be understood as an upper bound on achievable efficiency under the assumption of free difficulty estimation, not a realized deployment gain.

What evidence exists in the paper. The cost discrepancy is not measured, plotted, or quantified anywhere. The paper mentions the issue once (Section 3.2) and does not include difficulty estimation cost in any budget calculation throughout Sections 4–7. The FLOPs-matched comparison (Section 7) accounts for pretraining and inference FLOPs carefully but does not amortize difficulty estimation FLOPs into the test-time compute budget, meaning the reported advantages of test-time compute over pretraining are overstated by an unknown factor that depends on the difficulty estimation scheme.

Mitigation status. The paper flags this as "a key avenue for future work" (Section 3.2) and suggests training models to predict difficulty directly from the question text, but no such model is developed or evaluated. No adaptive difficulty estimation scheme (e.g., using a small number of initial samples to estimate difficulty online) is tested. The paper does not provide any empirical upper bound on what fraction of the budget difficulty estimation would consume under a more practical scheme, so the practitioner has no guidance on whether the unaccounted cost is 2× or 10× the optimization budget.

The Ganho de Eficiência de 4× Baseia-se em Um Conjunto de Teste de 500 Questões com Estratégias Selecionadas em ~50 Amostras por Bin

The assumption or constraint. The compute-optimal strategy -- which search algorithm to use, what sequential-to-parallel ratio to deploy -- is selected per difficulty bin using two-fold cross-validation on the 500-question MATH test set. With five difficulty quintiles, each bin contains approximately 100 questions. Two-fold cross-validation splits each bin roughly in half, meaning the optimal strategy for a given bin is selected based on approximately 50 questions. This is a small sample for making discrete strategy choices (e.g., "beam search with M=4 at 64 generations" vs. "best-of-N weighted at 64 generations") from a combinatorial space of options that includes multiple search algorithms, beam widths, lookahead depths, sequential-to-parallel ratios, and generation budgets.

The paper does not report confidence intervals, standard deviations, or cross-validation variance on the compute-optimal scaling curves (Figures 4 and 8). No sensitivity analysis examines how the selected strategies would change with a different random split or a larger test set.

The consequence. The compute-optimal strategies presented in Figures 4 and 8 may be overfit to the specific 500-question test set. With only ~50 validation questions per bin determining which of several candidate strategies is best, a strategy that happens to perform 2–3 percentage points better on that small validation fold (potentially due to sampling noise rather than genuine superiority) could be selected as "optimal." The performance reported on the held-out fold would then be an optimistic estimate of generalization, since the strategy was cherry-picked to maximize held-out performance in the same cross-validation loop.

The concern is not that the paper is cheating -- the cross-validation is correctly implemented -- but that 500 questions split 5 ways and then split again for cross-validation provides insufficient statistical power to reliably distinguish between strategies whose true performance differences might be in the 1–3% range. Since the compute-optimal curves in Figures 4 and 8 show improvements over baselines that are often in this range (e.g., Figures 4 shows compute-optimal at ~27% vs. best-of-N at ~23% at 16 generations), the apparent gains could partially reflect estimation noise advantageously selecting strategies on the validation fold.

What evidence exists in the paper. The paper uses a fixed 500-question test set throughout (Section 4). No standard deviations, confidence intervals, or cross-validation variance estimates appear on any main results figure (Figures 3, 4, 6, 7, 8, 9). The two-fold cross-validation description (Section 3.2) is brief and does not discuss statistical power or the effect of bin size on strategy selection reliability. Figure 4 shows the predicted-difficulty and oracle-difficulty curves largely overlapping, which provides some reassurance (if both were noisy, they might diverge more), but this is indirect evidence and does not address the question of whether a different random split would produce different strategy selections.

Mitigation status. Not addressed. The paper does not compute confidence intervals, does not repeat the cross-validation with multiple random splits, does not analyze strategy selection stability, and does not discuss the statistical limitations of operating on 500 questions split into 5 bins. The authors do not claim statistical significance for any result, but the presentation of precise numbers (e.g., "4× better efficiency") without error bars implicitly invites readers to treat these as point estimates with unknown variance.

Only One Model Family (PaLM 2-S*) and One Benchmark (MATH) Are Tested, Limiting Generality Claims

The assumption or constraint. All experiments -- search, revisions, compute-optimal allocation, FLOPs-matched comparisons -- use PaLM 2-S* as the base model and the MATH benchmark (500 test questions, competition-level math) as the evaluation task. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an assertion, not an empirical demonstration. No other model families (GPT, Claude, LLaMA, Mistral, Qwen) are used as base models. No other reasoning benchmarks (GSM8K, MMLU, ARC, code generation, logical reasoning) are used for evaluation.

The consequence. Several aspects of the findings could be model-specific or domain-specific:

• PRM quality and over-optimization behavior (Section 5.3) depend on PaLM 2-S*'s output distribution. A model with different calibration properties -- particularly one that is better or worse at math -- might exhibit qualitatively different difficulty-dependent scaling curves. The finding that beam search over-optimizes on easy problems (Figure 3, right) could be specific to the interaction between PaLM 2-S*'s error patterns and the Monte Carlo-trained PRM.

• Revision model training (Section 6.1) depends on the base model's in-context learning capabilities and its ability to learn from incorrect→correct example trajectories. These capabilities vary substantially across model families, and the specific fine-tuning recipe (edit-distance-based pairing, 4-turn training sequences) may not transfer.

• The MATH benchmark consists exclusively of symbolic math problems requiring step-by-step deductive reasoning. The difficulty-dependent patterns -- revisions helping on easy problems (where the model's initial output is roughly correct and needs refinement) and search helping on medium problems (where exploration matters) -- may be specific to problems with a well-defined solution space and verifiable intermediate steps. Tasks requiring factual recall (e.g., "What is the capital of Burkina Faso?") or tasks with open-ended output (summarization, creative writing) have fundamentally different difficulty structures and would not necessarily exhibit the same patterns.

What evidence exists in the paper. The paper provides zero evidence for cross-model or cross-domain generalization. No ablation uses a different base model. No results on any benchmark other than MATH appear in the main text or appendices. The FLOPs-matched comparison uses one additional model (a ~14× larger PaLM variant) but still within the same family.

The single-model, single-benchmark scope is acknowledged implicitly rather than explicitly -- the paper never claims generality to other models or domains, but the framing ("we believe this model is representative") invites the reader to assume generality without evidence. The "representativeness" claim is particularly suspect given that PaLM 2-S* is a proprietary model (Anil et al., 2023) whose training data, architecture details, and specific capabilities are not fully public, making it impossible for external researchers to assess whether it is indeed "representative."

Mitigation status. Not addressed. The paper does not attempt experiments with any other model family or benchmark. The acknowledgments and limitations sections do not flag this as a limitation. The claim of representativeness ("we believe") is stated without qualification or caveat.

The ~14× Larger Model Baseline In the FLOPs-Matched Comparison Is Not Compute-Optimally Trained, Making the Comparison Favorável ao Test-Time Compute

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time strategies to a model with approximately 14× more parameters trained on the same data, with no test-time augmentation (greedy decoding only). The paper explicitly acknowledges a key asymmetry:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The "canonical approach" referenced is the LLaMA paradigm (Touvron et al., 2023) -- scaling parameters while holding training data fixed -- which is known to be suboptimal compared to Chinchilla-optimal training (Hoffmann et al., 2022), where both parameters and data are scaled proportionally. A Chinchilla-optimal model trained with 14× more total FLOPs would allocate some of that budget to additional training data, likely yielding better performance than a pure parameter-scaled model.

The consequence. The reported advantages of test-time compute over pretraining (Figure 1 bar charts, Figure 9) are measured against a weaker baseline than they should be. For example, the +27.8% relative improvement on easy/medium questions at R << 1 (Figure 1, top-right) compares a compute-optimally-inferenced smaller model against a non-compute-optimally-trained larger model. The true gap against a properly scaled baseline is unknown and likely smaller -- possibly substantially so.

There is a second asymmetry: the smaller model receives all of its budget advantage as test-time compute (allowing strategies like beam search, sequential revisions, or adaptive allocation), while the larger model receives none -- it uses greedy decoding with no best-of-N, no search, no revisions. A fairer comparison would allocate some fraction of the larger model's inference budget to test-time augmentation as well. This means the reported advantages of test-time compute are confounded with the larger model being given a weaker inference strategy.

What evidence exists in the paper. The paper explicitly states the Chinchilla caveat (quote above) and the greedy decoding choice but does not quantify how much these choices affect the comparison. No ablation examines what happens if the larger model is given even modest test-time compute (e.g., best-of-4, best-of-8). No estimate of how much a Chinchilla-optimal 14× larger model would outperform the parameter-scaled version is provided.

The practical effect is visible in Figure 9: the 14× larger model's performance (stars) is at a fixed level regardless of the value of R, because it uses greedy decoding with no variable inference budget. A more realistic baseline would show the larger model's performance improving as R increases (since at higher R, the larger model gets a proportionally larger inference budget in absolute terms), which would narrow or reverse the test-time compute advantage more quickly than the paper's static comparison suggests.

Mitigation status. The paper acknowledges the limitation ("leave the analysis of compute-optimal scaling of pretraining compute... to future work") but does not treat it as a caveat on the reported results. The headline claim ("a smaller model augmented with compute-optimal test-time strategies can outperform a ~14× larger pretrained model") is stated in the abstract and executive summary without qualification, despite the acknowledged asymmetry in the baselines. A reader who does not carefully read Section 7's methodology would reasonably interpret this as a fair comparison between pretraining and inference compute, when it is actually a comparison between compute-optimal inference + non-compute-optimal pretraining vs. compute-optimal inference + compute-optimal pretraining.

Test-Time Compute Cannot Help on the Hardest Problems -- a Hard Upper Bound on Applicability

The assumption or constraint. The entire framework assumes that the base model produces correct solutions at some non-negligible rate for the problems being solved. On MATH difficulty bin 5 (the hardest quintile of questions), PaLM 2-S*'s pass@1 is approximately 1–3%, meaning it almost never produces a correct solution on the first try.

The consequence. As the paper's own results demonstrate, no amount of test-time compute helps on these problems:

• In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods at all budgets from 4 to 256 generations. Beam search, best-of-N, lookahead -- all fail equally.
• In Figure 7 (right), bin 5 accuracy is roughly 2–3% irrespective of the sequential-to-parallel ratio at 128 generations.
• In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, while the 14× larger model's performance (stars) is above it for all values of R. On hard problems, pretraining is always better.

The finding is stark: test-time compute amplifies existing capability but cannot create capability from nothing. If the base model's pass@1 is near zero on a problem class, no amount of search, revision, or adaptive allocation will produce correct answers -- the proposal distribution simply contains no correct samples to find or refine. This establishes a hard upper bound: the approach works only on problems that are within the base model's approximate capability range.

What evidence exists in the paper. The evidence is strong and consistent across every experiment. Bin 5 appears as a flat line at 0–5% in every figures that breaks out results by difficulty (Figures 3 right, 7 right, 9). The paper acknowledges this in Section 5.3 ("On the hardest questions (bin 5), no method makes meaningful progress") and in the Section 7 takeaway box, but the severity of the bound -- "the model simply lacks the capability to produce correct solutions regardless of how the budget is allocated" -- is buried in the results sections rather than foregrounded as a fundamental limitation.

Mitigation status. The limitation is inherent and cannot be mitigated within the test-time compute paradigm. It is not a bug but a feature of the framework: test-time compute searches over or refines the model's existing output distribution, and if that distribution places zero probability mass on correct answers, search and refinement are futile. The only path to solving truly hard problems is pretraining (or fundamentally different inference approaches like tool use, retrieval, or multi-agent collaboration -- none of which are studied in this paper). The paper acknowledges this but could be clearer in stating that its approach offers no path forward for problems where the model has zero or near-zero baseline capability.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate, and Mitigations Are Heuristic

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). This means the model has never seen a correct answer in its context during training -- it only learns "incorrect → correct" transitions. At test time, as the revision chain progresses, earlier steps may produce correct answers that become part of the context for later steps. The model, untrained for this scenario, has a significant tendency to "revise" correct answers into incorrect ones.

The paper quantifies this: approximately 38% of correct answers get converted back to incorrect ones using a naive approach (Section 6.1). This is not a minor edge case -- it means that roughly 4 out of every 10 correct revisions are subsequently corrupted by the model's next revision step, a catastrophic failure rate for any system that relies on sequential refinement.

The consequence. The within-chain selection mechanisms used to mitigate this -- majority voting across the chain or verifier-based selection of the best answer from any point in the chain -- are heuristic patches that do not address the root cause. They work by detecting when the model has regressed and selecting an earlier, correct answer, but they cannot prevent the regression from happening. This means:

• The effective length of useful revision chains is capped. If every correct answer has a 38% chance of being corrupted in the next step, long revision chains (which the paper shows continuing to improve out to 64 steps, Figure 6 left) are unreliable -- the chain may contain high-quality answers at some points, but relying on post-hoc selection to find them adds variance.
• The selection mechanisms add computation. Majority voting requires generating enough revisions to form a consensus; verifier-based selection requires a trained verifier (and the paper shows in Appendix J, Figure 15a, that the base-LM PRM doesn't transfer well to revision model outputs, requiring a separate revision-specific ORM). These are practical costs not reflected in the headline generation budget comparisons.
• The failure mode is a direct consequence of the training data construction. The paper chose to train on only incorrect→correct trajectories to simplify data generation. Including "correct→correct" training examples (where the model is trained to recognize when no revision is needed and output the same answer) would address the root cause, but this requires a more complex data generation pipeline that can identify when an answer is already correct.

What evidence exists in the paper. The 38% figure is reported in Section 6.1. The paper implements two mitigations -- majority voting and verifier-based selection -- and shows they partially recover performance (Figure 6, right, shows sequential + selection outperforming parallel). However, no ablation tests the revision model without selection mechanisms to isolate how much the 38% reversion rate actually degrades performance. The paper does not experiment with "correct→correct" training data to demonstrate that the problem is solvable.

The ReST$^{EM}$ experiment (Appendix K, Figure 16) provides additional evidence of revision training fragility: attempting to optimize the revision model with RL-style training caused performance to degrade substantially with sequential revisions, with the authors hypothesizing that "on-policy data collection exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This suggests that the revision approach is sensitive to training methodology in ways that are not fully understood, and the positive results depend on specific choices (offline data construction, edit-distance-based pairing) that may not transfer to other settings.

Mitigation status. Partially addressed. The paper implements and evaluates selection mechanisms that mitigate the symptom (correct→incorrect reversion) without curing the cause (training data distribution). The authors do not frame the 38% reversion rate as a limitation requiring deeper solution, nor do they explore training data modifications that would eliminate it. The ReST$^{EM}$ negative result is reported but not analyzed as a symptom of training data fragility. A more principled solution -- training the model to recognize when no revision is needed, or using a verifier to gate whether a revision step is attempted -- is not explored.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around agent training from an environment-scarcity mindset — where researchers make do with a handful of hand-crafted benchmarks and hope trained agents generalize — to an environment-synthesis mindset, where diverse training environments are treated as a scalable resource to be generated algorithmically rather than authored manually. The shift is analogous to what ImageNet did for computer vision or what Atari did for deep RL: it provides the infrastructure (the environments) and the method (the synthesis pipeline) that makes a previously niche research paradigm (large-scale agentic RL) accessible to the broader community.

The magnitude is best characterized as enabling infrastructure rather than a paradigm shift or an incremental refinement. The paper does not claim a new learning algorithm, a new model architecture, or a new theoretical insight. It claims something more foundational: a way to produce the thing that learning algorithms need — diverse, executable, state-consistent environments for tool-use agents — at a scale (1,000 environments, 35,000 tools, 10,000 tasks) that is 5× larger than the nearest prior work and 200–500× larger than what human-authored benchmarks provide. This is infrastructure work, and its impact depends on adoption: if the community uses AWM's pipeline and released environments to train better agents, the paper will be retrospectively important; if the community ignores it, the paper will be a curiosity. The fact that the pipeline and environments are fully open-source (unlike DeepSeek-V3.2 and Qwen Tongyi's unreleased pipelines) makes adoption likely.

What contradictions does this work resolve? The paper implicitly resolves a tension that has been building in the agent training literature: the gap between evaluation benchmarks and training environments. Existing benchmarks (τ-bench, τ²-bench, MCP-Universe, BFCLv3) are well-designed for measuring agent capability, but they are impossible to use for RL training — they have too few environments, they can't be reset efficiently, they depend on real APIs with rate limits and changing interfaces. The paper demonstrates that this gap is not insurmountable: synthetic environments that share structural properties with real services (database-backed state, MCP interface, multi-step tasks) can provide effective training signal without requiring access to real APIs. The counterintuitive finding is that agents trained on purely synthetic environments — with no overlap with evaluation domains — outperform agents trained on LLM-simulated environments and match or exceed agents trained on programming-based environments derived from real task sets. This suggests that what matters for training signal is not realism (looking like a real service) but consistency (deterministic state transitions, reliable rewards, diverse action spaces) — and that synthetic environments can provide these properties more reliably than LLM-simulated ones and at greater scale than human-authored ones.

Which research directions become more attractive? The paper makes several directions newly tractable:

• Large-scale agentic RL becomes feasible without access to proprietary APIs or expensive human environment design. Any research group with access to a strong LLM (for environment generation) and moderate GPU resources (for RL training) can now train tool-use agents at scale.
• Scaling law studies for agent training become possible. With 1,000 environments (and the pipeline to generate more), researchers can systematically vary environment count, task diversity, tool complexity, and interaction length to characterize how agent performance scales — analogous to how Hoffmann et al. (2022) characterized pretraining scaling laws.
• Environment quality and diversity as first-class research objects. The paper's quality analysis (Table 5, Figure 3) and scaling analysis (Figure 4) demonstrate that environment design choices (schema coherence, bug rates, diversity) directly impact downstream agent performance. This opens a research agenda around environment optimization — how to design synthesis pipelines that maximize agent learning per environment, rather than treating environments as a fixed given.

Which directions become less attractive? The paper's strong negative result for LLM-simulated environments (Simulator baseline underperforms AWM across all benchmarks, and regresses below Base on τ²-bench for the 8B model) should give pause to research that treats LLM-based environment simulation as a viable training strategy for stateful tool-use tasks. The finding that code-driven environments with SQL-backed state consistency produce better agents is not surprising in retrospect, but the magnitude of the gap — and the fact that Simulator training can be actively harmful — suggests that LLM-based simulation has fundamental limitations (hallucination in state transitions, inability to maintain consistency over long trajectories) that prompt engineering alone cannot solve. Research energy may shift from "how can we make LLMs better at simulating environments?" to "how can we synthesize code-driven environments more efficiently and at greater scale?"

A note on what the paper does NOT change. The paper does not claim that synthetic environments are a substitute for real-world deployment or that agents trained on AWM are ready for production. The environments are simplified approximations — they don't handle authentication, they don't model real-world latency or failures, they don't capture the full complexity of production services. The paper is also transparent that the hardest problems (bin 5 in the MATH analogy, though this paper doesn't use difficulty bins) — tasks requiring capabilities far outside the model's training distribution — are not addressed. AWM is infrastructure for training, not for deployment, and the paper does not blur this boundary.

Follow-Up Research This Work Enables

Scaling environment count to 10,000+ and characterizing the scaling law shape. The paper's scaling analysis (Figure 4) shows monotonic improvement from 10 → 100 → 526 environments with no saturation, but uses only three data points and one model scale (4B). A natural follow-up would train 4B, 8B, and 14B agents on environment counts spanning 10, 25, 50, 100, 200, 400, 800, 1,000, and 2,000+ (by running the AWM pipeline to generate additional environments beyond the current 1,000). The goal would be to fit a power-law or logarithmic function to the accuracy-vs-environments curve and determine whether gains saturate or continue indefinitely. This would answer: (a) is there an environment count beyond which additional environments provide diminishing returns? (b) does the scaling exponent differ across model sizes? (c) does the scaling behavior differ across benchmarks (suggesting some benchmarks require more environment diversity than others)? The paper's diversity analysis (Figure 3a) showing embedding diversity remaining stable at 1,000 environments suggests saturation is not yet reached, but this needs direct empirical verification through agent training results.

Combining AWM's code-driven environments with curriculum learning over difficulty. The paper treats all environments uniformly — every training step samples tasks uniformly from the 3,315-task pool. A strong follow-up would implement difficulty estimation for AWM tasks (analogous to the difficulty bins in the analysis paper's Section 3.2, but using task completion rates or tool-call complexity as the difficulty signal) and train agents with a curriculum: start with simple tasks (1–3 tool calls, straightforward state changes) and progressively introduce harder tasks (6+ tool calls, conditional logic, error recovery). The hypothesis is that AWM's automatically generated task set contains a natural difficulty gradient — simple queries vs. complex multi-step transactions — and that curriculum learning would improve sample efficiency and final performance. Concretely, one could use the average number of agent steps per task (reported as 9.8 in Table 2) as a proxy for difficulty, or train a lightweight difficulty predictor on the first few rollouts per task. The paper's infrastructure already supports this: tasks are generated independently for each environment, so difficulty labeling can be added post-hoc without changing the pipeline.

Ablating SQLite vs. NoSQL/key-value stores for state consistency. The paper argues that SQL-backed state management provides stronger consistency guarantees than the NoSQL or key-value stores used in concurrent work (EnvScaler, AutoEnv). This claim is asserted but never tested directly. A clean ablation would generate AWM environments with identical scenarios, tasks, and toolsets but with two backends: (a) the current SQLite backend with foreign key constraints and transactional guarantees, and (b) a simplified key-value store backend (e.g., a Python dictionary serialized to disk) that tracks state but without relational constraints. Training identical agents on both environment sets would isolate the effect of relational state management on agent learning. The hypothesis is that SQL-backed environments produce better agents because they enforce data integrity — the agent cannot, for example, create an order referencing a non-existent product — which provides implicit negative feedback when the agent makes inconsistent tool calls. If the performance gap is small, it would suggest that the code-driven determinism (rather than the relational structure) is the active ingredient, which would simplify future environment synthesis.

Training on AWM + evaluating on completely held-out real-world APIs (e.g., Stripe, Gmail, Salesforce). The paper evaluates on three tool-use benchmarks that, while out-of-distribution, still involve structured API interactions. A more stringent test of generalization would evaluate AWM-trained agents on real-world production APIs that have no synthetic counterpart in AWM — for example, the Stripe payments API, the Gmail API, or the Salesforce CRM API. These APIs have idiosyncratic naming conventions, authentication patterns, pagination schemes, and error handling that differ from AWM's simplified environments. Strong performance would demonstrate that AWM teaches general tool-use meta-skills (discovering tools, composing API calls, recovering from errors) rather than overfitting to the AWM tool distribution. Weak performance would bound the generalization claim and suggest that some level of domain-specific training (or at least domain-specific environment synthesis) is necessary. The Berkeley Function Calling Leaderboard (BFCLv3) already includes real-world API categories, but a dedicated study with a curated set of 5–10 production APIs would provide clearer signal than aggregate benchmark scores.

Replacing GPT-5 as the LLM-as-a-Judge with an open-source model to eliminate the proprietary dependency and reduce cost. The paper's verification judge uses GPT-5 at ~$1.80 per training step. For a full training run of 96 steps, this is ~$173 in judge costs — manageable but not negligible, and it creates a dependency on a proprietary model with changing API access and pricing. A practical follow-up would ablate judge model quality: train AWM agents with judges ranging from small open-source models (Qwen2.5-7B, LLaMA-3-8B) to mid-size models (Qwen2.5-72B, Mixtral) to GPT-5, and measure both (a) judge accuracy on a human-annotated verification dataset (which the paper currently lacks) and (b) downstream agent performance. If a Qwen2.5-72B judge achieves comparable agent performance to GPT-5, the approach becomes fully self-hostable. The paper's verification code already provides structured evidence that should reduce the judge's reasoning burden — an open-source model may perform adequately when grounded in concrete database diffs even if its raw reasoning is weaker than GPT-5.

Stress-testing the pipeline's domain boundaries: what kinds of environments can AWM NOT synthesize? The paper focuses on CRUD-heavy stateful applications (e-commerce, banking, booking, task management) and explicitly excludes content-centric sites (news, wikis), search engines, AI inference services, and real-time data streams. But the boundary between "synthesizable" and "non-synthesizable" is fuzzy: can AWM generate a code-review platform (stateful: tracking pull requests, comments, approvals)? A healthcare claims processing system (stateful but with complex business logic and regulatory constraints)? A multiplayer game backend (stateful with concurrent user interactions)? A systematic stress test would attempt to generate environments at increasing distance from the CRUD template, characterize failure modes (where does the LLM start hallucinating incoherent schemas? where does self-correction fail?), and identify the minimal set of human inputs (scenario descriptions? schema templates? task examples?) needed to extend the pipeline to new domains. This would transform AWM from a point solution (good at e-commerce/banking/booking) to a general-purpose environment generator with well-understood capability boundaries.

Practical Applications and Downstream Use Cases

Cost-efficient training of custom tool-use agents for enterprise SaaS integrations. Many enterprises need agents that can interact with their internal tools (Salesforce, Jira, ServiceNow, custom HR systems) but cannot expose these production systems to RL training — the systems have rate limits, contain sensitive data, and cannot be reset after agent actions. AWM provides a blueprint: synthesize a synthetic environment that mimics the target system's schema, toolset, and task distribution, train an agent on the synthetic environment, then deploy the trained agent on the real system. The paper's results suggest this approach works: agents trained on synthetic e-commerce/booking/CRM environments generalize to real-world tool-use benchmarks (BFCLv3 +12.1 points for 8B, MCP-Universe +6.9 points). For an enterprise with 5–10 internal tools, generating a custom AWM environment for each tool (using the pipeline with scenario descriptions describing the tool's functionality) would cost ~$10–20 in LLM API calls per environment (extrapolating from Table 1's per-100-sample costs) and produce a training sandbox that can be reset thousands of times without touching production infrastructure. The trained agent could then be fine-tuned on a small number of real trajectories for final adaptation.

Data generation for supervised fine-tuning of open-source tool-use models. While the paper focuses on RL, the synthesized environments would also serve as powerful data generators for SFT. For each of the 10,000 tasks across 1,000 environments, one could run a strong agent (e.g., a prompted GPT-5 or Claude-4.5-Sonnet) with the ground-truth reward signal (the verification code) to collect successful trajectories — producing a dataset of ~10,000 verified tool-use demonstrations spanning diverse scenarios. This dataset could be used to fine-tune open-source models (Qwen, LLaMA, Mistral) for basic tool-use competence before RL fine-tuning. The paper's Table 2 shows tasks require an average of 9.8 agent steps with 35.1 tools available — these are non-trivial multi-step trajectories that would be expensive to collect from real APIs. The verification code provides automatic correctness filtering: only trajectories that pass the code-augmented judge would be included in the SFT dataset. This approach could produce a "ToolUse-10K" dataset analogous to existing instruction-tuning datasets, but with the advantage of being fully synthetic (no copyright concerns) and verified (no hallucinated trajectories).

Rapid prototyping and evaluation of new RL algorithms for agent training. The paper's infrastructure (1,024 parallel isolated environment instances per training step, pre-fetching, standardized MCP interface, structured reward signal) provides a reproducible testbed for algorithmic research on agentic RL. A researcher proposing a new RL algorithm (e.g., a variant of GRPO, a new exploration strategy, a hierarchical RL approach for tool composition) can benchmark it on AWM with controlled environment difficulty, task complexity, and interaction length — something impossible with existing benchmarks that have 2–5 environments and no support for parallel reset. The paper's training on 526 environments serves as a baseline; algorithmic improvements could be measured as sample efficiency gains (achieving the same 65.94 BFCLv3 score with fewer training steps) or asymptotic performance gains (achieving higher final accuracy at 96 steps). The open-source release of both the pipeline and the environments means this testbed is immediately available to any research group, lowering the barrier to entry for agentic RL research that currently requires access to proprietary APIs or custom infrastructure.

When to Prefer This Method

The paper positions AWM against three alternatives — human-authored environments, real-world APIs, and LLM-simulated environments — and the experimental results provide clear guidance on when each is appropriate:

• Prefer AWM-style synthetic environments when: (a) you need to train agents via RL at scale and the number of available real environments is insufficient (the paper shows severe overfitting at 10 environments, and existing benchmarks provide 2–5), (b) you cannot expose production APIs to RL training due to rate limits, cost, data sensitivity, or inability to reset state, (c) the target domain involves CRUD-heavy stateful applications where database-backed environments can provide realistic state transitions (e-commerce, booking, banking, task management, CRM, inventory), and (d) you need the training infrastructure to support parallel isolated instances for efficient online RL (the paper launches 1,024 instances per step).

• Prefer LLM-simulated environments only when: the paper's results suggest limited scenarios where Simulator might be appropriate — the Simulator baseline underperforms AWM across all benchmarks and actively regresses on τ²-bench. However, the paper does not test Simulator with enhanced state-tracking mechanisms, so this recommendation comes with the caveat that improved simulators may perform better. LLM simulation may be preferable when the target domain cannot be easily captured by a relational schema (e.g., open-ended dialogue, creative collaboration, multi-agent negotiation) and when perfect state consistency is less critical than interaction diversity.

• Prefer real-world APIs or human-authored environments when: (a) the target deployment involves specific production APIs where the exact API behavior, error modes, and edge cases must be learned — synthetic approximations will miss idiosyncratic behaviors, (b) regulatory or safety constraints require training on the actual production system (e.g., healthcare, finance with real compliance requirements), or (c) the number of distinct environments needed is small (2–5) and overfitting is acceptable because the agent will only ever interact with those specific environments.

• Prefer a hybrid approach when: you have a small number of real environments (3–5) but need to avoid overfitting — train primarily on AWM's 1,000 synthetic environments, then fine-tune on the real environments. The paper doesn't test this hybrid, but the generalization results (consistent improvement across three benchmarks with no overlap with training environments) suggest the synthetic pre-training provides a strong initialization that would reduce the amount of real-environment interaction needed.