Scaling Agent Learning via Experience Synthesis

ArXiv: 2511.03773

🎯 Pitch

DreamGym replaces expensive real-environment rollouts with a reasoning-based experience model that synthesizes coherent state transitions and reward signals, achieving a >30% improvement on WebArena over all baselines where conventional RL fails due to sparse rewards. Remarkably, its sim-to-real variant yields a 40% performance gain while using less than 10% of real-world interactions, proving that structured, reasoning-rich synthetic experience can nearly eliminate the need for costly environment interaction in agent training.

---

1. Executive Summary

This paper introduces DreamGym, the first unified framework that synthesizes diverse, reasoning-driven experiences to enable scalable online reinforcement learning for LLM agents without relying on costly real-environment rollouts. Evaluated across WebShop, ALFWorld, and the non-RL-ready WebArena benchmarks using Llama-3.2-3B, Llama-3.1-8B, and Qwen-2.5-7B backbones, DreamGym combines a reasoning-based experience model (distilling environment dynamics into CoT-driven state transitions and reward signals), an experience replay buffer (grounding synthetic rollouts in offline data co-evolving with the agent's policy), and a curriculum task generator (producing progressively harder task variations selected via reward-entropy heuristics). On WebArena—where RL baselines fail due to infrastructure and reward sparsity—DreamGym outperforms all methods by over 30%, while on RL-ready environments it matches GRPO and PPO trained on 80K real interactions using purely synthetic data, and its sim-to-real variant yields an additional 40% improvement while requiring less than 10% of real-world interactions, establishing that structured, reasoning-rich synthetic experience can substitute for costly environment interaction only when the experience model preserves domain-consistent transitions and accurate reward signals rather than pixel-level fidelity.

2. Context and Motivation

The Core Problem: RL for LLM Agents Is Fundamentally Bottlenecked by Environment Interaction

The central challenge this paper confronts is both simple to state and enormously consequential in practice: reinforcement learning needs massive amounts of diverse, high-quality interaction data to train capable agents, but the real environments where these agents must ultimately perform are stubbornly resistant to providing it. The paper identifies four interlocking barriers that collectively make conventional online RL for LLM agents—where the agent learns by interacting directly with its target environment—prohibitively expensive, unreliable, or simply impossible at scale.

This matters because autonomous agents built on large language models are being deployed across an increasingly broad range of interactive domains: web navigation (Zhou et al.), embodied control in 3D environments (Shridhar et al.), multi-turn tool orchestration (Yao et al., 2024), and operating-system control (Xie et al., 2024b). While these agents benefit from powerful language priors acquired during pretraining, their performance in downstream interactive settings remains limited when they must execute long chains of context-dependent decisions (Wang et al., 2024). RL offers a theoretically compelling path forward—agents can bootstrap from their own experience, learning which actions work and which fail through trial and error—but the practical infrastructure for collecting that experience is broken in four distinct ways:

• Prohibitive cost and low sample efficiency of real rollouts. Real environments involve long interaction sequences (often 5–15 steps per task), high computational cost per step (browser rendering, physics simulation, API calls), and sparse reward feedback (success/failure only at trajectory's end). Modern RL algorithms like PPO and GRPO need thousands to hundreds of thousands of trajectories to converge, making the total cost of real-world data collection enormous. The paper's experiments quantify this directly: training agents on WebArena requires operating multiple AWS-hosted server instances with manual resets, with the authors managing only four parallel interaction sessions despite extensive engineering effort (Appendix A.3). This throughput bottleneck means that even well-engineered RL pipelines cannot generate enough diverse experiences to drive stable policy improvement.

• Lack of diverse, scalable task suites. Most existing agent environments provide only a limited, static set of instructions—WebArena offers 812 tasks, ALFWorld provides 3,553 training tasks across six families, and WebShop contains roughly 12,000 crowd-sourced instructions. While these numbers may seem adequate at first glance, RL training requires a broad range of tasks for effective exploration and generalization (Eysenbach et al., 2018). An agent that only sees the same few hundred instructions will overfit to narrow strategies rather than acquiring generalizable skills. Moreover, expanding task collections is itself expensive: verifying that a new instruction is actually feasible in the target environment typically demands costly human expertise (Xue et al., 2025), since a poorly designed task might be unsolvable, ambiguous, or reward-hacking-prone. This creates a frustrating circular dependency—you need diverse tasks to train general agents, but generating those tasks requires the very expertise you're trying to automate.

• Unstable and unreliable reward signals. Many realistic interactive settings—web pages, GUIs, dynamic APIs—are highly non-stationary. Content changes, layouts shift, and behaviors that worked yesterday may fail today. This produces noisy, sparse, or even incorrect reward feedback that confounds credit assignment (Deng et al., 2023). When an agent takes an action that should succeed but the environment returns a failure signal due to a transient rendering issue or a race condition, the policy update propagates misleading gradients that can destabilize training. The paper's WebArena baselines exhibit exactly this pathology: "some trajectories fail to execute properly and certain tasks are incorrectly judged by WebArena's original evaluation function, a known issue also reported in prior work" (Appendix A.3). Beyond noise, safety concerns compound the problem—certain actions are irreversible in real environments (deleting a file, purchasing an item, posting content), and most systems lack reliable reset mechanisms (Zhou et al.), making large-scale exploration practically and ethically risky.

• Engineering complexity of RL-ready infrastructure. Existing agent environments are heterogeneous and often rely on heavyweight backends—Docker containers (Jimenez et al.), virtual machines (Xie et al., 2024b), or real web servers—that make large-batch rollout sampling engineering-intensive, slow, and fragile. The WebArena experience is instructive: even after implementing a "vanilla RL pipeline" following standard workflows, the authors could only operate four parallel AWS servers, requiring manual restarts and environment resets between training iterations (Appendix A.3). This is not a problem that better algorithms can solve; it is a fundamental mismatch between the throughput demands of modern RL and the infrastructure realities of realistic agent environments.

Together, these four barriers create a situation where RL for LLM agents is simultaneously the most promising path to robust, adaptive behavior and the least practical to implement at scale. This tension—the gap between RL's theoretical promise and its practical intractability—is the paper's motivating problem.

Why This Matters: The Broader Stakes for Agent Development

The significance of this problem extends well beyond academic benchmarking. As Silver and Sutton (2025) argue in their framing of the "era of experience," the next major advances in AI capability will come not from scaling pretraining on static datasets but from systems that learn continuously through interaction with their environments. For LLM agents specifically, RL represents the primary mechanism for closing the gap between language understanding and grounded decision-making—converting the broad but shallow knowledge acquired during pretraining into the specific, context-sensitive behavioral policies needed to navigate websites, manipulate tools, and coordinate multi-step plans.

If RL remains bottlenecked by environment interaction costs, the practical consequence is that agent development will remain stuck in one of two unsatisfying regimes: either (a) relying on static imitation learning from expensive human demonstrations, which produces brittle policies that cannot adapt to novel situations, or (b) deploying agents that are essentially glorified few-shot prompters—capable of impressive individual steps but unable to learn from their mistakes over time. The paper's vision is that solving the environment bottleneck would unlock a third regime: agents that improve continuously through self-generated experience, bootstrapping toward increasingly competent behavior without proportional increases in human supervision or infrastructure cost.

There is also a theoretical dimension to the problem's importance. The paper is not merely proposing an engineering fix; it is making a claim about what kind of experience is sufficient for learning. If an agent can acquire effective policies from synthetic rollouts that differ substantially from real environment dynamics, this challenges the implicit assumption that faithful simulation is a prerequisite for transfer. The paper's theoretical analysis in Appendix B.1 formalizes this: policy improvement in the real environment depends on reward accuracy and domain consistency (bounded by ε_R and ε_P in Theorem 1), not on pixel-level state reconstruction error. This is a conceptual reframing of what "environment fidelity" means for agent learning, with implications for how future training systems should be designed.

Prior Approaches and Where They Fall Short

The paper organizes prior work along two axes: approaches that attempt to train agents without synthetic environments (battling directly against the interaction bottleneck) and approaches that build synthetic environments (but with limitations that prevent them from serving as general-purpose RL substrates).

Training Without Synthetic Environments: The Imitation Learning Bottleneck and Static Trajectory Generation

The traditional approach to training LLM agents has been behavioral cloning from expert demonstrations, typically via supervised fine-tuning (SFT). Early work collected oracle trajectories by having humans label correct action sequences or by generating them from stronger teacher models (Yao et al., 2022; Pahuja et al., 2025; Deng et al., 2023). While straightforward, this approach suffers from a fundamental limitation: it produces static, off-policy data that cannot adapt to the agent's evolving policy or explore novel situations not covered by the demonstrations. The agent learns to imitate, not to reason about novel action-outcome contingencies. As the paper's experiments confirm (Table 1, WebArena rows), SFT and DPO—the standard offline methods—consistently underperform online approaches across all backbones, often by large margins (e.g., 6.1% success for SFT on WebArena versus 13.3–14.5% for DreamGym-trained agents).

To reduce the manual effort of collecting demonstrations, subsequent work explored indirect sources of expertise. Synatra (Ou et al., 2024) used online tutorials to guide trajectory synthesis, converting instructional content into agent-executable action sequences. AgentTrek (Xu et al.) similarly leveraged web tutorials as indirect supervision. These approaches reduce human annotation cost but do not escape the fundamental problem: they still produce fixed datasets that cannot provide the on-policy exploration and adaptive difficulty that RL requires. The agent can only learn from what was demonstrated, not from what could be achieved through trial-and-error.

A separate line of work focused on synthesizing diverse tasks rather than diverse trajectories. AgentSynth (Xie et al., 2025) and SCA (Zhou et al., 2025) both aim to expand the task space for RL exploration by generating new instructions. However, these methods still depend on collecting data in real environments—they need to validate that generated tasks are actually solvable, which requires executing trajectories in the target environment to check for feasibility. They therefore "inherently suffer from the scalability issue and are subject to the limitations outlined earlier" (Section 2.2): every new task still requires real-world rollouts at real-world cost.

Building Synthetic Environments: From Pixel-Level World Models to Reasoning-Based Simulators

The idea of replacing real environments with learned simulators has a long history in RL, dating back to Dyna-style architectures (Sutton's integrated planning and learning) and exemplified more recently by world models that learn to predict environment dynamics from raw observations. Dreamer (Hafner et al., 2020) and its successors learn latent dynamics models from pixel inputs, enabling agents to "imagine" future trajectories for planning and policy improvement. However, these approaches were developed for continuous control and video game domains where the state space is reasonably low-dimensional and dynamics are physically grounded. Applying the same paradigm to web environments—where states consist of hundreds or thousands of interactive elements rendered as HTML/DOM trees with complex visual layouts—would require enormous amounts of training data to even approximately capture the diversity of real web pages.

Two recent works come closest to the DreamGym approach. WebDreamer (Gu et al., 2024) constructed a world model for web environments that produces environment feedback for agent planning. However, WebDreamer was designed primarily as a planning tool—generating foresight for action selection—rather than as a training substrate for policy optimization. It does not address the task diversity problem (it operates over a fixed set of instructions) and does not provide the curriculum generation or experience replay mechanisms that DreamGym introduces. The paper's experiments in Section 6.4 directly compare against WebDreamer: while it initially outperforms the Llama-3.1-8B experience model at very small data scales (2K offline samples) due to web-specific pretraining, this advantage disappears as more offline data becomes available, with both models converging to similar performance around 20K samples. This suggests that domain-specific pretraining offers an early advantage but is not a prerequisite for effective experience synthesis.

WebEvolver (Fang et al., 2025) frames web agent training as a co-evolving system where both the agent and a world model improve through interaction. While conceptually aligned with DreamGym's co-evolution philosophy (the experience replay buffer is continuously enriched with fresh interactions), WebEvolver focuses on a narrower setting and does not provide the unified infrastructure—experience model, replay buffer, curriculum generator—that DreamGym packages as a complete RL training framework.

The work most directly comparable in technical approach is UI-Simulator (Wang et al., 2025b), which also leverages LLMs as step-wise simulators for agent training. However, UI-Simulator "requires substantial expert engineering to adapt to different environments and is limited to generating only trajectory variations for supervised fine-tuning of the policy" (Section 2.2). It functions as an SFT data augmentation tool rather than an online RL environment, treating synthetic rollouts as additional static training examples rather than as an interactive training loop where the agent's policy and the experience model co-evolve.

RL for LLM Agents in Real Environments: The Sparsity and Stability Crisis

The paper also evaluates against direct application of RL algorithms to real environments, establishing baselines with GRPO (Shao et al., 2024) and PPO (Schulman et al., 2017) trained on 80K real interactions. On RL-ready environments like WebShop and ALFWorld, these baselines achieve strong performance (e.g., 62.1–66.1% on WebShop with GRPO across backbones), demonstrating that given sufficient computational resources and engineering effort, real-environment RL can work. However, the limitations become apparent in two ways:

• On WebArena, RL baselines simply fail. Despite implementing a "vanilla RL pipeline" with significant engineering effort (managing AWS servers, manual resets, handling evaluation bugs), GRPO and PPO achieve only 4.8–7.3% success across backbones—barely above random behavior. This is the most dramatic illustration of the infrastructure bottleneck: even with the best available tooling, WebArena's combination of high cost, sparse rewards, and infrastructure complexity makes it essentially non-RL-ready.

• Even on RL-ready environments, RL is sample-inefficient. The training curves in Figure 3 (right) show that baseline RL in WebShop exhibits large oscillations caused by sparse or unstable rewards, while DreamGym-trained agents improve more smoothly and rapidly within the first 40 steps. This suggests that even when RL is technically feasible, the quality of the experience data—its consistency, informativeness, and density of learning signal—limits training stability and sample efficiency.

How DreamGym Positions Itself

DreamGym's core positioning claim is that the RL bottleneck for LLM agents is not fundamentally about environment fidelity—it is about the quality and structure of interaction data. The paper reframes the problem from "how do we make real environments cheaper and faster?" to "what properties must synthetic experience possess to enable effective policy learning, and how can we achieve those properties without replicating the real environment?"

This reframing is grounded in the theoretical analysis of Appendix B.1, which proves that policy improvement in the real environment depends on two controllable error terms—reward accuracy (ε_R) and domain consistency (ε_P)—rather than on pixel-level reconstruction fidelity. The experience model design in Section 4.1 operationalizes this insight: by operating in an abstract textual state space (clean element listings rather than raw HTML), using explicit chain-of-thought reasoning to ensure causal grounding, and retrieving relevant past experiences to reduce hallucination, DreamGym aims to make ε_R and ε_P small without attempting to faithfully reproduce every pixel and interaction quirk of the real environment.

The paper further positions DreamGym as a complete RL training framework, not merely a simulator. This is what distinguishes it from WebDreamer (a planning tool) and UI-Simulator (an SFT data augmenter). DreamGym unifies three components that address the full spectrum of RL data needs:

• The reasoning experience model addresses cost and stability: it replaces heterogeneous external environments with a unified, LLM-hosted interaction loop that can scale across parallel instances without infrastructure bottlenecks, providing consistent, causally-grounded transitions and rewards.

• The experience replay buffer addresses data quality: it grounds synthetic rollouts in real-world knowledge (seeded with offline trajectories from the target domain) and co-evolves with the agent, ensuring that generated experiences remain aligned with the agent's current policy rather than becoming stale or irrelevant.

• The curriculum task generator addresses diversity: it identifies tasks where the agent's performance is most variable (high reward entropy, indicating the task is at the edge of the agent's capability) and generates progressively harder variations, creating an adaptive curriculum that pushes the agent beyond its comfort zone without requiring human-designed task expansions.

By packaging these three components together, DreamGym positions itself as a drop-in replacement for real-environment interaction in the RL training loop—not a perfect simulation but a purpose-built experience generator designed to maximize learning per unit of computation. The experiments are designed to test this claim in three regimes: where real RL is impossible (WebArena, proving DreamGym opens previously closed doors), where real RL is possible but expensive (WebShop, ALFWorld, proving DreamGym matches real-environment performance with zero real interaction cost), and where synthetic pretraining combined with limited real fine-tuning yields the best of both worlds (DreamGym-S2R, proving that synthetic experience provides an effective warm-start that makes subsequent real interaction far more sample-efficient).

A final positioning nuance: DreamGym is explicitly algorithm-agnostic. Section 3.2 states that "our proposed DreamGym is orthogonal to specific RL algorithms and focuses on scaling the synthesis of diverse, informative experiences, thereby amplifying the effectiveness of RL training." The framework supports both PPO and GRPO, with the main results evaluating both, and the curriculum task generator handles the group-relative advantage computation of GRPO by using training groups directly, while for PPO it clusters tasks via semantic embeddings to form pseudo-groups. This agnosticism is important because it means DreamGym is not proposing yet another RL algorithm variant but rather a data infrastructure layer that any policy gradient method can plug into.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

DreamGym is a synthetic environment framework that wraps a large language model into a multi-turn simulator capable of playing the role of any target environment (e-commerce websites, embodied household scenes, enterprise web applications) by predicting consistent next states and reward signals from agent actions through explicit chain-of-thought reasoning. It solves the problem that training LLM agents via reinforcement learning normally requires hundreds of thousands of expensive, slow, and fragile interactions with real environments by replacing those real rollouts with a controllable, scalable, reasoning-driven experience generator that produces interaction data sufficiently informative and causally grounded to drive effective policy improvement—enabling RL training in domains where it was previously impossible (WebArena) and matching real-environment RL performance in domains where it is merely expensive (WebShop, ALFWorld) without ever touching the real environment during training.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five interconnected components that operate in a continuous cycle:

1. Reasoning Experience Model (M_exp): A fine-tuned LLM that serves as the environment surrogate. Given the current task instruction, interaction history, similar past experiences retrieved from the replay buffer, and the agent's chosen action, it produces a chain-of-thought reasoning trace explaining why that action should lead to a particular outcome, then generates the next textual state and reward signal. This is the core replacement for external environments.

2. Experience Replay Buffer: A memory store seeded with offline real-world trajectory data and continuously enriched with trajectories generated during training. At each agent step, the experience model retrieves the top-k semantically similar state-action pairs from this buffer to ground its predictions and reduce hallucination. The buffer co-evolves with the agent: as the policy changes, new trajectories are added, keeping the retrieved demonstrations aligned with the current policy's behavior.

3. Curriculum Task Generator: A task proposal mechanism (sharing parameters with M_exp) that identifies high-value tasks from the current set using a reward-entropy heuristic—tasks where the agent sees both successes and failures within a group of rollouts—and generates progressively more challenging variations of those tasks to push the agent beyond its current capability ceiling.

4. Agent Policy (π_θ): The LLM-based agent being trained, parameterized by θ, which maps observed states to actions. It can be any model family and size; the experiments use Llama-3.2-3B, Llama-3.1-8B, and Qwen-2.5-7B.

5. RL Algorithm Wrapper (PPO or GRPO): A standard policy gradient optimizer that consumes the synthetic rollout data—state-action-reward sequences—and updates the agent policy parameters. DreamGym is algorithm-agnostic; the same synthetic experiences serve both PPO (with learned value functions and GAE) and GRPO (with group-relative advantage normalization).

The cycle operates as follows: given a set of seed tasks, the curriculum generator produces an expanded task set → for each task, the agent and experience model alternate over multiple turns (agent selects action from state, experience model reasons about the action and produces next state + reward) → collected trajectories are fed to the RL algorithm to update π_θ → the replay buffer is updated with fresh trajectories → the task generator identifies high-entropy tasks from the updated policy's performance and generates new variations → the cycle repeats.

3.3 Roadmap for the Deep Dive

• First, the motivation and theoretical foundation (Theorem 1 and Lemma 1 from Appendix B.1), which establish why a reasoning-based experience model can provably guarantee policy improvement in the real environment—this provides the design principles that drive every architectural choice.

• Second, the reasoning experience model (M_exp): how it operates at inference time to produce multi-turn rollouts (the forward pass), how it is trained from limited offline data with reasoning trace annotations, and why abstract textual state representations are sufficient.

• Third, the experience replay buffer: how offline seeding, semantic retrieval, and online co-evolution work together to ground synthetic transitions in real-world knowledge and maintain alignment with the agent's changing policy.

• Fourth, the curriculum task generator: the reward-entropy heuristic that identifies valuable tasks, the variation generation mechanism, and the λ hyperparameter that balances synthetic and original task proportions.

• Fifth, the RL integration: how the collected synthetic trajectories feed into PPO and GRPO, including the difference in how groups are formed for GRPO's relative advantage computation versus PPO's clustering-based approach.

• Sixth, the sim-to-real transfer mechanism (DreamGym-S2R): how pre-training in the synthetic environment provides a warm-start that dramatically reduces the real-world interaction budget needed for fine-tuning.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems-building paper whose core idea is that RL for LLM agents can be made practical by replacing real-environment rollouts with a purpose-built synthetic experience generator that optimizes for the properties of interaction data that matter for learning (causal consistency, reward accuracy, domain plausibility) rather than striving for pixel-level simulation fidelity. The paper provides both a theoretical framework proving that such synthetic training is sound and a complete engineering implementation demonstrating its effectiveness across multiple environments and model families.

---

3.4.1 Theoretical Foundation: When and Why Synthetic Training Works (Appendix B.1)

Before designing the synthetic environment, the paper establishes formal conditions under which a policy optimized in a synthetic MDP is guaranteed to also improve in the real MDP. This theoretical result is not merely decorative—it directly motivates the key design choices in the experience model architecture.

Setup. Let the real environment be a Markov Decision Process M = (S, A, P, R, γ), where S is the state space (in DreamGym, this is the abstract textual representation, not raw observations), A is the action space, P(s_{t+1}|s_t, a_t) is the transition dynamics, R(s_t, a_t) is the reward function, and γ ∈ (0,1) is the discount factor. Let the synthetic environment induced by the experience model M_exp be M̂ = (S, A, P̂, R̂, γ), where P̂ and R̂ are the model's predicted transitions and rewards. The critical design choice is that both environments share the same state and action spaces S and A—this is what enables seamless transfer of the trained policy.

Define two error terms that capture how well the synthetic environment preserves learning-relevant properties of the real environment:

$\varepsilon_R = \sup_{s,a} |R(s,a) - \hat{R}(s,a)|$

where R(s,a) is the true reward function, R̂(s,a) is the experience model's predicted reward, and the supremum is taken over all state-action pairs.

What it computes: the maximum absolute difference between real and synthetic reward signals across the entire state-action space—essentially, the worst-case reward hallucination of the experience model.

Why this form: using the supremum (worst-case bound) rather than an expectation gives a uniform guarantee that holds for all state-action pairs the agent might encounter, not just typical ones. This is important because an RL agent will systematically explore the state space, including rare or off-distribution regions.

$\varepsilon_P = \sup_{s,a} \text{TV}\left(P(\cdot|s,a), \hat{P}(\cdot|s,a)\right)$

where P(·|s,a) is the true distribution over next states given state s and action a, P̂(·|s,a) is the experience model's predicted distribution over next states, and TV denotes total variation distance (half the ℓ₁ distance between the two probability distributions).

What it computes: the maximum total-variation distance between real and synthetic transition dynamics—how much the experience model's state predictions differ in distribution from what would actually happen in the real environment.

Why this form: TV distance is the standard metric for bounding differences in expected values under different dynamics, which is exactly what we need when comparing cumulative rewards under real versus synthetic transitions. Alternatives like KL divergence are asymmetric and can be infinite when supports differ.

Theorem 1 (Policy Improvement in Real Environment via Synthetic Experiences). Let π be the current agent policy, π' be an updated policy obtained by optimizing in the synthetic environment M̂ under a trust-region constraint (i.e., the per-state KL divergence between π' and π is bounded by δ, as enforced by the soft KL penalties in PPO and GRPO). Then the improvement of π' over π in the real environment M is bounded below:

$J_M(\pi') - J_M(\pi) \geq \frac{1}{1-\gamma} \mathbb{E}_{s \sim d_{\hat{M}}^{\pi}, a \sim \pi'(\cdot|s)}\left[A_{\hat{M}}^{\pi}(s,a)\right] - \frac{4\gamma}{(1-\gamma)^2} V_{\text{max}} \sqrt{\delta} - 2\left(\frac{\varepsilon_R}{1-\gamma} + \frac{2\gamma R_{\text{max}}}{(1-\gamma)^2} \varepsilon_P\right)$

where J_M(π) is the expected cumulative discounted reward of policy π in environment M (starting from the initial state distribution), d_{M̂}^π is the discounted state visitation distribution under π in the synthetic environment, A_{M̂}^π(s,a) is the advantage function in the synthetic environment, V_max = R_max / (1-γ) is the maximum possible value, R_max is the upper bound on per-step rewards, and δ is the KL divergence trust-region radius.

What it computes: a lower bound on how much the policy improves in the real environment. This lower bound has three terms: (1) the improvement the policy achieves in the synthetic environment itself (the surrogate gain), minus (2) a trust-region penalty that increases with how far the update moved from the previous policy (scaled by √δ), minus (3) a term proportional to the experience model's errors in reward prediction (ε_R) and transition dynamics (ε_P), scaled by the effective horizon 1/(1-γ) and 1/(1-γ)² respectively.

Why this form: the decomposition reveals exactly what the synthetic environment must get right. The surrogate gain term A_{M̂}^π(s,a) captures whether the update actually improves behavior as judged by the synthetic environment's own reward and dynamics. The trust-region penalty (4γ/(1-γ)²) V_max √δ prevents over-optimistic updates that exploit synthetic-environment quirks (analogous to the KL penalty in standard PPO that prevents the policy from changing too much). The critical term is 2(ε_R/(1-γ) + (2γ R_max/(1-γ)²) ε_P): this quantifies how much the synthetic environment's inaccuracies translate into real-environment performance degradation. The key insight is that the transition error ε_P is multiplied by 1/(1-γ)² while the reward error ε_R is multiplied only by 1/(1-γ)—for long-horizon tasks (γ close to 1), getting the dynamics right matters far more than getting the rewards precisely correct, because small transition errors compound over many steps.

This theorem provides the design principles for the experience model:

1. Reward accuracy (ε_R) must be high but need not be perfect: the 1/(1-γ) scaling means that even modest reward errors are tolerable, especially since DreamGym uses outcome-based sparse rewards (r = 1 only at successful task completion, r = 0 otherwise), which simplifies the reward prediction problem to a binary classification at trajectory's end.

2. Transition consistency (ε_P) is the critical bottleneck: the 1/(1-γ)² scaling means that for tasks requiring ~10 steps (effective horizon ~10, γ ~ 0.9), transition errors are penalized roughly 10× more heavily than reward errors. This motivates DreamGym's focus on causal reasoning, history conditioning, and replay buffer retrieval—all designed to minimize ε_P.

3. Faithful simulation is not required—domain-consistent, causally-grounded simulation is: neither ε_R nor ε_P penalizes deviations in raw observation features (pixels, HTML structure, visual layout). They only penalize deviations that affect the probability distribution over abstract states and the correctness of reward signals. This justifies DreamGym's design choice to operate in an abstract textual state space rather than attempting to reproduce raw environment outputs.

Lemma 1 (Multi-turn experience synthesis error bound) establishes the intermediate result that the gap between a policy's value in the real and synthetic environments is bounded by:

$\Delta_{\text{model}} := \frac{\varepsilon_R}{1-\gamma} + \frac{2\gamma R_{\text{max}}}{(1-\gamma)^2} \varepsilon_P$

where the notation is as in Theorem 1.

What it computes: the maximum possible difference between the expected cumulative reward of any policy in the real environment versus the synthetic environment, expressed purely in terms of the one-step reward and transition errors ε_R and ε_P.

Why this form: the proof uses the contraction property of the Bellman operator—the value function gap at any state propagates forward, scaling with ε_P over each step due to mismatch in state distributions, and ε_R at each step due to mismatch in immediate rewards. The 1/(1-γ) factor on ε_R arises because reward mismatch at each of the 1/(1-γ) effective steps accumulates linearly, while the 1/(1-γ)² factor on ε_P arises because a transition mismatch at one step shifts the state distribution for all subsequent steps, creating a compounding effect.

The practical import of Lemma 1 is that DreamGym's design does not need to minimize some hard-to-define "realism" metric—it needs to minimize two well-defined, measurable quantities: reward prediction error and total-variation distance between synthetic and real state distributions. The chain-of-thought reasoning, history conditioning, and replay buffer retrieval are all mechanisms for driving down ε_P; the outcome-based reward scheme simplifies ε_R to near-zero for the sparse-reward case.

---

3.4.2 Reasoning Experience Model (M_exp): Inference-Time Operation

The experience model M_exp is the core component that replaces external environments. At inference time, it operates as a turn-by-turn simulator: given the current state of the interaction, the agent selects an action, and M_exp predicts what state and reward should follow.

Abstract state representation. A critical design decision is that M_exp operates not on raw environment observations (HTML source, rendered pixels, JSON API responses) but on an abstract textual state space S. The paper provides a concrete illustration in Section 4.1 and Figure 6: in a WebArena task, instead of processing the full accessibility tree with raw HTML tags, CSS classes, and JavaScript attributes, the state is compressed to clean element listings such as [227] link:'My Account' link to user..., [1238] menuitem:'Grocery & Food ...', [1474] table:Orders table show the.... Each element is identified by a numeric ID and a semantic type, discarding irrelevant structural artifacts like deeply nested <div> hierarchies, style information, and non-interactive DOM nodes.

The rationale is twofold. Efficiency: abstract states are far more token-efficient, reducing the context window usage per step and enabling longer interaction sequences within the LLM's context limit. Learning signal quality: abstract states strip away irrelevant variation (different websites use different CSS frameworks, different DOM structures, different visual layouts) and preserve only the information that matters for decision-making—what interactive elements exist, what their semantic functions are, and what content they display. This aligns with Theorem 1's implication that transition consistency (ε_P) matters, but only with respect to the distribution over semantically meaningful states, not over raw pixels.

Inference procedure (Equation 4). At step t of a rollout, the experience model receives four types of context and must produce the next state s_{t+1} and reward r_{t+1}:

$(s_{t+1}, r_{t+1}) = M_{\text{exp}}\left(\mathcal{R}_t \mid \{(s_i, a_i)\}_{i=0}^t, \{d_j\}_{j=1}^k, \tau\right)$

where {(s_i, a_i)}_{i=0}^t is the complete interaction history from the initial state through the current action, {d_j}_{j=1}^k are the top-k demonstrations retrieved from the experience replay buffer based on semantic similarity to the current state-action pair, τ is the task instruction (the goal the agent is trying to accomplish), and R_t is an explicit chain-of-thought reasoning trace that M_exp generates internally before producing s_{t+1} and r_{t+1}.

What it computes: a next-state prediction and reward signal derived through explicit reasoning rather than through black-box next-token prediction. The model first generates R_t, which is a step-by-step analysis of whether the agent's action is valid, whether it advances the task, and what the environment's response should be. This reasoning trace then conditions the generation of s_{t+1} and r_{t+1}, ensuring that the state transition is causally grounded in an interpretable rationale rather than produced from opaque pattern matching.

Why this form: the separation of reasoning from state prediction is the paper's key architectural insight. A naive approach would directly fine-tune an LLM to map (history, action) → next_state without explicit reasoning, effectively learning a black-box world model. The paper shows in Section 6.3 (Figure 4) that removing reasoning ("w/o Reasoning") substantially increases hallucination and reduces informativeness of generated states. The reasoning trace serves multiple functions: (a) it forces the model to explicitly consider action validity (should click(463) take the agent to a commit history page, or is that element not clickable in the current context?), (b) it provides a mechanism for incorporating task awareness (the same action on the same page might be appropriate for one task but not another), and (c) it creates an intermediate computational step that decomposes the complex mapping from (history, action) to next_state into two simpler mappings—reason about consequences, then render consequences—each of which is easier for a language model to learn accurately.

The three required contexts in detail:

1. Interaction history {(s_i, a_i)}_{i=0}^t: including the full trajectory in the context window prevents temporal inconsistencies. If the agent clicked "My Account" three steps ago and is now on a profile page, the experience model must know this to avoid hallucinating that the agent is still on the homepage. The paper's ablation (Figure 4, "w/o History") confirms that removing history causes a significant drop in "Causal State Consistency"—the model drifts off-topic and breaks causal coherence in multi-step interactions.

2. Task instruction τ: conditioning on the current goal enables M_exp to interpret actions relative to task objectives. A click(1144) on a commit list entry might be part of "find the first commit in April 2023" or might be irrelevant to "list all repository contributors"—the experience model must know which is the case to predict whether this action advances toward success or is a dead end. This task-awareness is what allows the synthetic environment to provide meaningful learning signals: when the agent takes an action that moves away from the goal, M_exp can generate a state transition that reflects this (e.g., navigating to an irrelevant page, presenting information that doesn't answer the query) and produce a zero reward at the trajectory's end.

3. Past experiences {d_j}_{j=1}^k: the top-k demonstrations retrieved from the replay buffer serve as few-shot examples that ground the experience model's predictions in real-world knowledge. The retrieval uses cosine similarity between an embedding of the current state-action pair and embeddings of stored transitions:

$\{d_j\}_{j=1}^k = \text{Top}_k\left(\cos\left(\phi(s_t, a_t), \phi(s_i, a_i)\right)\right)$

where φ(·) is an arbitrary semantic encoder that maps state-action pairs to vector representations, cos(·,·) is cosine similarity, and Top_k selects the k stored transitions with highest similarity scores.

What it computes: for the current (s_t, a_t), find the k most similar transitions in the replay buffer, ranked by cosine similarity of their embeddings.

Why this form: cosine similarity with embeddings provides a semantic (rather than surface-level) notion of similarity—two state-action pairs are similar if they involve analogous situations (e.g., clicking a navigation link, searching with a vague query, filling a form field) even if the literal text differs. This is important because exact text matching would rarely retrieve useful demonstrations: the same button might have different text on different pages, but the semantic action type "navigate to account settings" is consistent. The paper does not specify the embedding model used, but typical choices would be a sentence-transformer or the base LLM's own hidden states.

Reasoning trace generation. The reasoning trace R_t is a chain-of-thought produced by the experience model before generating s_{t+1} and r_{t+1}. Figure 6 provides a concrete example from a WebArena trajectory:

"The agent clicks the 'Total Commits' button on the overview page, which is the intended action. I should transit to a successful state that displays a list of commits grouped by date, including entries from April 2023, which will let the agent further inspect and ..."

This reasoning process explicitly considers: (a) whether the action is valid in the current context, (b) what the action is intended to accomplish relative to the task, (c) what state transition logically follows, and (d) whether the transition moves toward or away from task completion. The reasoning trace is generated first, then conditions the subsequent token generation for s_{t+1} and r_{t+1}.

Reward scheme. The paper adopts an outcome-based sparse reward structure following Feng et al. (2025): r = 1 only at the final step of a trajectory when the task is successfully completed, and r = 0 at all other steps. This means the experience model only needs to predict a binary success/failure signal at trajectory termination, which dramatically simplifies the reward prediction problem—the model does not need to produce shaped rewards (partial credit for progress) that would require finer-grained judgment and could introduce training instability if miscalibrated.

Why this design over alternatives:

• Versus building a world model in raw observation space (like Dreamer applied to web environments): raw-space world models would need to predict the full pixel content or full HTML of the next page, requiring enormous amounts of training data and model capacity to capture the diversity of real web pages. The abstract state space collapses this diversity into a manageable vocabulary of element types, actions, and semantic relationships.

• Versus using retrieval alone without reasoning (retrieve similar past transitions and directly apply their next state): retrieval without reasoning would be a form of nearest-neighbor regression in state space, which would fail when the agent takes novel actions not well-represented in the replay buffer—the core use case in RL training is exploration, which by definition involves actions and states not seen in offline data.

• Versus using an LLM to generate states directly without retrieved demonstrations: the paper's ablation (Table 2, "w/o Exp. Replay") shows that removing the replay buffer causes a 4.7% drop on WebShop and a 3.6% drop on WebArena. The demonstrations provide factual grounding that reduces hallucination for knowledge-intensive predictions (e.g., what specific products exist in a particular category, what the layout of a real website looks like).

---

3.4.3 Training the Experience Model (Equation 5)

The experience model is trained via supervised fine-tuning (SFT) on a dataset of offline trajectories augmented with reasoning traces. The training is designed to be highly sample-efficient—the paper reports that competitive performance can be achieved with as few as 2,000–10,000 transition steps (Figure 5), which is orders of magnitude less than what would be needed to train a raw-observation world model.

Dataset construction. For each target environment, the authors collect a dataset D = {(s_t, a_t, s_{t+1}, r_{t+1})} of trajectories from the real environment using a combination of sources:

• Expert demonstrations: Successful trajectories from human demonstrations or high-performing agents. For WebShop: 1,600 human demonstration trajectories from the official repository plus 2,000 trajectories from an oracle agent and random exploration. For ALFWorld: 3,200 expert demonstration trajectories plus 2,000 trajectories from oracle and random policies. For WebArena: successful demonstrations from top-performing agents on the public leaderboard (IBM CUGA, ScribeAgent, Learn-by-Interact, AgentOccam), supplemented with trajectories from a high-performing agent and a random policy, totaling 4,800 trajectories (Appendix A.3).

• Augmentation with reasoning traces: Each transition (s_t, a_t, s_{t+1}, r_{t+1}) is augmented with an explicit reasoning trace R*_t generated by a "strong teacher LLM" (the paper does not specify which, but the prompts in Appendix C.1 indicate it is a powerful model tasked with analyzing environment dynamics). The reasoning trace explains:

"why the action a_t taken in state s_t consequently leads to the next state s_{t+1} and reward r_{t+1} given the available contexts" (Section 4.1.2)

This reasoning includes analysis of whether the action is likely to succeed or fail based on the task, a description of how the environment should respond, and what critical information the next state should contain. The prompt template for annotation (Appendix C.1) asks the teacher LLM to produce both a "task tutorial" (high-level guidance for the entire task's state progression) and per-step "state transition plans."

Training objective. The model is trained with a joint SFT loss over both reasoning generation and next-state prediction:

$\mathcal{L}_{\text{SFT}} = \mathbb{E}_{(s_t, a_t, s_{t+1}, \mathcal{R}_t^*) \sim \mathcal{D}} \left[ -\log P_\theta(\mathcal{R}_t^* \mid s_t, a_t, H_t, D_k) - \log P_\theta(s_{t+1} \mid s_t, a_t, \mathcal{R}_t^*, H_t, D_k) \right]$

where D is the offline trajectory dataset, R*_t is the teacher-generated reasoning trace for the transition from step t to t+1, H_t denotes the interaction history up to step t, D_k denotes the top-k retrieved demonstrations, and θ denotes the parameters of M_exp.

What it computes: the sum of two negative log-likelihood terms. The first term -log P_θ(R*_t | s_t, a_t, H_t, D_k) is the standard next-token prediction loss for generating the reasoning trace: given the current state, action, history, and retrieved demonstrations, how likely is the model to produce the teacher's reasoning? The second term -log P_θ(s_{t+1} | s_t, a_t, R*_t, H_t, D_k) is the next-state prediction loss: given all the same context plus the reasoning trace, how likely is the model to produce the correct next state? The expectation is taken over the training data distribution.

Why this form: the joint objective with two terms—rather than a single term predicting (R*_t, s_{t+1}) jointly—enforces a causal structure: the model must learn to generate faithful reasoning traces that actually help predict the next state. If the reasoning trace were irrelevant to state prediction, the model could learn to generate plausible-sounding but causally uninformative reasoning in the first term while relying on memorized patterns for the second term. By conditioning the second term on the reasoning trace, the objective ensures that the reasoning must capture information that is actually predictive of the state transition. This is a form of explain-then-predict training that has been shown to improve the faithfulness of generated explanations in other domains.

The training hyperparameters for the experience model are not explicitly stated in the main paper, but the base model is Llama-3.1-8B-Instruct (stated in Section 5.1), fine-tuned via SFT. The paper emphasizes that "abundant offline trajectory datasets from public benchmarks such as the WebArena Leaderboard are sufficient for training," and Figure 5 demonstrates that performance saturates around 20K offline samples, suggesting that the model does not require scaling to massive offline datasets.

Key design choices in training:

• Using a strong teacher LLM to annotate reasoning traces, not human annotators. This leverages the scalability of LLM-based annotation while avoiding the cost and inconsistency of human reasoning annotation. The quality of the teacher's reasoning traces is critical—if the teacher produces incorrect or causally flawed reasoning, the experience model will learn to replicate those flaws. The paper does not evaluate the teacher's reasoning accuracy directly, but the downstream RL results suggest it is sufficient.

• Including retrieved demonstrations D_k in the context during training. This means the model learns to use retrieved demonstrations effectively—it develops an internal mechanism for integrating retrieved knowledge with its own reasoning. During inference, the same retrieval mechanism is used, so the training distribution matches the inference distribution.

• Training only on successful trajectory data plus random exploration. The dataset includes both successful trajectories (where the agent reached the correct answer) and random/exploratory trajectories (where the agent may have taken suboptimal or incorrect actions). This diversity is important because during RL training, the agent will frequently take suboptimal actions, and the experience model must be able to predict the consequences of those actions (e.g., navigating to irrelevant pages, applying incorrect filters, selecting wrong products) rather than only knowing how to respond to optimal behavior.

• The experience model is trained once per environment and not updated during RL training. This is a notable design choice: unlike Dreamer-style world models that are continuously updated with new environment interactions, DreamGym's M_exp is fixed after offline training. The mechanism for adapting to the agent's changing policy is instead implemented through the replay buffer (which is continuously updated) and the curriculum task generator (which produces new tasks). This simplifies the training pipeline and avoids the stability issues that can arise when the world model and policy are trained simultaneously.

---

3.4.4 Experience Replay Buffer: Grounding and Co-Evolution

The experience replay buffer is a memory store that serves two purposes: (1) it grounds the experience model's predictions in real-world knowledge to reduce hallucination, and (2) it co-evolves with the agent to ensure the retrieved demonstrations remain relevant to the current policy's behavior.

Initialization with offline data. The buffer is seeded with trajectories from the same offline dataset used to train the experience model. This provides essential context about the target domain—what kinds of actions are possible, what typical state transitions look like, what successful task completion entails—that the experience model can draw on during its reasoning. The paper states that this offline knowledge provides "essential context" (Section 4), without which the experience model would be operating purely from its pretrained knowledge, which may not be accurate for specific websites or environments.

Online enrichment. During RL training, every trajectory generated by the interaction between the agent and M_exp is added to the replay buffer. This means the buffer starts containing only offline trajectories from expert demonstrations and random exploration, but over time it accumulates synthetic trajectories that reflect the current agent's actual behavior. The paper calls this a "co-evolving" buffer because as the agent's policy changes through RL updates, the distribution of trajectories it generates changes, and the buffer adapts to contain more examples drawn from the updated policy's distribution.

This co-evolution is critical for a subtle reason: the experience model's predictions are conditioned on top-k retrieved demonstrations. If those demonstrations came only from expert trajectories, they would be off-policy—they would show optimal actions leading to successful outcomes, while the agent during training is taking suboptimal actions that may not resemble any expert trajectory in the buffer. The resulting mismatch between what the agent actually does and what the retrieved demonstrations suggest would increase ε_P (transition error) because the model would have no examples of how the environment responds to the agent's specific type of mistakes.

By continuously adding synthetic trajectories, the buffer builds a library of examples showing what happens when the agent takes various actions—both good and bad—in contexts relevant to the current policy. This reduces the gap between the demonstrations the model sees and the actions the agent takes, keeping ε_P small even as the policy changes.

Retrieval mechanism. At each step, the experience model retrieves the top-k most similar state-action pairs from the buffer:

$\{d_j\}_{j=1}^k = \text{Top}_k\left(\cos\left(\phi(s_t, a_t), \phi(s_i, a_i)\right)\right)$

The retrieved demonstrations serve as few-shot examples in the experience model's context window, showing how similar situations played out in past trajectories (both offline and previously synthesized). This retrieval-based grounding is what the paper's ablation in Table 2 ("w/o Exp. Replay") shows is worth 4.7% on WebShop and 3.6% on WebArena.

Why a replay buffer rather than fine-tuning the experience model online. An alternative design would be to periodically fine-tune M_exp on the accumulating synthetic trajectories, similar to how Dyna-style architectures update the world model. The paper opts for the replay buffer instead, likely for three practical reasons: (a) fine-tuning an 8B-parameter LLM at each RL iteration would be computationally prohibitive, (b) retrieval provides a more direct and interpretable mechanism for conditioning on relevant examples than implicit parameter updates, and (c) avoiding online fine-tuning of the experience model prevents a potentially unstable feedback loop where errors in the experience model compound through self-training.

---

3.4.5 Curriculum Task Generator: Adaptive Difficulty via Reward Entropy

The third major component addresses the task diversity problem: RL needs a broad range of tasks for effective exploration, but manually designing and validating new tasks is expensive. DreamGym's solution is to have the experience model itself generate new task variations, selecting for tasks that are at the edge of the current agent's capability.

Task generation mechanism (Equation 6). The task generator shares parameters with the experience model M_exp (i.e., it is the same LLM with a different prompting context):

$\tau_t = M_{\text{task}}(\{\tau_{t-1}^i\}_{i=1}^m)$

where {τ_{t-1}^i}_{i=1}^m is a set of m seed tasks from the previous iteration, and τ_t denotes the newly generated task variations.

What it computes: given a set of existing tasks, generate new task instructions that are variations of the seeds—preserving the same general action type (search, filter, navigate, purchase) but changing the target entities, constraints, context, or required precision.

Why this form: generating variations from seed tasks rather than generating tasks from scratch anchors the generated tasks in the distribution of realistic tasks. A task like "Find the most expensive three dining tables under $500" is semantically meaningful because it mirrors the structure of real shopping tasks; a model generating entirely free-form instructions might produce tasks that are syntactically plausible but nonsensical in the target environment (e.g., asking the agent to do something the environment doesn't support). By conditioning on seed tasks—which come from the original benchmark and are known to be feasible—the generator is more likely to stay within the bounds of what the synthetic environment can handle.

The prompt templates for task variation generation (Appendices C.1, C.2, C.3) show that the model is asked to produce variations that are "challenging but feasible," "clear, grammatical, and realistic," and involve "meaningful differences" in entities, filters, or constraints. For WebArena, a separate selection prompt (Appendix C.3) then filters the generated variations to pick the highest-quality one based on those criteria.

Selecting which tasks to vary: the reward-entropy heuristic (Equation 7). Not all tasks are equally valuable for training. The paper introduces a group-based reward entropy metric to identify tasks that are most informative for the current policy:

$V_\tau = \frac{1}{n} \sum_{i=1}^n (r_i - \bar{r})^2, \quad \text{where} \quad \bar{r} = \frac{1}{n} \sum_{i=1}^n r_i$

where V_τ is the value of task τ, n is the number of rollouts sampled for this task within a group G, r_i is the outcome reward (0 or 1) for the i-th rollout, and r̄ is the mean reward across the group.

What it computes: the sample variance of outcome rewards across n rollouts of the same task. Since rewards are binary (0 for failure, 1 for success), this is equivalent to computing the variance of a Bernoulli random variable, which is maximized when the success probability is 0.5 and minimized when it is 0 or 1. In practical terms: if the agent always succeeds on a task (V_τ = 0), the task is too easy and provides no learning signal (no opportunity for the policy to improve). If the agent always fails (V_τ = 0), the task is either impossible or too hard given the current policy, providing no gradient for credit assignment (the agent cannot learn what distinguishes successful from unsuccessful actions since it never succeeds). Maximum variance occurs when the agent sometimes succeeds and sometimes fails, indicating that the task is at the edge of the agent's current capability—small differences in action choices determine the outcome, making this the regime where RL can most effectively assign credit and improve the policy.

Why this form: the variance-of-outcomes heuristic is a computationally cheap proxy for the information-theoretic value of a task for learning. It requires only sampling n rollouts and counting successes—no additional model evaluations, no reward shaping, no difficulty estimation. The paper draws an explicit connection to recent findings that "LLMs learn most effectively from tasks of intermediate difficulty" (Gao et al., 2025), where tasks at the boundary of capability provide the strongest gradient signal. Alternative heuristics—such as using the magnitude of the policy gradient or the KL divergence between the current and previous policy on the task—would require more computation and would be algorithm-specific; reward entropy works for any RL method.

Forming groups for GRPO and PPO. The group concept is algorithm-dependent:

• For GRPO, the group G is naturally provided by the algorithm itself: GRPO samples a group of n responses (in this case, trajectories) for each task instruction and normalizes advantages within the group. The reward entropy for a task can therefore be computed directly from the GRPO training batch without additional sampling.

• For PPO, which does not have a built-in grouping mechanism, the paper clusters tasks using a semantic embedder—each cluster forms a pseudo-group G, and task variations are generated from high-entropy tasks within each cluster. This ensures that the task generator operates on tasks that are grouped by semantic similarity, preventing an imbalance where the reward entropy of a shopping task and a navigation task are compared directly even though they have different baseline success rates.

Curriculum generation cycle. After each RL iteration, the process is:

1. Compute V_τ for all tasks using rollouts from the current policy.
2. Select tasks with highest reward entropy as seeds (the exact threshold or selection procedure is not specified, but the principle is to prioritize tasks at the boundary of the agent's capability).
3. Feed these seed tasks to M_task, which generates new variations. The paper states that the variations are "progressively more challenging" but does not specify the mechanism for progressive difficulty increase beyond the prompt instructions asking for tasks that are "slightly more reasoning, precision, or constraints than the original."
4. Add the generated variations to the task pool for the next iteration.

Balancing synthetic and original tasks (the λ hyperparameter). To prevent the task distribution from drifting entirely into synthetic variations that may lose touch with the original task distribution, the paper introduces a hyperparameter λ that bounds the proportion of synthetic tasks sampled per iteration (Section 4.2):

"To stabilize training, we introduce a hyperparameter λ that bounds the proportion of synthetic tasks sampled per iteration. This preserves sufficient coverage of the original task distribution while directing exploration toward the current policy's weakness regions for curriculum-based improvement."

The exact value of λ is not specified in the main text or appendix, but this is a standard technique in curriculum learning to prevent the curriculum from collapsing into a narrow region of task space that happens to have high entropy but does not generalize.

Ablation evidence for the task generator. Table 2 shows that removing the task generator causes a 6.6% drop on WebShop (from 63.9% to 57.3%) and a 6.0% drop on WebArena (from 13.3% to 7.3%). The paper interprets this as evidence that "without adaptive task generation, the replay buffer may saturate with low-entropy, repetitive trajectories, which limits the diversity of experiences and stalls exploration." Figure 3 (right) shows the temporal dimension: without task generation, the agent makes initial progress but then plateaus, while with task generation, learning continues to improve over more training steps. A λ that is too high (too many synthetic tasks too early) or a task generator that produces tasks far outside the original distribution could potentially harm performance, but the paper does not report ablations on these dimensions.

---

3.4.6 RL Integration: Training Policies from Synthetic Experiences

The final piece of the system is the integration with standard policy gradient algorithms. DreamGym produces multi-turn rollouts—sequences of (s_t, a_t, r_{t+1}, s_{t+1}, ...)—that are structurally identical to rollouts from a real environment, making them compatible with any RL algorithm that expects this format.

Rollout collection. For each epoch of training (the paper uses the term "iteration"), the cycle is:

1. Sample a batch of task instructions from the current task pool (a mix of original seed tasks and synthetic variations, controlled by λ).
2. For each task, initialize the rollout by generating an initial state s_0 from the experience model conditioned on the task instruction τ. The exact mechanism for initial state generation is not detailed beyond what is shown in Figure 6, where the experience model produces "generate initial state" as its first action.
3. For each step t = 0, 1, 2, ... until the experience model signals termination (via a "Done: True" flag and a success/failure indicator, as shown in Figure 2):
   • The agent policy π_θ selects an action a_t given the current state s_t.
   • The experience model receives (s_t, a_t), retrieves top-k demonstrations, generates reasoning R_t, and predicts (s_{t+1}, r_{t+1}).
4. The complete trajectory—the sequence of states, actions, and rewards—is added to the data buffer for the RL algorithm and also added to the experience replay buffer for future retrieval.

Policy update with PPO. When using PPO as the RL algorithm, the collected trajectories are used to compute advantages via Generalized Advantage Estimation (GAE), as given in Equation 2:

$\hat{A}^{\text{PPO}}_t = \sum_{l=0}^{K-1} (\gamma \lambda)^l \left[r_{t+l} + \gamma V(s_{t+l+1}) - V(s_{t+l})\right]$

where V(·) is a value function (implemented as an LLM head that predicts the expected future return from a state), γ is the discount factor, λ is the GAE bias-variance tradeoff parameter, and K is the trajectory length. The PPO clipped objective then updates the policy parameters θ using the standard surrogate loss with a KL penalty or clipping to enforce the trust-region constraint.

Policy update with GRPO. When using GRPO (Equation 3), the group-relative advantage is computed without a learned value function:

$\hat{A}^{\text{GRPO}}_t = (r_t - \text{mean}_{i \in G}(r_i)) / \text{std}_{i \in G}(r_i)$

where G is the group of trajectories sampled for the same task instruction, r_t is the outcome reward for the current trajectory, and mean_{i∈G}(r_i) and std_{i∈G}(r_i) are the mean and standard deviation of rewards across all trajectories in the group. The advantage is thus simply how many standard deviations above or below the group mean this trajectory's reward falls.

What it computes: a normalized advantage that measures how much better or worse a particular trajectory's outcome is compared to other trajectories attempting the same task. A positive advantage means the trajectory succeeded while many others failed (or succeeded in fewer steps, though the paper uses outcome-only rewards so this distinction collapses to binary success/failure). A negative advantage means the trajectory failed while many others succeeded.

Why this form: GRPO discards the value function V(·) entirely, using the group mean as a baseline. This makes the algorithm more scalable (no need to train a separate value head) but potentially less sample-efficient because advantage estimates are noisier when group sizes are small. DreamGym's synthetic rollouts mitigate this concern because the experience model can generate arbitrarily many trajectories for any given task at low cost, allowing large group sizes for stable advantage estimation—a luxury that real-environment GRPO does not have due to interaction cost.

Algorithm-agnosticism. The paper evaluates both PPO and GRPO trained entirely within DreamGym (Table 1, "DreamGym" rows), demonstrating that the synthetic experiences are agnostic to the choice of optimization algorithm. On WebShop with Llama-3.1-8B, DreamGym + GRPO achieves 63.9% versus DreamGym + PPO achieving 58.1%, both close to real-environment GRPO (65.0%) and PPO (64.2%). This suggests that the synthetic experiences serve as a general-purpose substrate for policy optimization—the quality of the experience data, not the particular RL algorithm, is the primary determinant of performance.

Training curve analysis (Figure 3, right). The paper compares training curves between DreamGym-trained agents and traditional RL agents on WebShop:

• DreamGym convergence is faster and smoother: success rate improves more rapidly within the first 40 training steps, and curves exhibit lower variance across runs. The paper attributes this to synthesized trajectories offering "more informative gradients than sparse real rollouts" and providing "not only denser but also more consistent feedback."

• Traditional RL exhibits oscillations: baseline curves show "larger oscillations caused by sparse or unstable rewards." This reflects the real-environment challenges of noisy observations, non-deterministic transitions, and occasional evaluation errors discussed in Appendix A.3. DreamGym's controlled, reasoning-driven environment eliminates these sources of instability.

• Without task generation, DreamGym plateaus: Figure 3 (right) directly compares DreamGym with and without the curriculum task generator. The "DreamGym w/o Task" curve rises initially but then flattens, while the full DreamGym continues to improve. This visualizes the claim that the task generator prevents the replay buffer from saturating with low-entropy experiences.

---

3.4.7 Sim-to-Real Transfer (DreamGym-S2R): Warm-Starting Real-Environment RL

The final component is the mechanism for transferring a policy trained purely in the synthetic environment to fine-tuning on real environment interactions.

Motivation. While DreamGym-trained agents can match real-environment RL performance with zero real interactions, the paper shows that combining synthetic pre-training with a small amount of real-environment fine-tuning yields the best results overall—outperforming both pure synthetic training and pure real-environment training from scratch. This is the sim-to-real (S2R) paradigm: use the cheap, scalable synthetic environment to build a strong initialization, then use expensive real interactions only for the final fine-tuning stage where grounding in actual environment dynamics matters most.

State space consistency for transfer. For the policy trained in the abstract textual state space S to function in the real environment, there must be a mapping from real observations to the abstract states the policy was trained on. The paper states:

"To enable seamless transfer, we ensure consistency of the state space between synthetic and real environments by applying the same rule-based mapping function or a lightweight fine-tuned model" (Section 4.3).

For WebArena, Appendix C.3 provides the specific mapping prompt that converts a raw accessibility tree (AXTree) observation into the refined abstract state format. The prompt instructs the mapping model to "extract relevant webpage elements based on your high-level reasoning" and produce "a subset of the webpage observation that contains necessary information for completing the user instruction." This is described as a "lightweight fine-tuned model" that learns to compress raw observations into the same format the experience model generates.

Transfer procedure. The DreamGym-S2R pipeline has two phases:

1. Synthetic pre-training: Train the agent policy π_θ entirely within DreamGym using the full pipeline—experience model rollouts, replay buffer retrieval, curriculum task generation—for a fixed number of iterations (the paper uses the full DreamGym training budget, not an abbreviated version).

2. Real-environment fine-tuning: Transfer the pre-trained policy to the real environment and continue RL training (using the same PPO or GRPO algorithm) for a small number of additional iterations. The paper uses 5,000 real transitions for this phase (Table 1, "DreamGym-S2R" rows), compared to 80,000 for the traditional RL baselines—a 16× reduction in real-world interaction.

Results and interpretation. Table 1 shows that DreamGym-S2R consistently outperforms both pure DreamGym training (synthetic only) and traditional RL (real only) across all environments and backbones:

• On WebShop with Llama-3.1-8B: DreamGym-S2R + GRPO achieves 75.0% versus DreamGym-only at 63.9% and traditional GRPO at 65.0%. The improvement over pure synthetic training is 11.1 percentage points, and over pure real training is 10.0 percentage points.

• On ALFWorld with Qwen-2.5-7B: DreamGym-S2R + GRPO achieves 82.4% versus DreamGym-only at 71.0% and traditional GRPO at 79.8%. The synthetic pre-training provides a 2.6 percentage point advantage over training from scratch in the real environment.

• On WebArena: DreamGym-S2R shows smaller or mixed improvements over pure DreamGym (e.g., Llama-3.1-8B: 9.7% S2R vs 9.1% DreamGym-only for GRPO), likely because the 5,000 real transitions are insufficient to overcome the large domain gap between the abstract state space and WebArena's complex, dynamic web environments. However, even maintaining parity with pure synthetic training while adding real-environment grounding is a positive result.

The paper attributes the S2R improvement to the synthetic pre-training providing "broad knowledge at low cost" and "expanding exploration coverage across diverse tasks," which establishes a strong initialization that makes subsequent real-environment learning more sample-efficient (Section 4.3). The theoretical analysis in Theorem 1 supports this interpretation: if the policy after synthetic pre-training already achieves a reasonable real-environment value J_M(π) (because ε_R and ε_P are small), then even a small amount of real-environment fine-tuning can push it higher by reducing the reliance on synthetic dynamics entirely.

Why this is a "scalable warm-start strategy for general-purpose RL." The critical economic argument is that the most expensive resource—real environment interaction—is used only at the end, after the policy has already acquired competent behavior from cheap synthetic experience. This inverts the traditional RL cost structure, where all learning happens through expensive real rollouts. If the ratio of synthetic pre-training cost to real fine-tuning cost is favorable (as the paper claims, with a 16× reduction in real interactions), then DreamGym-S2R makes RL economically viable for domains where pure real-environment RL would be prohibitively expensive, and potentially opens the door to continuous improvement loops where agents are periodically fine-tuned on small batches of real data between longer periods of synthetic training.

4. Key Insights and Innovations

Innovation 1: Reframing Environment Fidelity from Pixel-Level Simulation to Learning-Centric Signal Quality

The paper's most fundamental conceptual move is redefining what it means for a synthetic environment to be "good enough" for RL training. The dominant assumption in prior work on world models for agent training—from Dreamer (Hafner et al., 2020) through WebDreamer (Gu et al., 2024) and UI-Simulator (Wang et al., 2025b)—has been that simulation quality should be measured by how faithfully the synthetic environment reproduces the raw outputs of the real environment: pixel values, HTML structure, exact page layouts. This assumption is natural—if the synthetic environment looks like the real one, the policy trained in it should transfer—but it creates an enormous practical burden: building a simulator that captures the full diversity of real web pages or embodied scenes is itself an unsolved research problem requiring massive data and model capacity.

DreamGym's core reframing, formalized in Theorem 1 (Appendix B.1), is that policy improvement in the real environment depends on only two controllable error terms—reward accuracy (ε_R) and domain consistency of state transitions (ε_P)—rather than on pixel-level reconstruction fidelity. The proof shows that the gap between a policy's value in the real versus synthetic environment is bounded by a quantity Δ_model = ε_R/(1-γ) + (2γ R_max/(1-γ)²) ε_P, with no term corresponding to "state appearance error" or "perceptual similarity." This means the experience model can compress raw observations into an abstract textual state space, discard irrelevant structural artifacts (HTML tags, CSS classes, visual layout details), and still provably enable policy improvement—as long as the abstract states preserve the semantic content that determines what actions are available, what their consequences are, and whether the task is complete.

What makes this insight distinctive is not the mathematical result itself—trust-region policy improvement bounds with model error are standard in the RL theory literature (Schulman et al., 2015)—but rather the design implication the paper extracts from it: that the right abstraction level for synthetic environments is not the raw observation space but a reasoning-conditioned semantic representation. Prior work on learned world models (Dreamer, WebDreamer) attempted to model transitions in the original observation space, which forced them to learn representations that captured both task-relevant dynamics (what happens when you click "Submit"?) and task-irrelevant appearance variation (what color is the button? what font is used?). DreamGym's theoretical analysis provides formal justification for the counterintuitive claim that an agent can learn effective policies from synthetic rollouts that look nothing like real environment observations, as long as the synthetic environment preserves causal structure and reward semantics.

The empirical evidence supporting this reframing is both direct and indirect. The indirect evidence is the main results in Table 1: agents trained purely in DreamGym's abstract state space match or exceed real-environment RL performance on WebShop and ALFWorld, and dramatically outperform all baselines on WebArena, despite never seeing a real webpage or embodied scene during training. If pixel-level fidelity mattered for policy learning, this level of transfer would be impossible. The direct evidence is in the experience model ablation (Figure 4), which evaluates generated states across four criteria using GPT-4o as judge: causal consistency, diversity, informativeness, and hallucination. DreamGym achieves high scores on all four without any metric corresponding to "looks like the real environment," confirming that the properties the theory identifies as important—consistency, reward accuracy, factual grounding—are the ones the system actually optimizes.

This is a fundamental shift rather than an incremental refinement. It changes the optimization target for synthetic environment design from "build a faithful replica" to "build a causally consistent, reward-accurate experience generator," which is a dramatically easier engineering problem. It also implies that future work on synthetic environments should invest effort in improving reasoning quality and factual grounding (to reduce ε_P and ε_R) rather than in higher-resolution rendering or more detailed world modeling.

---

Innovation 2: The Difficulty-Conditioned Synthetic Experience as a Unified Abstraction Over Task Generation, Environment Simulation, and Curriculum Design

Prior work on scaling RL for LLM agents treats task generation, environment simulation, and curriculum design as three separate problems requiring three separate solutions. Task generation methods like AgentSynth (Xie et al., 2025) and SCA (Zhou et al., 2025) focus on producing diverse task instructions but still require real environments to validate and execute those tasks. World model approaches like WebDreamer (Gu et al., 2024) focus on simulating environment dynamics but operate over fixed task sets. Curriculum learning methods for LLMs (Gao et al., 2025) focus on ordering or selecting tasks but assume the task set already exists. Each of these partial solutions addresses one piece of the RL bottleneck while leaving the others unsolved.

DreamGym's architectural innovation is recognizing that these three functions can be unified within a single reasoning-based experience model that serves simultaneously as task generator, environment simulator, and curriculum designer. The experience model M_exp generates initial states conditioned on task instructions (simulating the environment's response to a new query), produces multi-turn state transitions conditioned on agent actions (simulating environment dynamics), and generates task variations conditioned on high-entropy seed tasks (simulating the process of task design). The same reasoning mechanism—explicit chain-of-thought about action consequences, task objectives, and state consistency—underlies all three functions.

What makes this unification conceptually significant rather than merely an engineering convenience is that it resolves a circular dependency that previously made scalable RL for agents intractable. To train an RL agent, you need diverse tasks → to generate diverse tasks, you need to validate them in the target environment → to validate them, you need an agent that can execute them (or expensive human evaluation) → to train such an agent, you need RL on diverse tasks. Prior approaches broke this cycle by injecting human expertise at some point (human-designed tasks, human-labeled demonstrations, human-validated task variations). DreamGym breaks it by making the experience model itself the validator: because M_exp can simulate arbitrary multi-turn interactions for any proposed task, whether a generated task is "feasible" is no longer a question about the real environment but a question about whether M_exp can produce coherent rollouts for it. And since M_exp is trained on offline data from the target domain and grounded in a replay buffer of real trajectories, its notion of feasibility is anchored in actual environment constraints.

This is a fundamental architectural innovation rather than an incremental improvement over prior simulators. It transforms the experience model from a passive component (a learned transition function that responds to agent actions) into an active participant in the training loop that shapes the agent's curriculum and expands the task space. The evidence that this unification matters is in the ablation results (Table 2): removing the task generator from an otherwise identical system causes a 6.6% drop on WebShop and a 6.0% drop on WebArena, even though the experience model and replay buffer remain intact. The synergy is that the task generator operates on exactly the same environment dynamics that the experience model simulates—when it generates a variation of a WebArena task, it "knows" what states and actions are available because it learned those dynamics during fine-tuning, not because it has separate task-design knowledge.

This unification also has economic implications that the paper does not fully explore. In a traditional pipeline, maintaining three separate subsystems (task generator, environment simulator, curriculum scheduler) means three separate models to train, maintain, and debug. DreamGym's unified architecture means a single fine-tuned LLM serves all three roles, proportional to its size and training data. The paper's data efficiency results (Figure 5) suggest that this unified model achieves competitive performance with as few as 2,000–10,000 transition steps of offline training data—a remarkably small amount for a system that simultaneously learns environment dynamics, task structure, and curriculum design.

---

Innovation 3: Proving That Synthetic-Only Training Can Match Real-Environment RL Without Any Real Interaction

The paper's third distinctive contribution is not a method but an empirical finding with broad implications: that an agent trained entirely on synthetic rollouts from a reasoning-based experience model can match or exceed the performance of agents trained via standard RL in the real environment, across multiple benchmarks and model families. Table 1 demonstrates this concretely: on WebShop with Llama-3.1-8B, DreamGym + GRPO achieves 63.9% versus traditional GRPO at 65.0%, and on ALFWorld with Qwen-2.5-7B, DreamGym + PPO achieves 72.7% versus traditional PPO at 81.1%. These are not uniformly better results—traditional RL holds an edge in several configurations—but they are competitive in a regime (zero real interactions) that prior work would have considered impossible.

The significance of this finding lies in what it refutes. A reasonable prior—implicit in much of the world-model literature and explicit in the design of systems like UI-Simulator (which generates synthetic rollouts only as SFT augmentation, not as an RL training substrate)—is that synthetic data is useful as a supplement to real data (pre-training, data augmentation, demonstration bootstrapping) but cannot serve as a complete replacement for real interaction during RL. The intuition is that synthetic environments inevitably have blind spots—states they cannot accurately simulate, dynamics they oversimplify, edge cases they miss—and that RL's exploratory nature will systematically find and exploit these blind spots, leading to policies that perform well in simulation but fail in reality (the classic sim-to-real gap).

DreamGym's results challenge this intuition. The fact that synthetic-only training matches real-environment RL performance on WebShop and ALFWorld—environments where the real RL baselines are themselves quite strong (60–80% success rates)—suggests that the reasoning-based experience model's transition errors (ε_P) and reward errors (ε_R) are small enough that the trust-region penalty and model-error penalty in Theorem 1 do not dominate the surrogate gain. In practical terms: the synthetic environment is sufficiently accurate that optimizing against it does not systematically exploit its inaccuracies. The paper's qualitative analysis (Figure 6) shows why: the chain-of-thought reasoning explicitly considers whether actions are valid, what their consequences should be, and whether they advance the task, producing states that are causally grounded rather than statistically plausible but physically impossible.

This finding also reframes what "sim-to-real transfer" means. The conventional framing (as in Dreamer and related work) is that synthetic pre-training provides a warm-start that must be followed by real-environment fine-tuning to achieve competitive performance. DreamGym-S2R is consistent with this framing—synthetic + limited real fine-tuning outperforms either alone. But the more striking result is that synthetic-only training already matches pure real-environment RL on RL-ready benchmarks, which implies that the synthetic environment is not merely a warm-start mechanism but a viable replacement for the real environment in the bulk of training. The remaining value of real interaction is marginal (5–11 percentage points on WebShop, as shown in Table 1) rather than essential—a refinement on top of an already-competent policy rather than a prerequisite for competence.

This is a fundamental empirical finding with practical consequences. It means that for environments where building RL infrastructure is prohibitively expensive (WebArena being the paper's prime example), DreamGym does not merely provide a way to make some progress—it provides a way to achieve performance that would otherwise be completely inaccessible. The WebArena results in Table 1 are the most dramatic illustration: traditional GRPO and PPO achieve 4.8–7.3% across all backbones, barely above random behavior, while DreamGym achieves 9.1–14.5%, a 30–100% relative improvement. For organizations deciding whether to invest in RL infrastructure for complex agent environments, this finding suggests that synthetic training may be not just cheaper but also more effective when real-environment RL is too unreliable to converge.

---

Innovation 4: Reward Entropy as a Training-Signal Quality Heuristic That Bridges Task Selection and Curriculum Generation

The paper introduces a simple but conceptually elegant heuristic for identifying which tasks are most valuable for training: group-based reward entropy (Equation 7). The idea is that tasks where the agent sometimes succeeds and sometimes fails—where the variance of outcome rewards across multiple rollouts is high—provide the strongest learning signal, because they are at the boundary of the agent's current capability where small differences in action choices determine the outcome.

What makes this a conceptual innovation rather than an obvious design choice is its dual role in the DreamGym pipeline. The reward-entropy heuristic serves simultaneously as: (1) a task selection criterion that identifies which existing tasks should be prioritized for training (high-entropy tasks provide more informative gradients), and (2) a curriculum generation seed selector that identifies which tasks should be used as templates for generating harder variations (high-entropy tasks represent the frontier of capability where pushing further is most productive). The same scalar quantity—variance of binary outcomes across a group of rollouts—drives both decisions, creating a tight feedback loop between what the agent currently finds challenging and what new challenges the system produces.

This is distinct from prior work on curriculum learning and task selection in several ways. Difficulty-based curricula (common in RL and supervised learning) typically require an external estimate of task difficulty—either human-provided difficulty labels or a separately trained difficulty predictor. Learning-progress-based curricula (Graves et al., 2017; Matiisen et al., 2019) track the rate of improvement on each task and prioritize those where learning is fastest. Uncertainty-based selection uses model uncertainty to identify informative examples. DreamGym's reward-entropy heuristic is simpler than all of these—it requires only sampling n rollouts and counting successes, with no additional model, no difficulty labels, and no tracking of learning trajectories over time—yet it captures the key insight that tasks at a 50% success rate provide maximum information for credit assignment (a direct consequence of the Bernoulli variance formula: Var = p(1-p), maximized at p = 0.5).

What elevates this from a practical trick to a conceptual contribution is its connection to the information-theoretic value of experience for policy gradient methods. The magnitude of the policy gradient depends on the advantage estimates—actions that lead to better-than-expected outcomes receive positive updates, actions that lead to worse-than-expected outcomes receive negative updates. When a task has zero reward entropy (always succeeds or always fails), the advantage estimates are uniformly zero (since every trajectory gets the same outcome), and the policy gradient provides no information about which specific actions were good or bad. When reward entropy is high, different trajectories produce different outcomes, creating non-zero advantages that attribute credit to specific action choices. The reward-entropy heuristic can therefore be understood as a computationally cheap proxy for the expected magnitude of the policy gradient on a given task, which is exactly the quantity one would want to maximize when selecting training data.

The ablation evidence (Table 2, Figure 3 right) demonstrates that this heuristic matters in practice: removing the task generator (and thus the entropy-driven curriculum) causes a 6.0–6.6% absolute drop in success rate across environments and leads to earlier plateauing during training. The paper does not ablate the entropy criterion itself against alternative task selection heuristics (e.g., random selection, difficulty-based selection using the PRM's confidence, or selection based on policy gradient norm), so we cannot conclude that reward entropy is uniquely effective—but the conceptual framework of using outcome variance as a unified signal for both selection and generation is a clean, principled contribution that future work can build on or compare against.

---

Innovation 5: The Experience Replay Buffer as a Co-Evolution Mechanism Rather Than a Static Knowledge Store

The final conceptual contribution is a subtle but important refinement of how replay buffers should function in synthetic training pipelines. The standard role of a replay buffer in RL (since Mnih et al., 2015) is to store past experiences for off-policy learning, breaking the temporal correlations that destabilize neural network training. In DreamGym, the replay buffer serves an additional and more structurally significant role: it is a co-evolving grounding mechanism that keeps the experience model's predictions aligned with the agent's changing behavior.

The innovation is in recognizing that a static set of offline demonstrations is insufficient to ground a synthetic environment during RL training, because the agent's policy will systematically explore regions of state-action space that are not well-covered by the offline data. If the experience model retrieves only expert demonstrations—which show optimal actions leading to successful outcomes—to condition its predictions, then when the agent takes suboptimal or exploratory actions during training, the model has no relevant examples to guide its reasoning about what should happen next. The resulting prediction errors (increased ε_P) would compound over multi-step rollouts, eventually producing trajectories that diverge substantially from real environment dynamics.

DreamGym's solution is to continuously update the buffer with synthetic trajectories generated during training. As the agent's policy evolves—exploring new actions, making new kinds of mistakes, discovering new strategies—the buffer accumulates examples of these behaviors and their (synthetic) consequences, providing the experience model with increasingly relevant conditioning information. This is a co-evolution between the agent and the experience model: the agent's behavior shapes the distribution of trajectories in the buffer, which shapes the experience model's predictions, which shape the experiences the agent learns from, which shape the agent's future behavior. The system is a coupled dynamical system rather than a static environment.

What distinguishes this from standard Dyna-style architectures (where a world model is updated online via supervised learning on real transitions) is that DreamGym's experience model parameters are frozen after offline training—only the retrieval buffer changes. This decouples the stability problem of online world-model learning (where errors in the model can compound through self-training) from the adaptation problem of keeping predictions policy-relevant. The buffer provides a lightweight, interpretable, and computationally cheap mechanism for adaptation that does not risk the catastrophic forgetting or representational collapse that can occur when fine-tuning world models on their own outputs.

This is a conceptual refinement of how replay should work in synthetic training pipelines rather than a fundamental breakthrough, but it has important practical implications. It suggests that the key to effective synthetic RL is not just the quality of the initial experience model but also the quality of the retrieval mechanism that conditions its predictions. The ablation in Table 2 (4.7% drop on WebShop and 3.6% on WebArena when removing the replay buffer) confirms that this mechanism matters quantitatively, and Figure 4 confirms that removing history (which the buffer enriches) primarily hurts causal consistency. The paper does not experiment with alternative mechanisms for co-evolution (e.g., periodic fine-tuning of the experience model, learned retrieval policies, or dynamic buffer pruning strategies), but the core insight—that the experience model and agent must co-evolve through a shared memory structure—is a design principle that future synthetic training frameworks should incorporate irrespective of implementation details.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three agentic benchmarks spanning diverse domains and RL-readiness levels: WebShop (Yao et al., 2022), a large-scale e-commerce environment with 1.18M real-world products and 12,087 crowd-sourced natural language instructions requiring agents to search, customize, and purchase items; ALFWorld (Shridhar et al.), a text-and-embodied benchmark pairing abstract text interactions from TextWorld (Côté et al., 2018) with photo-realistic execution in ALFRED/AI2-THOR across six household task families (Pick & Place, Clean & Place, Heat/Cool & Place) with 3,553 training tasks across 120 rooms; and WebArena-Lite (Zhou et al.), a more balanced subset of 165 high-quality tasks selected from the original 812 WebArena tasks, spanning e-commerce, social forums, GitLab, and content management, where the remaining 647 tasks (excluding the evaluation set) are used for training (Appendix A.3). The paper fixes the test set for evaluation stability and collects training trajectories only from the remaining tasks to avoid test-set contamination.

Base model(s). Agent backbones span different model families and sizes to assess generalizability: Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct (Grattafiori et al., 2024), and Qwen-2.5-7B-Instruct (Team, 2024). All three are used with full-parameter fine-tuning. The reasoning experience model M_exp is trained from Llama-3.1-8B-Instruct for all main results (Section 5.1), with Section 6.4 ablating the experience model backbone to Llama-3.2-3B and comparing against WebDreamer (Gu et al., 2024). The paper states the base models are "representative of the capabilities of many contemporary LLMs" (Section 4) and span a useful range for evaluating whether synthetic training works across model scales and families.

Metrics. The primary metric throughout is success rate (%), defined as the fraction of evaluation tasks for which the agent's final action sequence achieves the task objective as judged by the environment's evaluation function. For WebShop, this means correctly identifying and purchasing the specified product with appropriate attributes. For ALFWorld, this means completing the specified household task (e.g., "put a clean apple in the fridge"). For WebArena, this uses the functional correctness evaluation from Zhou et al. rather than action-trace matching. The paper does not report confidence intervals or standard deviations on success rates, which is a limitation given the relatively small test sets (WebArena-Lite: 165 tasks; WebShop/ALFWorld test sizes are not explicitly stated in the paper but follow standard splits).

Baselines. The paper compares against four groups of training strategies (Table 1):

• Offline imitation learning:

  • SFT (supervised fine-tuning): standard behavioral cloning on 20K real transitions from expert demonstrations.
  • DPO (Direct Preference Optimization, Rafailov et al., 2023): preference-based fine-tuning on 40K real transitions (paired successful and unsuccessful trajectories).

• Online RL in real environments (Traditional):

  • GRPO (Group Relative Policy Optimization, Shao et al., 2024): trained on 80K real environment interactions using group-relative advantage normalization.
  • PPO (Proximal Policy Optimization, Schulman et al., 2017): trained on 80K real environment interactions using Generalized Advantage Estimation with a learned value function. Both RL baselines use full-parameter fine-tuning with standard hyperparameter settings from Verl-Agent (Feng et al., 2025).

• DreamGym variants (this paper):

  • DreamGym: agents trained via GRPO or PPO entirely within the synthetic environment, with zero real interactions.
  • DreamGym-S2R: agents first trained with synthetic experiences (same as DreamGym), then fine-tuned with RL in the real environment using only 5K real transitions—a 16× reduction from the 80K used by traditional RL baselines.

Generation budget / compute accounting. For DreamGym, there is no generation budget in the traditional sense—synthetic rollouts are generated on-the-fly by the experience model for each training step, and the primary cost is GPU hours for LLM inference rather than environment interaction cost. The paper measures training efficiency in terms of training time (Figure 3, left) including both rollout sampling time and GPU hours, showing that DreamGym reduces total training effort to "roughly one-third or even one-fifth of RL baselines in the real environment" on WebArena. For traditional RL baselines, the computation budget is measured in number of real transitions (80K for traditional RL, 5K for DreamGym-S2R fine-tuning, 20K for SFT, 40K for DPO), with each transition representing one state-action-reward step (a trajectory typically has ~10 steps, so 80K transitions ≈ 8K trajectories). All experiments are conducted on 8 nodes with A100 GPUs and 4 nodes with H100 GPUs (Appendices A.1, A.2, A.3).

Cross-validation / statistical protocol. The paper does not describe a formal cross-validation or statistical testing protocol. For WebArena, the evaluation uses the fixed WebArena-Lite subset of 165 tasks, with the remaining 647 tasks used for training—a standard held-out split. For WebShop and ALFWorld, standard training/test splits from the original benchmarks are used. The paper does not report multiple random seeds, confidence intervals, or statistical significance tests for any of its results, which is a limitation for assessing the robustness of the reported performance differences, particularly given that some margins between methods are relatively small (e.g., DreamGym vs. Traditional GRPO differences of 1–3 percentage points in Table 1).

Main Quantitative Results

Non-RL-Ready Environment: WebArena

The most dramatic results are on WebArena, where real-world RL baselines fundamentally fail due to infrastructure constraints (sparse rewards, no reliable reset mechanisms, limited parallelization to 4 AWS server instances as described in Appendix A.3). Table 1 reports:

• Offline methods (SFT, DPO) achieve only 4.8–7.3% across all backbones, barely above random behavior for a benchmark where tasks require multi-step navigation and information retrieval.

• Traditional GRPO and PPO trained on 80K real interactions achieve only 4.8–7.3%, demonstrating that even with substantial engineering effort, real-environment RL on WebArena is essentially non-functional. The paper notes that "some trajectories fail to execute properly and that certain tasks are incorrectly judged by WebArena's original evaluation function" (Appendix A.3), explaining the poor baseline performance.

• DreamGym-trained agents achieve 9.1–14.5% across backbones—a >30% relative improvement over all baselines. For Llama-3.2-3B with PPO: DreamGym achieves 14.5% vs. Traditional PPO at 6.7%; for Qwen-2.5-7B with GRPO: DreamGym achieves 12.7% vs. Traditional GRPO at 6.1%.

• DreamGym-S2R adds limited real fine-tuning (5K transitions) to achieve 9.7–13.9%, which is largely comparable to DreamGym-only (the real fine-tuning provides minimal additional benefit on WebArena—for Llama-3.1-8B with GRPO, S2R achieves 9.7% vs. DreamGym-only at 9.1%). This suggests that the domain gap between the abstract state space and WebArena's complex web dynamics is large enough that 5K real transitions are insufficient to substantially improve over synthetic-only training, though the synthetic-only training itself still dramatically outperforms all real-environment baselines.

The training efficiency comparison in Figure 3 (left) reinforces these findings: DreamGym achieves its performance gains while reducing total training time (rollout sampling + GPU hours) to roughly one-third to one-fifth of RL baselines in the real environment. The efficiency gain arises from "both the dense feedback offered by curriculum-based rollout synthesis and the lightweight abstract state transitions produced by unified experience models hosted by scalable LLM services."

RL-Ready Environments: WebShop and ALFWorld

On environments where real RL is feasible but expensive, DreamGym matches or closely approaches real-environment RL baselines using purely synthetic data. Table 1 reports:

• WebShop:

  • Traditional GRPO achieves 62.1–66.1% across backbones. DreamGym + GRPO achieves 59.3–68.3%—statistically indistinguishable performance with zero real interactions. The largest gap is for Llama-3.2-3B where DreamGym trails by 2.8 percentage points (59.3% vs. 62.1%); the largest advantage is for Qwen-2.5-7B where DreamGym leads by 2.2 points (68.3% vs. 66.1%).
  • Traditional PPO achieves 59.9–68.1%. DreamGym + PPO achieves 58.1–65.0%—a maximum gap of 6.1 points (Llama-3.1-8B at 58.1% vs. 64.2%).
  • DreamGym-S2R (5K real fine-tuning after synthetic pre-training) achieves 63.9–75.0%, consistently outperforming both pure synthetic and pure real training. For Llama-3.1-8B with GRPO, S2R reaches 75.0% vs. Traditional GRPO at 65.0% and DreamGym-only at 63.9%—a 10-point improvement over real-only training while using 16× fewer real interactions.

• ALFWorld:

  • Traditional GRPO achieves 65.3–79.8%. DreamGym + GRPO achieves 62.1–71.0%—trailing by 3.2–8.8 points, a larger gap than on WebShop. The paper does not analyze why the gap is larger on ALFWorld, but possible explanations include the embodied domain having sparser semantic structure for the abstract state representation or the experience model having less offline training data for embodied dynamics.
  • Traditional PPO achieves 47.0–81.1%. DreamGym + PPO achieves 40.5–72.7%—with the largest gap for Llama-3.2-3B (47.0% vs. 40.5%).
  • DreamGym-S2R achieves 49.1–82.4%, again consistently outperforming both pure synthetic and pure real training. For Qwen-2.5-7B with GRPO, S2R reaches 82.4% vs. Traditional GRPO at 79.8% and DreamGym-only at 71.0%.

Key pattern across both RL-ready environments: DreamGym-S2R consistently achieves the highest performance, demonstrating that synthetic pre-training provides a strong initialization that makes subsequent real-environment RL more sample-efficient—the synthetic training expands exploration coverage and builds broad behavioral priors, then limited real interaction provides grounding in actual environment dynamics.

Training Dynamics and Sample Efficiency

Figure 3 (right) compares training curves on WebShop for Traditional GRPO, DreamGym, and a DreamGym variant without the curriculum task generator:

• DreamGym success rate improves more rapidly within the first 40 training steps, with the curve showing that "synthesized trajectories offer more informative gradients than sparse real rollouts."

• DreamGym curves exhibit lower variance across runs, while "baseline curves exhibit larger oscillations caused by sparse or unstable rewards." This suggests that the synthetic environment provides not only denser but also more consistent feedback, mitigating training instabilities commonly reported in WebShop and ALFWorld.

• Without the curriculum task generator, DreamGym makes initial progress but plateaus more quickly, demonstrating that adaptive task generation is essential for sustained learning progress. The full DreamGym curve continues to improve while "DreamGym w/o Task" flattens.

Cross-Domain Transferability

Figure 3 (middle) evaluates the generalization of policies trained in DreamGym across environments, revealing both strengths and limitations:

• Positive transfer between related domains: When trained on WebShop, the policy generalizes to WebArena and surpasses SFT models directly trained there. When trained on WebArena, it similarly transfers back to WebShop with superior performance compared to SFT. This suggests that DreamGym learns within an abstract meta-representation space that captures domain-agnostic behavioral priors (how to navigate, how to search, how to interpret task objectives) rather than memorizing environment-specific patterns.

• Negative transfer across large domain gaps: When transferring from web-based environments (WebShop/WebArena) to ALFWorld (embodied control), "the performance drops significantly, indicating the limits of current meta-representations." This is an important boundary condition: the abstract state space can bridge similar domains (different types of websites) but cannot bridge fundamentally different interaction modalities (web navigation vs. physical manipulation).

Ablation Studies and Robustness Checks

Curriculum task generator (Table 2, Figure 3 right): Removing the task generator causes a 6.6% absolute drop on WebShop (63.9% → 57.3%) and a 6.0% drop on WebArena (13.3% → 7.3%). The training curve in Figure 3 (right) shows that the "DreamGym w/o Task" variant makes initial progress but then plateaus, supporting the claim that "without adaptive task generation, the replay buffer may saturate with low-entropy, repetitive trajectories, which limits the diversity of experiences and stalls exploration." The paper does not report the optimal value of the synthetic task proportion hyperparameter λ or ablate it, which leaves open the question of how sensitive these results are to the balance between original and synthetic tasks.

Experience replay buffer (Table 2): Removing the replay buffer ("w/o Exp. Replay") causes a 4.7% drop on WebShop (63.9% → 59.2%) and a 3.6% drop on WebArena (13.3% → 9.7%). This quantifies the value of grounding synthetic transitions in retrieved real-world demonstrations. The paper does not ablate the retrieval mechanism itself (top-k value k, choice of semantic encoder φ), so the sensitivity to these hyperparameters is unknown.

Experience reasoning (Table 2, Figure 4): Removing chain-of-thought reasoning from the experience model ("w/o Exp. Reasoning") causes the largest single-component drop: 8.1% on WebShop (63.9% → 55.8%) and 6.0% on WebArena (13.3% → 7.3%). Figure 4 provides a more granular analysis using GPT-4o as a judge to evaluate generated states across four criteria (causal consistency, diversity, informativeness, hallucination), each scored 0–2:

• Removing reasoning ("w/o Reasoning") primarily hurts informativeness and increases hallucination—"without reasoning capabilities, the generated experiences tend to become shallow and less factually grounded."
• Removing interaction history ("w/o History") primarily hurts causal consistency—"without awareness of prior turns, the model often drifts off-topic and breaks causal coherence in multi-step interactions."
• The full DreamGym achieves the best or near-best performance across all four criteria, confirming that reasoning and history provide complementary benefits: "history preserves temporal and causal structure, while reasoning enhances depth and factual reliability."

Experience model backbone and training data scale (Figure 5): Evaluated on (a) WebShop and (b) WebArena, varying both the number of offline training samples (2K, 10K, 20K, 40K transition steps) and the model backbone:

• Data efficiency: The Llama-3.1-8B experience model already exceeds 50% success rate on WebShop with only 10K offline samples, indicating that "large-scale offline datasets are not strictly necessary for effective experience synthesis." Performance largely saturates around 20K samples for both environments.
• Smaller backbones remain viable: Llama-3.2-3B underperforms the 8B model but improves steadily with more data, reaching about 55% on WebShop with 20K samples, suggesting "lightweight models can still serve as practical experience generators when computational resources are constrained."
• Domain-specific pretraining offers early advantage but is not essential: On WebArena, WebDreamer (a fine-tuned web world model) initially outperforms the Llama-3.1-8B experience model at small data scales (~13% vs. ~9% at 2K samples), but the gap closes as offline data increases, with both converging around 13% at 40K samples. This suggests that "web-specific pretraining offers an early advantage, it is not a strict prerequisite, as larger-scale experience learning can enable pre-trained models to bridge the performance gap."

Experience model quality evaluation (Figure 4): Using GPT-4o as a judge (prompt in Appendix C.4) to score 100 randomly sampled trajectories across four criteria on a 0-1-2 scale, comparing Traditional real environment, DreamGym, DreamGym w/o History, and DreamGym w/o Reasoning:

• The full DreamGym achieves the highest scores on all four criteria, with notably strong performance on causal consistency and low hallucination.
• Removing history degrades causal consistency more than any other criterion—the model loses awareness of temporal context and produces transitions that break multi-step coherence.
• Removing reasoning degrades informativeness and increases hallucination more than removing history—the model produces shallower, less factually grounded states.
• The paper does not report inter-annotator agreement or calibration of the GPT-4o judge against human evaluations, so the reliability of these scores as absolute quality measures is uncertain, though the relative ordering (which component matters for which criterion) is informative.

No ablation on the λ hyperparameter: The paper introduces λ to bound the proportion of synthetic tasks per iteration but does not report its value or ablate it. Given that the task generator is one of the most impactful components (6.0–6.6% drop when removed), understanding the sensitivity to λ would be important for reproducibility.

No ablation on the number of difficulty bins or the reward-entropy threshold: The curriculum uses reward entropy to select seed tasks but the paper does not specify what threshold (if any) is used to determine which tasks are "high-entropy," nor does it ablate alternative task selection criteria (random selection, difficulty-based, policy-gradient-norm-based).

Negative result: ReST^EM revision model degradation not applicable. The paper does not report an analogous negative result to the ReST^EM failure in the reference example. The closest negative result is the cross-domain transfer failure (Figure 3, middle), where transferring from web environments to ALFWorld causes significant performance drops, indicating that the abstract meta-representation space does not bridge fundamentally different interaction modalities.

Critical Assessment

Claim 1: "On non-RL-ready tasks like WebArena, DreamGym outperforms all baselines by over 30%."

This claim is strongly supported by Table 1, but the nature of the improvement requires careful interpretation. The 30% figure refers to a relative improvement: DreamGym achieves 9.1–14.5% success versus 4.8–7.3% for baselines, which represents roughly a 50–100% relative increase but is an absolute improvement of only 4–8 percentage points. The baselines are genuinely weak—real-environment RL on WebArena essentially fails—so DreamGym's improvement, while genuine and meaningful (it makes RL training possible where it was previously impossible), starts from a very low baseline. The practical implication is that DreamGym enables some RL-based improvement on WebArena, not that it solves WebArena. A success rate of 13–14% on a 165-task benchmark means ~23 tasks are solved correctly, leaving substantial room for improvement.

A significant limitation of this claim is that the WebArena baselines were run under severely constrained conditions—only 4 parallel AWS server instances—and the paper acknowledges that "some trajectories fail to execute properly and that certain tasks are incorrectly judged by WebArena's original evaluation function." It is possible that with more engineering investment (more servers, better error handling, improved evaluation), real-environment RL on WebArena could achieve better performance, narrowing or eliminating DreamGym's advantage. The claim that DreamGym "outperforms all baselines by over 30%" should therefore be understood as "outperforms the specific real-environment RL baselines the authors were able to implement given infrastructure constraints," not as a fundamental result about the impossibility of real-environment RL on WebArena.

Claim 2: "In RL-ready but costly settings, DreamGym matches GRPO and PPO performance using only synthetic interactions."

This claim is supported with qualifications. On WebShop, DreamGym + GRPO achieves results within 0–3 percentage points of Traditional GRPO across backbones (Table 1), which is a genuinely strong result—matching real-environment RL with zero real interactions is remarkable. On ALFWorld, however, the gap is larger: DreamGym + GRPO trails Traditional GRPO by 3.2–8.8 percentage points, and DreamGym + PPO trails Traditional PPO by 6.5–8.4 points for some configurations (e.g., Qwen-2.5-7B: DreamGym PPO at 72.7% vs. Traditional PPO at 81.1%). The claim of "matching" performance is more accurate for WebShop than for ALFWorld. The paper does not analyze or explain the larger gap on ALFWorld.

Moreover, "matching" implies statistical equivalence, but the paper reports no confidence intervals or measures of uncertainty. With test sets of unstated size (WebShop's standard test set is 500 tasks; ALFWorld's test splits vary by configuration) and no multiple random seeds reported, differences of 3–8 percentage points could be statistically significant. The claim would be more precisely stated as "DreamGym achieves performance within a few percentage points of real-environment RL on WebShop, with larger gaps on ALFWorld."

Claim 3: "DreamGym-S2R yields significant additional performance gains while requiring far fewer real-world interactions."

This claim is strongly supported. Across all backbones and environments, DreamGym-S2R (synthetic pre-training + 5K real fine-tuning) consistently achieves the highest performance (Table 1), typically outperforming both pure DreamGym and pure real-environment training. The reduction in real interactions is substantial: 5K versus 80K, a 16× reduction. However, the claim that this provides a "scalable warm-start strategy for general-purpose RL" is broader than what the experiments demonstrate. The S2R results are shown only for environments where DreamGym was trained—there is no experiment demonstrating that a policy pre-trained on DreamGym for Environment A transfers well to real RL on Environment B (cross-environment S2R). The "general-purpose" claim would require cross-environment transfer experiments that the paper does not provide.

An important caveat: the S2R comparison gives the traditional RL baselines 80K real interactions from scratch, while DreamGym-S2R gets an unlimited budget of synthetic pre-training plus 5K real interactions. This is not a FLOPs-matched comparison—the synthetic pre-training has a computational cost (GPU hours for LLM inference) that is not accounted for in the "far fewer real-world interactions" framing. If the synthetic pre-training required, say, 10× more GPU hours than the real RL training (the paper claims it reduces total training time to "one-third or one-fifth" on WebArena, but does not report analogous figures for WebShop/ALFWorld), then the total computational cost of DreamGym-S2R might not be lower than traditional RL—it just shifts cost from environment interaction to LLM inference. The economic argument depends on the relative cost of these two resources, which the paper does not quantify.

Claim 4: "The key bottleneck in RL for LLM agents lies in the quality and structure of interaction data."

This claim is conceptually supported but not experimentally isolated. The paper's architecture demonstrates that synthetic data can substitute for real data, but it does not run experiments that isolate what properties of the synthetic data are necessary or sufficient. For example, there is no experiment comparing DreamGym's reasoning-driven synthetic data against a simpler synthetic data generation method (e.g., an experience model without reasoning, or with random state transitions, or with only static replay buffer retrieval). The ablation in Table 2 shows that removing individual components reduces performance, but does not establish that DreamGym's specific approach to data quality—as opposed to simply having any synthetic data—is what matters. A baseline that generates synthetic data via a simpler mechanism (e.g., template-based state generation) would be needed to directly support the claim that quality and structure, specifically, are the bottleneck.

Missing experiments that would strengthen the paper:

• Statistical significance testing or confidence intervals for all main results. The relatively small test sets (WebArena-Lite: 165 tasks) and unstated number of random seeds make it difficult to assess whether reported differences are reliable.

• FLOPs-matched or wall-clock-matched comparison between DreamGym (synthetic) training and real-environment training. The paper claims efficiency gains on WebArena (Figure 3, left) but does not provide analogous timing comparisons for WebShop or ALFWorld. Without these, the claim that DreamGym is "more scalable" is primarily about the feasibility of scaling (no infrastructure bottlenecks) rather than about computational cost.

• Ablation on the λ hyperparameter that controls the proportion of synthetic tasks. Given the task generator's importance, understanding sensitivity to this parameter is critical for reproducibility.

• Alternative task selection criteria for the curriculum generator (random selection, difficulty-based selection using other heuristics, selection based on policy gradient norm) to establish that reward entropy is uniquely effective rather than simply better than nothing.

• Cross-environment S2R transfer: training in DreamGym for Environment A, then fine-tuning on real Environment B, to test whether synthetic pre-training provides general behavioral priors rather than environment-specific knowledge.

• Evaluation on additional model families beyond Llama and Qwen (e.g., Mistral, Gemma) to strengthen the claim of backbone-agnosticism.

• Scaling the experience model size: all main experiments use Llama-3.1-8B as M_exp. How does performance scale with a smaller (3B) or larger (70B) experience model? Figure 5 partially addresses this with the 3B ablation, but only for the experience model's direct success rate, not for downstream RL training performance.

Where the claims hold conditionally: The synthetic-only training claim ("matches GRPO and PPO") holds most strongly on WebShop (0–3 point gap) and less strongly on ALFWorld (3–9 point gap). The WebArena claim (>30% improvement) holds against the specific baselines the authors implemented but may not generalize to better-engineered real-environment RL setups. The S2R claim holds for same-environment transfer but is untested for cross-environment transfer. All claims are evaluated on a single experience model backbone (Llama-3.1-8B) with a specific offline data scale (2K–20K samples depending on environment), and may not generalize to substantially different experience model architectures or data regimes.

6. Limitations and Trade-offs

The Experience Model Is Trained on Environment-Specific Offline Data and Does Not Generalize Across Modalities

The assumption or constraint. DreamGym trains a separate reasoning experience model M_exp for each target environment, using offline trajectory data collected from that specific domain—1,600 human demonstrations plus 2,000 oracle/random trajectories for WebShop, 3,200 expert demonstrations plus 2,000 oracle/random for ALFWorld, and 4,800 leaderboard trajectories for WebArena (Appendix A). The paper's cross-domain transfer experiment (Figure 3, middle) reveals the boundary of this design: when DreamGym is trained on WebShop or WebArena and the resulting policy is evaluated on the other, performance transfers positively (both are web-based environments with similar interaction patterns). But when transferring from web-based environments to ALFWorld (embodied control), "the performance drops significantly, indicating the limits of current meta-representations." The paper acknowledges this explicitly as a limitation in Section 7: "Our work primarily investigates single-environment learning setups, where DreamGym is applied to individual agentic scenarios."

The consequence. A practitioner wanting to deploy DreamGym across a diverse suite of agent tasks (web navigation, code generation, embodied control, tool use) would need to collect offline trajectory data and train a separate experience model for each environment type. The paper provides no evidence that a single experience model can serve multiple environments or that knowledge transfers across experience models. This fragments the scaling benefits: each new domain incurs the fixed cost of data collection, teacher-LLM reasoning annotation, and model fine-tuning, which—while modest per domain (2K–20K samples, Figure 5)—accumulates across a portfolio of applications. Moreover, the failed web-to-embodied transfer suggests that the abstract state representations learned by M_exp are domain-specific: they capture what makes a web page semantically structured but not what makes physical object interactions causally coherent. Environments with fundamentally different causal structures would require retraining from scratch, not merely fine-tuning.

What evidence exists in the paper. Figure 3 (middle) provides direct evidence of the cross-domain boundary: policies trained in DreamGym on web environments transfer well between WebShop and WebArena but fail on ALFWorld. No experiment trains a single experience model on mixed-domain data (e.g., WebShop + ALFWorld trajectories combined) to test whether a unified abstract state space could bridge modalities.

Mitigation status. The paper suggests in Section 7 that the framework "can be further extended to build a universal world model that unifies multiple environment models, enabling knowledge transfer across environments." This is presented as future work without any preliminary experiments or architectural proposals. The current system offers no mitigation.

---

The Difficulty Estimation Cost Is Unaccounted For and Potentially Dominates the Synthetic Training Budget

The assumption or constraint. The curriculum task generator requires computing reward entropy V_τ across n rollouts for each candidate task to identify high-value seeds for variation generation (Equation 7). This means DreamGym must repeatedly sample multiple full trajectories per task—using the experience model itself—purely to determine which tasks are worth varying, before those variations can be used for actual policy training. The paper does not report how many rollouts n are used for entropy estimation, what fraction of the total synthetic generation budget is consumed by this meta-evaluation versus actual training rollouts, or whether the entropy estimation cost is included in the efficiency comparisons (Figure 3, left). Section 3.2 states that "our experiments do not account for this cost largely for simplicity," acknowledging the omission but not quantifying it.

The consequence. If n is large (say, 16–32 rollouts per task to get a reliable entropy estimate) and the task pool contains hundreds of tasks, the meta-evaluation cost could rival or exceed the cost of policy training itself. This is analogous to the difficulty estimation problem in the reference paper, where generating 2,048 samples per question to estimate difficulty consumed more compute than the largest test-time budgets studied. In DreamGym's case, the cost is not in real environment interactions but in LLM inference compute: each rollout requires multi-turn generation from M_exp (typically 5–15 steps, each involving reasoning trace generation, state prediction, and retrieval). If 20% of the total synthetic generation budget goes to entropy estimation rather than policy training, the reported efficiency gains over real-environment RL would need to be adjusted accordingly, since real-environment RL does not incur a separate "task evaluation" tax—the training rollouts themselves double as the exploration data.

What evidence exists in the paper. None directly. The paper does not report n, does not ablate the tradeoff between more accurate entropy estimation (larger n) and more training rollouts, and does not include entropy estimation cost in any budget calculation. The training curve in Figure 3 (right) treats DreamGym with and without task generation, but the "without" variant entirely removes the component rather than varying n, so the cost-quality tradeoff at the margin is unknown.

Mitigation status. Not addressed. The paper acknowledges the unaccounted cost in Section 3.2 but offers no solution—no adaptive estimation scheme, no amortization strategy, no learned entropy predictor that could reduce the required number of rollouts. A natural mitigation (not explored) would be to reuse the training rollouts themselves for entropy estimation: since GRPO already samples G trajectories per task for advantage normalization, those same trajectories could provide the reward entropy signal without additional sampling. But this would only work for GRPO, not PPO, and would couple the estimation quality to the group size G rather than allowing independent control.

---

Hard Problems Where the Base Policy Has Near-Zero Success Probability Remain Unsolved

The assumption or constraint. DreamGym's training paradigm—and the curriculum generator in particular—depends on the agent having some non-zero probability of success on at least a subset of tasks. The reward-entropy heuristic (Equation 7) explicitly selects for tasks where the agent sometimes succeeds and sometimes fails (variance > 0). If the agent never succeeds on any task (all rewards are zero), the entropy is zero everywhere, the curriculum generator has no high-value seeds to vary, and the RL signal provides no positive examples for credit assignment. Theorem 1's bound confirms this dependency: if the initial policy π achieves near-zero real-environment value J_M(π), then even if ε_R and ε_P are small, the synthetic surrogate gain A_{M̂}^π(s,a) is near-zero (since there are no successful trajectories to provide positive advantages), and no amount of synthetic optimization can create capability that does not exist in the policy's behavioral repertoire.

The consequence. For genuinely hard environments where no reasonable base policy achieves above-chance performance—complex multi-step reasoning tasks, novel tool-use scenarios, environments with large combinatorial action spaces—DreamGym would provide no improvement over the base policy, regardless of synthetic training budget. This is the same boundary condition observed in the reference paper's difficulty bin 5 (hardest problems), where test-time compute provided essentially zero benefit because the base model simply could not produce correct solutions. In DreamGym's case, the failure mode is more subtle: the experience model could still generate plausible rollouts (the abstract states would look realistic, the transitions would be causally consistent), but those rollouts would never contain a successful trajectory, so the agent would receive no positive reinforcement signal to shape its behavior toward competence. The training would produce a policy that is well-optimized to generate coherent-looking but incorrect behavior—a form of reward hacking where the agent learns to satisfy the synthetic environment's consistency criteria without actually solving the task.

What evidence exists in the paper. The ALFWorld results in Table 1 provide suggestive evidence: DreamGym + PPO with Llama-3.2-3B achieves only 40.5% versus Traditional PPO at 47.0%, a 6.5-point gap. While this is not the near-zero regime described above, it shows that DreamGym's synthetic training struggles most on the smallest model (3B parameters) in the most complex environment (embodied control with multi-step object manipulation). The paper does not report a "difficulty bin" analysis analogous to the reference paper, so we cannot directly observe whether DreamGym's benefits concentrate on easier ALFWorld tasks while providing minimal improvement on the hardest ones. The WebArena results (9.1–14.5% success) suggest that even in a genuinely difficult environment, DreamGym finds some signal—but 85% of tasks remain unsolved, and whether further synthetic training would close this gap or asymptote at a low ceiling is unknown.

Mitigation status. Not directly addressed. The paper does not discuss the zero-signal regime or propose mechanisms for bootstrapping when reward entropy is uniformly zero. Potential mitigations—curriculum generation that starts with simplified versions of hard tasks (reducing the number of required steps, narrowing the search space), injecting expert demonstrations into the initial replay buffer more aggressively, or using the experience model itself to generate "hint" states that guide the agent toward success—are not explored.

---

The Abstract State Representation Requires Environment-Specific Engineering That Limits Portability

The assumption or constraint. DreamGym operates on an abstract textual state space S that is fundamentally different from raw environment observations. For this to work, two mappings must be designed for each new environment: (1) a raw-to-abstract mapping that converts real observations (HTML accessibility trees, rendered web pages, embodied scene descriptions) into the compact, semantic format the experience model expects, and (2) an abstract-to-action mapping that ensures the agent's actions in the abstract space correspond to executable actions in the real environment. The paper describes this as "the same rule-based mapping function or a lightweight fine-tuned model" (Section 4.3), with Appendix C.3 providing a specific prompt template for WebArena's AX-tree state mapping. However, the design of this mapping is a non-trivial engineering task: it requires understanding which elements of the raw observation are semantically meaningful (a navigation link, a product listing) versus which are irrelevant structural artifacts (CSS classes, <div> nesting, JavaScript hooks), and it must produce a consistent, token-efficient representation that preserves enough information for the agent to make correct decisions.

The consequence. The cost of adapting DreamGym to a new environment is not just the offline data collection and experience model fine-tuning discussed in Limitation 1, but also the design, implementation, and validation of the raw-to-abstract state mapper. For WebArena, this mapper is itself a fine-tuned LLM (Appendix C.3) that must learn to extract relevant elements from raw AXTree observations—requiring its own training data, hyperparameter tuning, and quality evaluation. For environments where the raw observation format differs substantially from WebArena's structured AXTree (e.g., pixel-based environments, multimodal observations with images and text, streaming API responses), the mapper would need to be redesigned from scratch, potentially requiring different model architectures (vision-language models for pixel inputs) or different abstraction strategies. This undermines the paper's framing of DreamGym as a "unified" and "general-purpose" framework: the experience model architecture is unified, but the interface between that model and real environments is environment-specific and requires substantial per-domain engineering.

What evidence exists in the paper. Appendix C.3 provides the full state mapping prompt for WebArena, revealing that the mapper is a prompted LLM that must reason about the page content, user instruction, and interaction history to produce a refined observation. This is not a simple rule-based mapping; it requires a capable model and careful prompt engineering. The paper does not report the accuracy of this mapper (how often it correctly includes all relevant elements, how often it introduces errors by omitting critical information or hallucinating element IDs), nor does it ablate the mapper quality against simpler alternatives. For WebShop and ALFWorld, the mapping details are not provided at all, leaving the engineering requirements for those domains opaque.

Mitigation status. Not addressed. The paper treats the state mapping as an implementation detail rather than a research contribution, but for practitioners, it represents a significant adoption barrier. The suggestion that the mapping can be "rule-based" or "lightweight fine-tuned model" understates the complexity, as evidenced by the fact that the WebArena mapper is itself a full LLM call per environment step. Future work on learning the mapping jointly with the experience model—or on designing experience models that can consume raw observations directly while still operating in an abstract reasoning space—would address this limitation.

---

Real-Environment Transfer Performance Depends on Unquantified State-Space Consistency, Which May Be Poor for Complex Domains

The assumption or constraint. DreamGym-S2R assumes that the policy trained in the abstract state space S can be transferred to the real environment with only a state-space mapping function. Theorem 1 provides theoretical justification: if ε_P (the total-variation distance between real and synthetic transition distributions) is small, the performance gap is bounded. However, ε_P is never measured empirically. The paper reports no experiment that directly evaluates how closely the experience model's predicted state distributions match real environment dynamics—no comparison of synthetic versus real state sequences for identical action trajectories, no measurement of how often the synthetic environment predicts a state that is semantically inconsistent with what the real environment would produce. The GPT-4o judge evaluation (Figure 4) assesses synthetic state quality along criteria like "causal consistency" and "informativeness," but these are evaluated against human (or rather, GPT-4o's) expectations of what a good state looks like, not against ground-truth real environment states. The gap between "looks causally consistent to a judge" and "matches the real environment's actual next-state distribution" is unquantified.

The consequence. The S2R results in Table 1 show that synthetic pre-training followed by limited real fine-tuning consistently outperforms pure synthetic training—this is evidence that the synthetic policy has not perfectly captured the real environment dynamics, and that real interaction provides additional value. However, the magnitude of this gap varies substantially across environments: on WebShop with Llama-3.1-8B + GRPO, S2R adds 11.1 percentage points over DreamGym-only (63.9% → 75.0%); on WebArena with the same configuration, S2R adds only 0.6 points (9.1% → 9.7%). This variation suggests that ε_P is substantially larger for WebArena than for WebShop—the synthetic environment is a less faithful proxy for the real dynamics—but without direct measurement, a practitioner cannot predict a priori whether S2R will provide large gains or negligible ones for their target environment. A pessimistic interpretation of the WebArena S2R results is that the synthetic policy has overfit to the experience model's idiosyncratic dynamics, and the 5K real transitions are insufficient to unlearn these idiosyncrasies—the policy is stuck in a local optimum of the synthetic environment that does not generalize.

What evidence exists in the paper. The S2R results in Table 1 provide indirect evidence of the state-space gap through the varying S2R improvement magnitudes. The cross-domain transfer results (Figure 3, middle) provide further indirect evidence: the fact that web-trained policies transfer between WebShop and WebArena but not to ALFWorld suggests that the abstract state space captures cross-environment similarities for similar domains but not for different modalities. However, no experiment directly compares synthetic states against real states for identical trajectories, which would provide a direct measurement of ε_P.

Mitigation status. The paper acknowledges the importance of "consistency of the state space between synthetic and real environments" (Section 4.3) and implements mapping functions to enforce it, but does not measure how effective those mappings are. A natural mitigation—periodically evaluating the synthetic-trained policy in the real environment during training to detect when overfitting to synthetic dynamics begins—is not implemented. The theoretical framework in Appendix B.1 provides the tools to bound performance degradation given estimates of ε_P, but these bounds remain theoretical because ε_P is never estimated in practice.

---

The Offline Training Data Requirements, While Modest, Depend on Strong Teacher Annotations That May Not Be Available for All Domains

The assumption or constraint. Training the experience model requires offline trajectory data augmented with reasoning traces generated by a "strong teacher LLM" (Section 4.1.2). For the three benchmark environments, the paper leverages publicly available demonstrations: human trajectories (WebShop), leaderboard agent outputs (WebArena), and expert demonstrations from existing frameworks (ALFWorld). The reasoning traces are then automatically generated by prompting a powerful LLM (unspecified, but the prompt templates in Appendix C suggest a model capable of sophisticated multi-step analysis of environment dynamics). This pipeline assumes: (1) some offline trajectory data exists for the target environment, (2) a teacher LLM exists that is sufficiently capable to generate faithful reasoning traces about that environment's dynamics, and (3) the teacher's reasoning is accurate enough that training on it produces a useful experience model rather than propagating the teacher's misconceptions.

The consequence. For novel or proprietary environments where no public trajectory data exists and no strong teacher model has been evaluated—custom enterprise web applications, internal tool suites, specialized robotic control interfaces—the training pipeline requires bootstrapping from scratch: someone must first collect demonstrations (human or heuristic), then validate that a teacher model can produce accurate reasoning traces for those demonstrations. If the teacher model frequently produces incorrect reasoning (e.g., misunderstanding what a particular UI element does, misattributing causality in a multi-step workflow, hallucinating plausible but factually wrong state transitions), the experience model distills these errors into its own behavior, producing synthetic rollouts that are internally consistent but factually misaligned with the real environment—the worst-case scenario where ε_P is large but invisible to any evaluation that does not compare against ground-truth dynamics. Figure 5 shows that the experience model can succeed with limited offline data (2K–20K samples), but this experiment uses high-quality data collected from the actual target environments—it does not test the scenario where the offline data itself is scarce, noisy, or from a different distribution than the target.

What evidence exists in the paper. Figure 5 demonstrates data efficiency—competitive performance with as few as 2K–10K offline samples—but all samples come from the target environments with high-quality annotations. The paper does not ablate reasoning trace quality: there is no experiment comparing SFT on teacher-generated reasoning traces versus SFT on automatically extracted reasoning (e.g., simple templates like "the agent clicked X, so the page navigated to Y") versus no reasoning at all during experience model training. The no-reasoning ablation in Table 2 and Figure 4 removes reasoning at inference time, not at training time—the model is still trained on reasoning-augmented data, it just does not generate reasoning during rollout collection. The value of the teacher annotations specifically, versus simpler forms of augmentation, is therefore unquantified.

Mitigation status. The paper does not address the teacher dependency. It presents the pipeline as if strong teacher annotations are a given, without discussing what happens when they are unavailable, unreliable, or expensive to obtain. For domains where a capable teacher model exists and can be prompted effectively (the likely case for many web and software environments given the capabilities of current frontier models), this limitation is minor. For domains where no such teacher exists—highly specialized technical domains, low-resource languages, environments requiring multimodal understanding that current text-only LLMs lack—this is a hard barrier to adoption. The paper's suggestion in Section 7 to build a "universal world model" trained across many environments could partially address this by enabling transfer of reasoning capabilities across domains, but this remains speculative.

7. Implications and Future Directions

How This Work Changes the Landscape

DreamGym introduces a fundamental reframing of how the field should think about synthetic training environments for LLM agents. The dominant assumption in prior work—from Dreamer-style world models (Hafner et al., 2020) through WebDreamer (Gu et al., 2024) and UI-Simulator (Wang et al., 2025b)—has been that synthetic environments must approximate real environments in terms of pixel-level or raw-observation fidelity to be useful for training. DreamGym's theoretical analysis (Theorem 1, Appendix B.1) and empirical results break this assumption cleanly: policy improvement in real environments depends on reward accuracy and domain-consistent transitions, not on perceptual similarity between synthetic and real states. This is not an incremental improvement in simulation quality but a redefinition of what "simulation quality" means for agent learning—shifting the optimization target from "does the synthetic environment look like the real one?" to "does the synthetic environment preserve the causal and reward structure that drives effective credit assignment?"

The magnitude of this shift is best understood by what it makes newly tractable. Before DreamGym, training RL agents on WebArena was effectively impossible—the paper's traditional GRPO and PPO baselines achieve 4.8–7.3% success across backbones, barely above random behavior, despite "extensive engineering effort" including managing AWS servers, manual resets, and handling evaluation bugs (Appendix A.3). This failure mode was not primarily algorithmic—the RL methods themselves are well-understood—but infrastructural: the real environment simply cannot provide the throughput, consistency, and reward density that modern policy gradient methods need. DreamGym breaks this deadlock by demonstrating that an environment purpose-built for RL, operating in an abstract textual state space, can produce training data that is more effective for learning than real environment data when the real environment is too noisy, sparse, or slow. The WebArena results (9.1–14.5% vs. 4.8–7.3% for baselines) are not just an efficiency gain—they represent opening a door that was previously closed, making RL-based agent improvement possible in a domain where it was not merely expensive but functionally impossible.

This work also reconciles a latent contradiction in the agent training literature. On one side, work on synthetic trajectory generation for SFT (Ou et al., 2024; Pahuja et al., 2025; Xu et al.) showed that synthetic data could augment or replace human demonstrations for behavioral cloning. On the other side, attempts to use synthetic data for RL training were largely pessimistic—world models trained on real observations overfit to their training distribution, and RL's exploratory nature systematically finds and exploits simulator inaccuracies (the classic sim-to-real gap). The resolution DreamGym offers is that RL needs qualitatively different properties from its training data than SFT does. SFT can succeed with static, off-policy demonstrations because it learns a direct mapping from states to actions. RL needs on-policy exploration, informative reward signals, and causally consistent transitions—properties that raw-observation world models struggle to provide but that a reasoning-based model operating at the right level of abstraction can deliver reliably. The fact that DreamGym matches real-environment RL performance on WebShop and dramatically exceeds it on WebArena, while prior synthetic data methods for agents were limited to SFT augmentation, validates this diagnosis.

The paper also shifts research priorities in a concrete way. If the primary bottleneck for RL-based agent training is not algorithmic innovation (new RL variants) nor computational scale (bigger models, more GPUs) but rather the quality and structure of interaction data, then investment should flow toward better experience models—improving their reasoning fidelity, expanding their domain coverage, reducing their offline data requirements—rather than toward marginal improvements in policy optimization or larger-scale real-environment infrastructure. This is analogous to how the pretraining scaling laws literature (Hoffmann et al., 2022) redirected investment from "always train the biggest model" to "train the right-sized model on more data"—DreamGym similarly redirects investment from "always interact with the real environment" to "build the right synthetic environment for the learning problem."

A subtler but important implication: DreamGym demonstrates that the experience model and the policy can co-evolve productively without either one needing to be perfect. The replay buffer mechanism—freezing the experience model parameters after offline training while continuously updating the retrieval buffer with synthetic trajectories—creates a stable dynamical system where the policy improves because the experience model provides consistent learning signals, and the experience model's predictions remain relevant because the buffer adapts to the policy's changing behavior. This design pattern—a fixed world model with adaptive retrieval-based conditioning—may generalize beyond the specific LLM-based implementation in DreamGym to other domains where learned simulators face the dual challenges of accuracy and adaptability.

Follow-Up Research This Work Enables

Measuring and minimizing ε_P directly rather than relying on downstream RL performance as a proxy. Theorem 1 establishes that the total-variation distance between real and synthetic transition distributions (ε_P) is the critical quantity governing policy transfer quality, with a 1/(1-γ)² penalty that makes it the dominant error term for long-horizon tasks. Yet the paper never measures ε_P empirically—it evaluates the experience model indirectly through downstream RL success rates and GPT-4o judge scores (Figure 4). A direct measurement would sample a fixed set of action sequences, execute them in both the real environment and DreamGym, and compare the resulting state distributions (or, more practically, compare whether the same task outcomes are achieved). This would provide the first empirical calibration of how DreamGym's reasoning-based design affects the error term that its own theory identifies as critical. A strong follow-up would measure ε_P across environments (WebShop vs. WebArena vs. ALFWorld), correlate it with the observed S2R improvement gaps (which vary from 0.6 to 11.1 percentage points in Table 1), and use that correlation to predict a priori how much real fine-tuning a new environment will need—turning the S2R decision from "try it and see" to a principled budget allocation problem.

Experience model scaling laws: how does downstream RL performance scale with experience model size, training data, and reasoning depth? The paper uses Llama-3.1-8B for all main experiments, with a single ablation (Figure 5) showing that Llama-3.2-3B underperforms the 8B model but improves with more data. This is a two-point scaling curve—insufficient to understand the relationship. A systematic scaling study would train experience models at multiple scales (1B, 3B, 8B, 70B parameters) with multiple data budgets (1K, 5K, 20K, 100K transition steps) and measure both direct experience model quality (using the GPT-4o judge criteria from Figure 4) and downstream RL performance after training agents with each experience model. The key question: does downstream RL performance saturate at the same experience model quality that direct evaluation suggests, or does RL training amplify subtle differences in transition quality that direct evaluation misses? The paper's own results provide suggestive evidence for amplification—the no-reasoning ablation causes an 8.1% drop on WebShop (Table 2) even though Figure 4 shows the no-reasoning variant still scores moderately on GPT-4o judge metrics—but a systematic study would clarify whether the relationship is smooth or thresholded.

Can DreamGym's abstract state representation be learned jointly with the experience model rather than engineered per-environment? The current system requires a separate raw-to-abstract state mapper for each environment—a rule-based function or fine-tuned model that converts raw observations (HTML AXTree, embodied scene descriptions) into the compact element-listing format the experience model expects. For WebArena, this mapper is itself a prompted LLM (Appendix C.3). This per-environment engineering is arguably the largest barrier to deploying DreamGym in new domains. A natural extension would train the experience model to consume raw observations directly while still producing abstract, reasoning-conditioned transitions—essentially folding the state mapper into the experience model itself. The training data would be pairs of (raw observation, action, next raw observation, reward) from offline trajectories, with the model learning to compress raw observations into an implicit abstract representation through the bottleneck of generating reasoning traces and next-state predictions. The experiment would compare this end-to-end trained model against the current two-stage pipeline (separate mapper + DreamGym) on both training efficiency (how much offline data is needed?) and downstream RL performance (does end-to-end training lose the benefits of the explicit abstraction?). A positive result would dramatically reduce the adoption cost of DreamGym for new environments; a negative result would clarify that explicit, human-designed abstraction is a necessary ingredient, not an implementation convenience.

Stress-testing DreamGym on environments where the base policy has near-zero success probability. The curriculum generator's reward-entropy heuristic (Equation 7) selects tasks where the agent sometimes succeeds and sometimes fails. If the agent never succeeds on any task—the cold-start problem—the entropy is uniformly zero and the curriculum provides no signal. This is not a hypothetical failure mode: the ALFWorld results with Llama-3.2-3B + PPO show DreamGym achieving only 40.5% (Table 1), and the paper's cross-domain transfer experiment (Figure 3, middle) shows catastrophic failure when transferring from web to embodied domains. A deliberate stress test would take an environment where the base policy achieves near-zero success (perhaps a deliberately difficult subset of WebArena tasks, or a completely novel tool-use benchmark) and measure whether DreamGym can bootstrap from zero or whether it requires an initial "priming" phase with expert demonstrations to create non-zero reward entropy. The experiment would test three bootstrapping strategies: (1) injecting a small number of expert demonstrations into the initial replay buffer (giving the experience model examples of successful trajectories even if the agent hasn't produced any), (2) using the experience model to generate simplified versions of hard tasks where success is more likely, and (3) using an exploration bonus or curiosity-driven objective to encourage the agent to discover successful trajectories. The result would define the boundary conditions for DreamGym's applicability: can it create capability (by generating synthetic success experiences that teach the agent new behaviors) or can it only amplify existing capability (by providing more consistent learning signals for behaviors the agent already occasionally exhibits)?

Multi-environment experience models: training a single DreamGym that serves WebShop, WebArena, and ALFWorld simultaneously. The paper's cross-domain transfer experiment (Figure 3, middle) shows that a policy trained on WebShop transfers to WebArena but not to ALFWorld, indicating that the abstract state space captures similarities within web domains but not across modalities. The open question is whether a single experience model trained on mixed-domain data (trajectories from all three environments, with domain tags in the context) could serve as a universal agent training substrate, or whether the differing causal structures of web navigation, embodied control, and tool use are fundamentally incompatible in a single abstract state space. The experiment would train a single Llama-3.1-8B experience model on the union of all offline trajectory datasets (~10K total transitions across environments), then train separate agent policies for each environment using this shared experience model, measuring whether performance matches or exceeds environment-specific DreamGym models. A positive result (shared model matches per-environment models) would be evidence that the reasoning mechanism abstracts environment-specific dynamics into a general "interaction schema" that transfers across domains—a step toward the "universal world model" the paper speculates about in Section 7. A negative result (shared model underperforms per-environment models) would quantify the cost of domain mixing and suggest that the abstract state space, while more general than raw observations, still encodes domain-specific causal structure.

Can DreamGym serve as a data engine for self-improvement loops? The paper demonstrates that synthetic rollouts can train agents via online RL, but does not close the loop: can the improved agent's trajectories be used to further train a better experience model, which in turn trains an even better agent? This is the self-play paradigm that has driven progress in games (Silver et al., 2016) and is proposed for LLM reasoning (Liu et al., 2025), but has not been demonstrated for agent training with synthetic environments. The experiment would iterate: (1) train agent π_θ in DreamGym v1, (2) use π_θ to collect synthetic trajectories on new tasks, (3) fine-tune the experience model on these trajectories (using the agent's successes and failures as new training data, with reasoning traces generated by a teacher LLM or by π_θ itself), producing DreamGym v2, (4) train a new agent in DreamGym v2, and repeat. The key measurement is whether the experience model improves over iterations (measured by GPT-4o judge quality and by downstream RL performance) or whether synthetic self-training causes the experience model to drift away from real environment dynamics—a form of model collapse analogous to the degradation observed when LLMs are trained on their own outputs. The paper's fixed experience model design (trained once, frozen during RL) avoids this risk entirely, but a self-improvement loop would need to address it explicitly, perhaps through periodic recalibration against the original offline data or through conservative fine-tuning objectives.

Practical Applications and Downstream Use Cases

Enabling RL-based improvement for agents deployed on proprietary or internal web applications. Many organizations operate complex internal web platforms—customer relationship management systems, enterprise resource planning dashboards, proprietary content management tools—where off-the-shelf LLM agents perform poorly because the interfaces, workflows, and terminology are domain-specific and not represented in public training data. Training these agents via RL in the real application is typically infeasible: the systems lack programmatic APIs for agent interaction, actions may have irreversible side effects (submitting orders, modifying customer records, triggering notifications), and the throughput required for RL-scale data collection would overwhelm backend services. DreamGym's pipeline offers a practical solution: collect a modest number of demonstrations (a few hundred to a few thousand trajectories from human users or from logged production interactions), annotate them with reasoning traces using a strong teacher LLM, train a DreamGym experience model specific to the internal application, and then train the agent policy entirely in the synthetic environment. The paper's data efficiency results (Figure 5: competitive performance with 2K–10K offline samples) suggest this is within reach for many organizations. The agent can be iteratively improved through synthetic RL without ever interacting with the production system, with optional limited real-environment fine-tuning (DreamGym-S2R) as a final validation step. The key metric from the paper: on WebShop, synthetic-only training matches real-environment RL performance (63.9% vs. 65.0% for GRPO with Llama-3.1-8B, Table 1), suggesting that for web-based tasks with similar structure to the evaluated benchmarks, synthetic training may be sufficient without any real interaction at all.

Cost-efficient data generation for training specialized coding or tool-use agents. The paper focuses on web navigation and embodied control, but the underlying mechanism—a reasoning-based experience model that simulates multi-turn interactions with consistent dynamics and reward signals—is directly applicable to code execution environments, API interaction benchmarks, and tool orchestration tasks. In these domains, the "real environment" is a code interpreter, a set of API endpoints, or a command-line interface—still programmatic, but expensive to query at RL scale due to rate limits, computational cost of execution, or the need for sandboxed environments with resets. A DreamGym experience model trained on a corpus of (code, execution output, error messages) or (API call, response, subsequent state) trajectories could generate unlimited synthetic rollouts for RL training at the cost of LLM inference rather than real execution. The paper's results suggest this would be particularly valuable for medium-difficulty tasks where the agent has non-zero but imperfect success (the regime where reward entropy is high and curriculum generation is most effective). A concrete deployment: fine-tune a coding agent using DreamGym for a specific library or framework where public training data is scarce but a modest corpus of example scripts and their outputs exists (from documentation, tutorials, or internal repositories). The experience model learns to simulate the library's behavior from these examples, and the agent improves through synthetic RL without needing access to a live execution environment.

Warm-starting RL for embodied robotics or physical systems where real-world interaction is expensive and risky. While the paper shows that DreamGym's cross-domain transfer fails from web environments to ALFWorld (Figure 3, middle), this does not preclude building DreamGym experience models specifically for embodied domains using offline embodied trajectory data. The ALFWorld results demonstrate that DreamGym works for text-based embodied control (62.1–71.0% with GRPO, Table 1), albeit with a larger gap to real-environment RL than on WebShop (3.2–8.8 points). For physical robotics, where each real-world interaction costs minutes to hours of supervised operation and failed actions can damage equipment or create safety hazards, DreamGym-S2R's recipe—synthetic pre-training with large-scale exploration followed by minimal real fine-tuning (5K transitions, a 16× reduction from full real RL)—is economically compelling even if the synthetic-only policy is imperfect. The 5K real transitions in WebShop represent roughly 500 trajectories (assuming ~10 steps each), which for a physical robot might correspond to a few days of supervised operation rather than the months required for 80K transitions of from-scratch RL. The paper's evidence that S2R consistently outperforms both pure synthetic and pure real training (Table 1) suggests this hybrid approach is the sweet spot: synthetic training provides broad exploration and a strong initialization, and a small amount of real interaction corrects the residual sim-to-real gap.

Automated curriculum design for agent training benchmarks. The paper's curriculum task generator—which identifies high-entropy tasks and produces progressively harder variations—addresses a persistent bottleneck in agent evaluation: benchmark designers must manually create diverse, challenging tasks that cover the range of capabilities they want to measure. DreamGym demonstrates that this process can be automated for the environments it models, generating task variations that are "challenging but feasible" and targeted at the agent's current capability frontier. For benchmark maintainers, this suggests a pipeline where a base set of seed tasks is expanded automatically through DreamGym-style variation generation, with the resulting tasks validated not by human experts (the traditional bottleneck) but by the experience model's ability to produce coherent multi-turn rollouts for them. The generated tasks can then be used for both training (as in the paper) and evaluation (by executing the trained agent in the real environment on held-out generated tasks). The key result supporting this: removing the task generator causes a 6.0–6.6% absolute drop in success rate (Table 2), confirming that the generated tasks provide learning signal beyond what the original benchmark tasks alone offer. The next step would be to validate that agents trained on DreamGym-generated task variations generalize to entirely novel human-designed tasks—closing the loop from automated generation to real-world capability improvement.