Agent Learning via Early Experience

ArXiv: 2510.08558

🎯 Pitch

Language agents can teach themselves to be better without rewards or human labels—just by observing what happens when they take different actions and reflecting on their own mistakes. Across eight diverse environments, this 'early experience' approach matches full imitation learning with only 12% of the expert data and, when combined with RL, achieves scores that expert-only training can’t reach.

---

1. Executive Summary

This paper proposes the early experience paradigm, a middle-ground training regime between imitation learning and reinforcement learning that enables language agents to learn from their own actions and the resulting future states without requiring external reward signals. The authors study two strategies under this paradigm across eight diverse environments (ALFWorld, WebShop, BFCLv3, Tau-Bench, SearchQA, ScienceWorld, TravelPlanner, WebArena-Lite) and three model families (Llama-3.2-3B, Qwen-2.5-7B, Llama-3.1-8B): (1) Implicit World Modeling, which trains the policy to predict next states from alternative actions (e.g., predicting the error message after entering an invalid date on a booking site), and (2) Self-Reflection, where the agent generates chain-of-thought explanations comparing its suboptimal actions to expert choices and learns from those contrastive rationales (e.g., explaining why clicking a $15 blue shirt satisfies a budget constraint while a$30 red shirt does not). Both methods consistently outperform imitation learning baselines across all environments, with Self-Reflection delivering the largest gains on long-horizon reasoning tasks (e.g., +15.0 percentage points on TravelPlanner with both Llama-3.1-8B and Qwen-2.5-7B) and Implicit World Modeling excelling where environment dynamics are stable (e.g., +18.4 percentage points on WebShop with Llama-3.2-3B). When used to warm-start reinforcement learning with GRPO, early-experience checkpoints achieve higher post-RL ceilings than imitation-only starts (e.g., 92.2% vs. 82.0% on WebShop with Llama-3.2-3B), establishing that the benefits of this reward-free supervision carry over into reward-driven optimization while scaling effectively to larger models and requiring as little as one-eighth of the expert demonstrations to match full-dataset imitation learning.

2. Context and Motivation

The Core Problem: Language Agents Learn Passively from Human Data, Not from Experience

The paper addresses a fundamental asymmetry in how language agents are trained: current systems overwhelmingly learn from static, expert-curated demonstrations, while a core aspiration of AI — articulated since the field's founding (Russell and Norvig, 1995) — is for agents to learn actively from their own interactions with the environment. The paper frames this aspiration as the progression toward an "era of experience" (Silver and Sutton, 2025), where agents improve through trial and error, but observes that the field is stuck in an earlier paradigm.

The specific gap is the absence of a practical mechanism for agents to learn from their own behavior in environments that lack verifiable reward signals. This is not a niche problem — it is the default condition in most real-world settings where language agents are deployed. When a language agent navigates a website to book flights, fills out forms on a government portal, or uses internal enterprise tools, the environment rarely provides explicit feedback about whether each action was correct. A form may appear to submit successfully, but the agent receives no signal about whether each field was completed accurately. A tool call may return syntactically valid output without indicating whether the right tool was invoked for the task.

The authors articulate this directly in Section 1:

"Many environments of interest lack verifiable or dense reward signals, especially in open-ended settings such as websites where platforms do not expose ground truth feedback. For example, a form may appear to be submitted successfully, but the agent receives no indication of whether each piece of information was filled out correctly."

This gap is both practically significant and theoretically limiting. Practically, it means that building capable language agents for real-world tasks requires continuous, expensive human annotation — a bottleneck that does not scale. Theoretically, it means that agents trained purely via imitation learning never experience the consequences of their own decisions, leaving them brittle to distribution shift (when deployed states diverge from those seen during training) and unable to recover from errors (since they never learned what happens after a mistake).

Why This Problem Matters Now

The paper's motivation is driven by three converging trends that make this gap urgent:

1. Language agents are being deployed in increasingly complex, open-ended environments. The environments studied in this paper — WebArena (Zhou et al., 2024) for web navigation, BFCLv3 (Patil et al., 2025) for multi-turn tool use, TravelPlanner (Xie et al., 2024a) for long-horizon planning — represent a new class of task that goes far beyond the controlled simulators where classical RL succeeded. These environments have combinatorial action spaces, long interaction horizons, and noisy text-based observations. Critically, most of them do not expose reward functions. An agent booking a multi-day trip with budget constraints must satisfy dozens of implicit requirements (cuisine preferences, room rules, round-trip completion), but receives no step-by-step feedback about whether it is progressing correctly. The gap between what these environments demand and what current training paradigms provide is widening.

2. The dominant training paradigm — supervised fine-tuning on expert demonstrations — has structural limits. The paper identifies several specific failure modes of imitation learning for language agents (Section 3.1):

• Distribution shift (Ross et al., 2011): An agent trained to mimic expert actions will inevitably deviate from the expert policy during deployment. When it enters states not covered in training data, errors compound because the agent has no model of what constitutes a "good" or "bad" outcome from that state. It has only seen expert trajectories through successful paths, never the consequences of deviating.

• Lack of action-consequence awareness: The agent "never observes what happens when it takes non-expert actions; it only sees expert state-action pairs without experiencing the outcomes of alternative choices. This limits its ability to recover from errors or reason about why certain actions fail" (Section 3.1).

• Scaling bottlenecks: Expert demonstrations are expensive to collect (human annotation) or limited in quality (synthetic generation from larger models). The paper cites Qi et al. (2025) to note that SFT "remains limited by the cost of high-quality demonstrations" and Chu et al. (2025) to highlight that SFT-trained models fail to generalize to novel states.

These limitations are not theoretical conjecture — they surface concretely in the paper's experiments. In Table 2, the zero-shot prompting performance of instruction-tuned models is disastrous on most benchmarks (e.g., 0.0% on TravelPlanner and WebShop for Llama-3.1-8B), and even imitation learning on expert data struggles on complex tasks (17.2% on TravelPlanner, 4.9% on WebArena-Lite with Llama-3.1-8B). The models have seen the right actions but cannot transfer that knowledge to the evaluation distribution.

3. Reinforcement learning is not yet viable at scale for language agent environments. The paper acknowledges RL as the long-term vision — "The envisioned Era of Experience builds upon environments with verifiable rewards, using them as the primary supervision for reinforcement learning" (Figure 1 caption) — but provides a detailed diagnosis of why current RL infrastructure falls short for language agents (Section 2.1):

• Reward unavailability: Many production environments simply do not provide reward signals. Websites do not tell you whether you booked the right flight; API environments do not indicate whether the right sequence of tool calls was used.

• Inefficient long-horizon rollouts: Even when rewards exist (e.g., task completion), they are sparse and delayed. A travel planning task might involve dozens of actions before any feedback is received, making credit assignment extremely challenging.

• Infrastructure immaturity: "Most real-world language agent environments lack reliable simulators, standard reset mechanisms, and scalable evaluation platforms, making large-scale RL training for language agents costly and brittle" (Section 2.1).

• Empirical brittleness: Current RL for language agents "remains difficult to apply effectively" and "still exploratory," with practitioners relying on "approximate rewards produced by larger teacher models" or "carefully curated reward functions and hand-tuned training recipes" to maintain stability (Section 2.1).

The paper's experiments confirm this diagnosis in Section 5.4, where applying GRPO directly from a raw pretrained model without any supervised warm-start "performs worst across all tasks and shows unstable training dynamics, highlighting the necessity of a strong initialization." RL unguided by any prior training fails in these environments — it needs a foundation.

The Missing Middle Ground

The paper's central conceptual contribution is identifying a vacant territory between the era of human data (imitation learning) and the era of experience (full RL). This territory — which the paper names early experience — is characterized by a specific capability: the agent can interact with the environment and observe the consequences of its actions, but it receives no explicit reward. Its own proposed actions and the resulting future states constitute experience that can be used for learning, even without a scalar feedback signal.

This middle ground has been largely overlooked. Prior work that attempted to use interaction data for agent training fell into two categories, both of which the paper argues are insufficient:

Inference-time self-correction without parameter updates. Methods like Reflexion (Shinn et al., 2023) and Self-Refine (Madaan et al., 2023) use prompting to have models revise their outputs or reflect on their reasoning at test time. The paper cites evidence that these approaches largely fail on reasoning tasks without access to external feedback: "later studies (Huang et al., 2024; Valmeekam et al., 2023) show that such inference-time methods often fail without access to external feedback (e.g., rewards)" (Section 2.2). Crucially, these methods do not update model parameters — they rely on the model's existing capabilities being sufficient for self-diagnosis, which the evidence does not support.

World models as separate simulators. Prior work on world models for language agents (Gu et al., 2025; Guo et al., 2025; Chae et al., 2025) trains separate models to predict state transitions for planning purposes. The paper notes that "most of these systems still treat the world model as a separate simulator, echoing classical control pipelines" (Section 2.2). This adds architectural complexity and planning overhead without integrating predictive knowledge directly into the policy.

Rationale-based bootstrapping without interaction. Approaches like STaR (Zelikman et al., 2022) generate rationales for correct answers and train on those rationales, but do so without interacting with the environment or exploring alternative actions. The paper's analysis in Section 6.1 shows that applying STaR-style data to these environments is ineffective — the generated rationales are "ungrounded, having never been tested in the environment, and frequently hallucinate tools or facts, so fine-tuning on them can even degrade performance." Table 4 shows that STaR on WebShop with Llama-3.1-8B achieves only 25.0% success (vs. 47.3% for imitation learning), a substantial regression.

How This Paper Positions Itself

The paper positions early experience as a practical bridge between imitation learning and reinforcement learning, not as a replacement for either. The positioning has several layers:

Relative to imitation learning: Early experience is an augmentation, not a replacement. The methods still require expert demonstrations as a starting point — they are seeded from $D_{\text{expert}}$ (Section 4.1). But by allowing the agent to branch off from expert trajectories, propose its own actions, and observe the consequences, early experience provides additional, complementary supervision that imitation learning cannot offer: knowledge of what happens when you deviate from the expert path. This directly addresses the distribution-shift brittleness and action-consequence blindness that plague pure imitation learning.

The paper demonstrates this complementarity empirically. Figure 4(a) shows that with only 1/8 of the expert demonstrations, early experience methods trained on WebShop surpass full-dataset imitation learning (25.8% → 38.3% with IWM, vs. 45.3% for full-dataset imitation). The expert data provides the correct paths; early experience provides the contrastive knowledge of what happens on incorrect paths.

Relative to reinforcement learning: Early experience is positioned as a pre-training or warm-start stage, not a competitor. The paper explicitly states that the long-term goal is the era of experience where RL is the primary training mechanism, but that current environments are not ready for that paradigm. Early experience "positions itself as a practical and general foundation for building more capable language agents in the upcoming era of experience" (Section 7). Section 5.4 demonstrates this concretely: checkpoints trained with early experience methods serve as better initializations for GRPO than imitation-only checkpoints, achieving higher final performance under identical RL recipes.

This is a nuanced claim. The paper is not arguing that early experience is better than RL — it is arguing that early experience is a necessary intermediate step given current infrastructure limitations, and that it compounds with RL rather than being superseded by it. The results in Figure 3 show that the performance gap from early experience persists or even widens after RL training, suggesting that the internalized knowledge about environment dynamics and decision principles cannot be fully recovered by reward optimization alone.

Relative to test-time compute scaling: Section 6.1 explicitly compares early experience against longer chain-of-thought reasoning at inference time (inspired by Snell et al., 2024). The results in Table 4 are striking: applying Long CoT to imitation-trained models dramatically reduces performance on WebShop (47.3% → 0.0%) and ALFWorld (80.5% → 25.8%). The paper's diagnosis is that "once fine-tuned only on expert trajectories lacking inherent rationales, models lose the ability to sustain coherent long-form reasoning, so extended chains often drift or collapse into invalid/off-policy actions." This emphasizes that the paper's contribution is about training-time learning from interaction, not inference-time prompt engineering — a clear positioning against a competing hypothesis that simply "thinking harder" at test time could achieve similar gains.

The unifying principle: Both of the paper's methods — implicit world modeling and self-reflection — are instantiations of the same core idea: the agent's own actions and the resulting future states constitute free supervision. The paper formalizes this in Section 4.1 with the $D_{\text{rollout}}$ dataset: for each expert state $s_i$, the agent proposes $K$ alternative actions, executes them in the environment, and collects the resulting next states $s_i^j$. These $(s_i, a_i^j, s_i^j)$ triples require no human annotation, no reward engineering, and no model-based relabeling — they are automatically generated by the agent interacting with the environment. The two methods differ only in how they convert these triples into training objectives: next-state prediction (Equation 3) versus contrastive rationales (Equation 4).

This unified framing is the paper's answer to the question: "How can we train agents to grow from their own experience, without any external reward signals?" (Section 1). The answer is to treat the environment's state transitions as supervision — whether for predictive learning (IWM) or for distilling decision principles (SR).

Where This Fits in the Broader Landscape

The paper situates early experience within a historical progression that mirrors the evolution of RL itself. Traditional RL systems like AlphaGo (Silver et al., 2016) used model-free policy optimization against verifiable game rewards. Model-based RL introduced world models (Sutton, 1991; Ha and Schmidhuber, 2018) to reduce sample complexity by learning transition dynamics. Early experience borrows the spirit of these ideas but adapts them to a domain where neither rewards nor explicit simulators are available: the world model is trained implicitly within the policy rather than as a separate component (Section 4.2), and the "reflection" serves as a form of credit assignment without numeric reward (Section 4.3).

The paper also draws an implicit parallel to multi-task and meta-learning paradigms. By exposing the agent to diverse state transitions from its own exploratory actions, early experience functions as a form of unsupervised environment exploration that builds generalizable representations. This is why the OOD results in Section 5.3 show gains that sometimes exceed in-domain gains — the agent has learned principles about the environment (e.g., "tool availability varies across settings," "filtering constraints matter") that transfer across distribution shifts.

In summary, the paper addresses a specific, well-defined gap: the absence of mechanisms for language agents to learn from their own interactions in reward-free environments. It diagnoses why current approaches (imitation learning, inference-time correction, separate world models) fail to fill this gap, and positions early experience as a practical intermediate step — one that provides immediate performance gains, scales to diverse environments and model families, and serves as an effective foundation for the eventual transition to full experience-driven learning when RL infrastructure matures.

3. Technical Approach

3.1 Reader Orientation

This paper builds a training pipeline that enables language agents to learn from their own interactions with reward-free environments by converting the agent's proposed actions and the resulting future states into supervision signals, without requiring any external reward function or additional human annotation. The system solves the problem of how to train agents beyond static expert demonstrations when verifiable rewards are unavailable, by constructing two complementary training objectives from exploration data—next-state prediction (implicit world modeling) and contrastive chain-of-thought reasoning (self-reflection)—that both derive their supervision from the same underlying principle: the environment's state transitions encode implicit knowledge about action quality, task constraints, and environment dynamics that can be extracted without numeric feedback.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components organized in a pipeline that starts from expert demonstrations and produces a trained policy:

1. Expert Dataset ($D_{\text{expert}}$): A collection of $(s_i, a_i)$ state-action pairs from successful human or model-generated trajectories across diverse environments. This provides the "correct path" knowledge that seeds the entire process.

2. Rollout Collector: For each expert state $s_i$, the current policy $\pi_\theta$ proposes $K$ alternative actions $a_i^1, a_i^2, \ldots, a_i^K$ (sampling from its own output distribution at various temperatures or uniformly from admissible actions). Each alternative action is executed in the actual environment, and the resulting next state $s_i^j$ is recorded. The output is $D_{\text{rollout}} = \{(s_i, a_i^j, s_i^j)\}$, a dataset of state-action-next-state triples that represents the agent's "early experience"—what happens when it deviates from the expert path.

3. Two Training Strategies, both consuming $D_{\text{rollout}}$ but converting it into different forms of supervision:

   • Implicit World Modeling (IWM): Reformats each $(s_i, a_i^j, s_i^j)$ triple as a next-token prediction task: the model takes the state-action pair $(s_i, a_i^j)$ as input and learns to generate the resulting next state $s_i^j$. This is trained as a first stage (one epoch), followed by standard imitation learning on $D_{\text{expert}}$ for the remaining training budget.
   • Self-Reflection (SR): For each expert state $s_i$, compares the expert action $a_i$ (and its resulting next state $s_{i+1}$) against the alternative actions $a_i^j$ (and their next states $s_i^j$), prompts a language model to generate a chain-of-thought explanation $c_i^j$ of why the expert action was preferable, and constructs a training dataset $D_{\text{refl}} = \{(s_i, a_i^j, c_i^j)\}$ where the target is the concatenated sequence $c_i^j \circ a_i$. This is mixed with $D_{\text{expert}}$ and trained jointly.

4. Trained Policy ($\pi_\theta$): The final output is a language model that has internalized both expert behavior and the consequences of non-expert actions, enabling better task performance, out-of-domain generalization, and serving as a stronger initialization for subsequent reinforcement learning.

Information flows as follows: expert demonstrations feed into the rollout collector → the rollout collector executes alternative actions in the environment and records next states → the two training strategies transform these triples into language-model training sequences → the model is fine-tuned on these sequences → the trained policy can be optionally further optimized with RL (GRPO).

3.3 Roadmap for the Deep Dive

• First, the formal MDP setup (Section 3 in the paper) and the limitations of imitation learning, because these establish the mathematical language and the precise failure modes that early experience addresses. Understanding why the standard supervised loss fails motivates the specific design of the alternative objectives.

• Second, the notation and data collection mechanism for early experience (Section 4.1), since this is the shared infrastructure that both IWM and SR depend on. The $D_{\text{rollout}}$ dataset is the concrete instantiation of "early experience"—how it is constructed, what properties it has, and why those properties matter.

• Third, implicit world modeling (Section 4.2): the training objective, the two-stage pipeline, the resulting capabilities, and the design choices that distinguish it from external world models.

• Fourth, self-reflection (Section 4.3): the prompt template, the contrastive data construction, the joint training with expert data, and why this works when purely inference-time self-correction fails.

• Fifth, the training and evaluation methodology (Section 5.1), detailing the specific models, datasets, hyperparameters, and evaluation protocols used across all eight environments, since these are critical for reproducibility and understanding the scale of the experiments.

• Sixth, the reinforcement learning warm-start integration (Section 5.4), explaining how early-experience checkpoints are used to initialize GRPO and what this reveals about the relationship between reward-free and reward-driven learning.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical methods paper that formalizes a new training paradigm and evaluates two concrete instantiations of it. The core technical idea is that environment state transitions—observed when the agent executes its own proposed actions—encode implicit supervision that can be extracted through either predictive learning (IWM) or contrastive rationalization (SR), both of which are purely supervised objectives operating on automatically collected data.

---

MDP Formalization and the Imitation Learning Baseline

The paper formalizes language agent decision-making as a Markov Decision Process (MDP) defined by the tuple $M = (S, A, T, R, \gamma, \rho_0)$.

$M = (S, A, T, R, \gamma, \rho_0)$

where $S$ is the state space (encoding environment configurations accessible to the agent, such as webpage contents, tool outputs, or textual environment descriptions), $A$ is the action space (discrete choices such as clicking elements, invoking tools, or generating text responses), $T: S \times A \rightarrow \Delta(S)$ is the transition function governing state dynamics (where $\Delta(S)$ denotes the probability simplex over $S$), $R: S \times A \rightarrow \mathbb{R}$ is the reward function providing feedback signals, $\gamma \in [0, 1]$ is the discount factor, and $\rho_0 \in \Delta(S)$ specifies the initial state distribution.

What this defines: a complete mathematical model of an agent interacting with an environment. At each step, the agent observes state $s$, selects action $a$ according to its policy $\pi_\theta: S \rightarrow \Delta(A)$, the environment transitions to a new state $s' \sim T(s, a)$, and the agent receives reward $R(s, a)$. The goal in a standard RL setting is to find policy parameters $\theta$ that maximize expected cumulative discounted reward.

Why this form: the MDP formalism is the standard mathematical language for sequential decision-making, which allows precise statements about what is available (states, actions, transitions) and what is missing (the reward function $R$ in many real-world language agent environments). The paper explicitly notes that "in many real-world settings this function may be unknown or unverifiable during training," which is the entire motivation for early experience.

The agent maintains a policy $\pi_\theta: S \rightarrow \Delta(A)$, parameterized by $\theta$, which maps states to action distributions. For language agents, both states and actions are represented in natural language, and the policy is implemented as an autoregressive language model that generates action tokens conditioned on state tokens.

Given a dataset of expert demonstrations $D_{\text{expert}} = \{(s_i, a_i)\}_{i=1}^N$ where $a_i$ denotes the expert action at state $s_i$, the standard imitation learning (behavior cloning) objective is:

$\mathcal{L}_{\text{IL}}(\theta) = -\sum_{i=1}^N \log \pi_\theta(a_i \mid s_i)$

where $\pi_\theta(a_i \mid s_i)$ is the probability the policy assigns to the expert action $a_i$ given state $s_i$, and the sum ranges over all $N$ state-action pairs in the expert dataset.

What it computes: the negative log-likelihood of the expert actions under the current policy, summed over all training examples. For a language model, this is implemented as standard next-token prediction: the model receives the state text as input and is trained to generate the action text as output. Minimizing this loss maximizes the probability of the expert actions.

Why this form: maximum likelihood estimation under the assumption that the expert policy is optimal. The loss is convex in the predicted probabilities, making optimization straightforward. However, the paper identifies two fundamental limitations: (1) distribution shift—the learned policy $\pi_\theta$ inevitably deviates from the expert policy during deployment, leading to states not covered in training, and (2) lack of action-consequence awareness—the agent never observes what happens when it takes non-expert actions, so it cannot learn why certain actions fail or how to recover from errors. These limitations are not fixable within the imitation learning framework because they stem from what the loss does not model—the transition dynamics $T$ and the consequences of alternative actions.

---

The Early Experience Data Collection Mechanism

The core primitive of early experience is the rollout dataset $D_{\text{rollout}}$, constructed by having the agent propose and execute alternative actions at each expert state. This section defines the formal notation and construction process.

For each expert state $s_i$ in the dataset $D_{\text{expert}} = \{(s_i, a_i)\}_{i=1}^N$, the agent defines a candidate action set $A_i = \{a_i^1, a_i^2, \ldots, a_i^K\}$ by sampling $K$ alternative actions from its own policy $\pi_\theta(\cdot \mid s_i)$. The expert action $a_i$ is also included in the analysis but is used differently across the two methods.

For the expert action $a_i$, executing it in the environment leads to the next state $s_{i+1}$ (which is part of the original trajectory and already known). For each alternative action $a_i^j \in A_i$, executing it in the environment leads to a next state $s_i^j$ sampled from the transition function $T(s_i, a_i^j)$. These next states capture the immediate consequences of taking action $a_i^j$ at state $s_i$, reflecting changes in the environment such as updated DOM structures, new tool outputs, error messages, or task progression.

The rollout dataset is formally defined as:

$D_{\text{rollout}} = \{(s_i, a_i^j, s_i^j) \mid i \in [N], j \in [K]\}$

where $[N] = \{1, 2, \ldots, N\}$ indexes the expert states and $[K] = \{1, 2, \ldots, K\}$ indexes the alternative actions per state. Each triple represents a state, an alternative action taken at that state, and the resulting next state. All actions $a_i^j$ differ from the expert action $a_i$, allowing the agent to experience diverse state transitions from its own proposed actions.

What this defines: a dataset of $N \times K$ interaction records, each showing what happens when the agent takes a non-expert action at a state where the expert had a specific behavior. The dataset is generated automatically through environment interaction—no human labeling, no reward engineering.

Why this form: the key insight is that the next states $s_i^j$ encode implicit feedback about action quality through environment responses. When the agent clicks the wrong button on a website, the resulting page state (an error message, a wrong product category, an empty form) is information about why that action was suboptimal. The paper states: "The next states $\{s_i^j \mid j \in [K]\}$ encode implicit feedback about action quality through environment responses, enabling the agent to learn from the consequences of both expert and non-expert behaviors" (Section 4.1). The triples format $(s_i, a_i^j, s_i^j)$ preserves the causal relationship—the state before the action, the action itself, and the state after—which is what enables both predictive learning (IWM) and contrastive reasoning (SR).

The number of alternative actions per state, $K$ (the branching factor), is a key hyperparameter studied in Section 6.3. The paper sweeps values of $K$ from 1 to 8 and finds that IWM improves steadily with larger $K$ ("consistent with learning richer transition regularities"), while SR improves at small-to-moderate $K$ and can be non-monotonic at very large $K$ because "comparing many alternatives occasionally includes other success-leading actions, reducing contrast with the expert, and current models have limited capacity to reason over many alternatives and outcomes in a single context."

The mechanism for sampling the $K$ alternative actions varies across environments, as detailed in Appendix B:

• ALFWorld: For implicit world modeling, the paper samples 8 non-expert actions uniformly without replacement from the admissible action list (excluding the expert action). For self-reflection, the policy model with temperature 1.0 proposes up to 3 alternative actions; if a proposed action is not in the admissible action space, it is discarded and replaced by uniform random sampling from remaining unselected admissible actions.

• WebShop: For IWM, the policy proposes actions at temperatures {0.5, 0.8, 0.9}, plus up to five admissible actions sampled uniformly at random per state. For SR, the same mechanism is used with 3 distinct alternatives retained after canonicalization and deduplication. Importantly, the paper notes that some expert trajectories contain suboptimal actions, so a quality filter retains only actions from trajectories whose tasks can be completed within fewer than 15 steps, yielding 6,235 reflection examples from the original 15,464 state-action pairs.

• BFCLv3: The target model samples ten alternative actions per state (in addition to the expert action), yielding 11,904 augmented samples.

• SearchQA: The model generates 30 alternative actions per state at temperature 1.0, substantially more than other environments. Invalid actions (queries not enclosed within <search></search> tags) receive the feedback "Format error! You must enclose the search query within the <search></search> tags if external knowledge is required."

• TravelPlanner: Rather than sampling, the paper performs "exhaustive augmentation by executing ALL available valid actions at each state in the expert trajectories," generating over 70,000 state-transition samples to maximize coverage.

• WebArena-Lite: The target model proposes 5 non-expert actions per state using free-form generation, excluding any identical to the expert action. For each resulting next state, an additional processing step uses the same model to generate a concise summary of the next-state observation conditioned on the task, replacing raw observations to reduce noise and emphasize task-relevant information.

This environment-specific sampling reflects a practical reality: the action space structure (closed vs. open, small vs. large) determines what constitutes a useful alternative. In closed-action-set environments like ALFWorld, uniform sampling from admissible actions is clean and efficient. In open-ended environments like WebArena, free-form generation is required because the action space is combinatorial and cannot be enumerated.

---

Implicit World Modeling: Training Objective and Two-Stage Pipeline

Implicit world modeling formulates environment dynamics as an auxiliary prediction task integrated directly into the policy. Unlike prior work that trains separate world models as explicit simulators for planning, IWM uses the same model parameters $\theta$ for both state prediction (during world modeling) and action prediction (during policy execution).

For each rollout triple $(s_i, a_i^j, s_i^j) \in D_{\text{rollout}}$, the paper constructs a prediction task where the model takes the state-action pair $(s_i, a_i^j)$ as input and learns to predict the resulting next state $s_i^j$. The training objective is a next-token prediction loss:

$\mathcal{L}_{\text{IWM}} = -\sum_{(s_i, a_i^j, s_i^j) \in D_{\text{rollout}}} \log p_\theta(s_i^j \mid s_i, a_i^j)$

where $p_\theta(s_i^j \mid s_i, a_i^j)$ is the language model's output probability for the token sequence representing the next state $s_i^j$, conditioned on the state $s_i$ and action $a_i^j$, and the sum ranges over all triples in the rollout dataset.

What it computes: the standard autoregressive language modeling loss applied to the task of next-state prediction. For each example, the model receives the text representation of the current state and the proposed action, and is trained to generate the text representation of the resulting next state token-by-token. The loss penalizes deviations between the model's predicted state description and the actual observed state description. This is identical in form to the loss used for pretraining and instruction tuning—it is simply next-token prediction with a specific input-output format.

Why this form: this design choice has several important properties. First, it uses the same architecture and training infrastructure as standard language model fine-tuning (no separate world model module, no different optimizer settings). Second, because states are represented entirely in natural language (webpage accessibility trees, tool outputs, environment descriptions), next-state prediction is naturally expressed as a text generation problem—the model predicts the text that would appear in the environment after the action. Third, by training the policy model itself on this objective, the knowledge of environment dynamics is internalized into the same parameters used for action selection, enabling implicit transfer without explicit planning at inference time. The paper contrasts this with inference-time world models: "Unlike inference-time world models used for planning, our implicit formulation integrates predictive signals directly into policy learning, serving as a lightweight warm-up before supervised learning or downstream optimization."

The paper adopts a two-stage training pipeline for IWM: first train with $\mathcal{L}_{\text{IWM}}$ to internalize coarse dynamics, then fine-tune on $D_{\text{expert}}$ (using $\mathcal{L}_{\text{IL}}$) for the remaining training budget. The total number of optimization steps is held equal to the imitation learning baseline across all environments to ensure fair comparison: "we begin with one epoch of the WM objective and then continue supervised updates so that the total updates equal the imitation budget without extra steps" (Section 5.1).

This two-stage design is justified by the relative scale of the datasets. The paper notes that "the rollout data are often an order of magnitude larger than $D_{\text{expert}}$" (Section 4.2). For example, on ALFWorld, the 21,031 expert state-action pairs are augmented with 8 alternatives each, yielding 189,279 IWM training triples. On WebShop, the 15,464 expert pairs yield 122,954 IWM triples. On TravelPlanner, exhaustive enumeration produces over 70,000 samples from only 1,395 expert state-action pairs. The first stage therefore provides broad exposure to diverse state transitions before the second stage focuses on expert behavior.

A critical implementation detail that varies across environments is how the "next state" is represented for the prediction task. The raw state observations in many environments contain substantial noise or extraneous information. The paper handles this differently across benchmarks:

• WebShop: Rather than predicting the raw HTML/DOM, the paper uses "an offline textual summary of the next state after executing that action" (average length 345 characters). This summary captures the semantically meaningful changes (e.g., "After clicking on a color option, this page is a product-details page") rather than low-level markup.

• WebArena-Lite: An additional processing step uses the same model to generate "a concise summary of the next-state observation conditioned on the task, replacing the raw observation to reduce noise and emphasize task-relevant information." This is essentially using the model itself to extract the salient information from raw accessibility tree states.

• SearchQA: Rather than predicting the full text of retrieved documents, the model predicts a summary: "we first instruct the model to summarize the retrieved documents, and then let the model predict these summaries rather than the full text." The paper justifies this by noting that "many tokens are not directly relevant to the search query."

This state representation choice reflects a practical tension: raw state text contains all information but is long and noisy, making prediction difficult and expensive; summarized state text is cleaner but may lose details. The paper's approach—using the model itself or simple heuristics to produce summaries—is a pragmatic compromise that makes the prediction task tractable.

The training hyperparameters are environment-specific and detailed in Appendix B. Representative settings include: on ALFWorld, batch size 16 and learning rate 1e-5 with LlamaFactory for 2 epochs; on WebShop, batch size 4 and learning rate 1e-5; on ScienceWorld, batch size 32 and learning rate 5e-6 for 1 epoch; on TravelPlanner, 5 epochs with learning rate 1e-5 and cosine scheduler on 8 H100 GPUs using DeepSpeed ZeRO-3. The paper uses at most 8 H100 GPUs for training and evaluation across all experiments.

---

Self-Reflection: Contrastive Rationale Generation and Joint Training

Self-reflection formulates learning from exploratory outcomes as a mechanism for generating and training on natural language explanations that compare expert actions against alternatives. The key difference from IWM is that instead of predicting the next state directly, the agent generates a chain-of-thought explanation of why the expert action is better, using the observed next states as grounding evidence.

The data construction process has three steps for each expert state $s_i$:

Step 1: Collect next states for comparison. The expert action $a_i$ is executed to obtain the expert next state $s_{i+1}$. For each alternative action $a_i^j$ (where $j \in \{1, \ldots, K\}$), the corresponding next state $s_i^j$ is obtained by executing $a_i^j$ in the environment. This provides concrete, observable consequences for each action choice.

Step 2: Prompt a language model to generate contrastive reasoning. The paper uses a carefully designed prompt template (shown in full in Section 4.3) that provides the model with the situation description $s_i$, the expert action $a_i$, the expected outcome (expert next state) $s_{i+1}$, and the list of alternative actions with their resulting states. The prompt instructs the model to produce a self-reflection monologue that: (1) analyzes the situation and the goal, (2) compares the possible actions, explaining why each may be less optimal, (3) justifies why the expert action is most suitable grounded in the expected outcome, and (4) highlights any relevant clues, constraints, or consequences.

The output is a chain-of-thought $c_i^j$—a natural language explanation that connects the observed state transitions to decision principles. For example, in the WebShop training data, when the expert action is "click[non-ears blue]" and an alternative is "click[< prev]" (which would return to search results), the generated reflection explains: "click[non-ears blue] wins because it directly addresses the color requirement and allows for further evaluation of the product details. Other actions fail because they either do not address the color requirement or may lead to irrelevant results."

Step 3: Construct training triples. The resulting triplets are collected into a dataset $D_{\text{refl}} = \{(s_i, a_i^j, c_i^j)\}$.

The training objective jointly predicts the chain-of-thought and the expert action conditioned on the state:

$\mathcal{L}_{\text{SR}} = -\sum_{(s_i, a_i^j, c_i^j) \in D_{\text{refl}}} \log p_\theta(c_i^j, a_i \mid s_i)$

where $p_\theta(c_i^j, a_i \mid s_i)$ is the language model's output probability for the concatenated sequence of chain-of-thought $c_i^j$ followed by the expert action $a_i$, conditioned on the state $s_i$, and the sum ranges over all reflection triples.

What it computes: the standard autoregressive language modeling loss applied to a target sequence that first reasons about why an alternative action is suboptimal, then outputs the correct expert action. The model learns to generate the reasoning and the action together as a single coherent output. During inference, the model can produce this reasoning before its action, but the paper does not mandate this—the choice of whether to include chain-of-thought at test time is not explicitly discussed.

Why this form: the concatenated target $c_i^j \circ a_i$ structures the learning as reasoning-then-acting. The chain-of-thought $c_i^j$ is conditioned on the state $s_i$ and serves as an internal rationale that justifies the expert action by contrasting it with a specific alternative $a_i^j$ and grounding the comparison in observed state differences ($s_{i+1}$ vs. $s_i^j$). This is fundamentally different from approaches like STaR (Zelikman et al., 2022) because the rationales are grounded in actual environment outcomes—the model generating the reflection has access to the concrete next states resulting from both the expert and alternative actions. This grounding is what prevents the hallucination problem that the paper identifies in STaR-style approaches: "The retained rationales are ungrounded, having never been tested in the environment, and frequently hallucinate tools or facts, so fine-tuning on them can even degrade performance" (Section 6.1).

In practice, the paper mixes the self-reflection data $D_{\text{refl}}$ with the expert dataset $D_{\text{expert}}$ and trains the model using a standard next-token prediction loss. Chain-of-thought reasoning is generated only for the self-reflection training data; for the expert trajectories, "we retain the original chain-of-thought reasoning in $D_{\text{expert}}$ whenever provided by the expert trajectories, for all models trained with $D_{\text{expert}}$" (Section 4.3). This ensures consistent formatting: the model always sees reasoning followed by action, but the source and nature of the reasoning differ (expert-provided vs. contrastively generated).

The self-reflection prompt template (Section 4.3) is environment-agnostic, designed to elicit explanations that are grounded in the specific state transitions observed. The template includes five explicit guidelines: "Stay strictly within the provided information," "Avoid meta-commentary about being an AI," "Use natural, step-by-step reasoning," "Focus on logical decision-making," and the output format should be "Directly write the self-reflection monologue, no extra headings, disclaimers, or external notes." These guidelines are important because without them, language models tend to produce generic reasoning patterns or hallucinate facts about the environment—the constraints force the model to anchor its reasoning in the actual observed state differences.

The paper does not specify which model generates the self-reflection data. From context and Appendix B, it appears the same base model being trained (e.g., Llama-3.1-8B-Instruct) is used for reflection generation, but this is an implementation detail that varies—on some benchmarks, a stronger model may have been used. The paper also applies quality filtering: "We filter out low-quality generations where the explanation incorrectly supports a non-expert action" (Appendix B, WebArena-Lite section), reducing 42,264 potential reflections to 3,190 high-quality examples. Similar filtering is applied on BFCLv3 ("filtering a small number of low-quality samples where the concluded action did not match the expert action") and Tau-Bench ("filter out a small number of low-quality reflection samples").

The training setup for self-reflection matches the imitation learning budget: "For Self-Reflection, we train for the same number of epochs as imitation" (Section 5.1). Representative hyperparameters from Appendix B: on ALFWorld, batch size 16 and learning rate 1e-5 for 2 epochs; on WebShop, batch size 4 and learning rate 1e-5; on TravelPlanner, the maximum generation length is extended to 8K tokens "to accommodate detailed reasoning"; on Tau-Bench, 6 epochs with learning rate 1e-5.

---

Design Choices and Their Justifications Across Both Methods

Why two distinct methods rather than one unified approach? The paper implicitly argues that IWM and SR capture complementary types of knowledge that are both valuable but operate differently. IWM is purely predictive—learning the mapping $(s, a) \rightarrow s'$ captures the mechanical regularities of the environment: which actions produce which state changes, what error messages appear when, how the state machine advances. SR is contrastive and normative—it captures decision principles: why one action is preferable to another given task constraints, budget limits, and goal conditions. These are complementary because an agent could know what happens after each action (IWM) without knowing which action to choose, or know which action to choose without understanding the full state-space dynamics (SR).

Why a two-stage pipeline for IWM (world modeling first, then expert fine-tuning)? The two-stage design separates the learning of environment dynamics from the learning of expert behavior. The first stage exposes the model to a broad distribution of state transitions—including transitions from suboptimal actions—which builds a coarse internal model of how the environment responds to different actions. The second stage then teaches the model to select actions that lead to desirable states. If the stages were reversed or interleaved, the model might overfit to the expert distribution early and fail to benefit from the broader transition data. The paper's decision to fix the total number of optimization steps to match the imitation learning baseline (Section 5.1) ensures that any performance difference comes from what the model learns, not from how long it trains.

Why mix self-reflection data with expert data rather than training on self-reflection alone? The paper states: "This joint training setup balances grounded decision-making from demonstrations with contrastive insights from exploratory outcomes" (Section 4.3). Training only on self-reflection data would teach the model to reason about alternatives but might not provide enough coverage of the correct action distribution. Training only on expert data (imitation learning) teaches correct actions but not why they are correct. The mixture ensures the model learns both what to do and why to do it.

Why $K$ alternative actions (rather than one or all possible)? The branching factor $K$ controls a tradeoff between coverage and quality. Too few alternatives limit the diversity of transitions and contrasts the model sees. Too many alternatives (especially for SR) "occasionally includes other success-leading actions, reducing contrast with the expert" and overwhelms the model's capacity to reason over many comparisons. The paper's ablation in Section 6.3 shows that IWM improves monotonically with larger $K$ (consistent with learning from more diverse transitions) while SR has a sweet spot around $K = 2$ to $4$, beyond which performance can degrade.

Why use the same policy parameters for both world modeling and action selection? This is a deliberate architectural choice to avoid the complexity of separate world model modules. The paper argues that by training the policy model directly on next-state prediction, "the model internalizes coarse environment dynamics without a standalone simulator" (Section 4.2). This has practical advantages: no additional parameters, no separate training pipeline, no planning overhead at inference time. The tradeoff is that the policy's capacity must be shared between two tasks (predicting states and predicting actions), but the paper's results suggest this is not a limiting factor in practice.

4. Key Insights and Innovations

Innovation 1: Formalizing "Early Experience" as a Distinct Training Paradigm Between Imitation and RL

The paper's most fundamental contribution is not a method but a diagnostic reframing of the language agent training landscape. Prior to this work, the field implicitly operated with a binary taxonomy: either you trained agents via imitation learning on expert demonstrations (the "era of human data"), or you trained them via reinforcement learning with environment rewards (the "era of experience"). The paper identifies a vacant third category that occupies the gap between these two poles—what it calls the early experience paradigm—and argues that this category has been systematically overlooked despite being the natural operating regime for most current language agent deployments.

What makes this reframing intellectually distinctive is that it identifies a shared assumption across both existing paradigms: that learning from interaction requires scalar rewards. Imitation learning sidesteps interaction entirely (the agent never acts during training). RL demands interaction but depends on rewards that most real-world environments don't provide. The paper's diagnostic move is to observe that the environment's state transitions themselves constitute a supervision signal that has been ignored—when an agent clicks the wrong button and observes an error page, that observation is feedback about action quality, even if no numeric reward is attached to it. This is not a new technical observation (model-based RL has used state transitions for decades), but formalizing it as a standalone training paradigm for language agents—complete with notation, data structures, and a taxonomy of strategies for extracting supervision from transitions—is a conceptual contribution that changes how researchers should think about the training pipeline.

Prior work had touched on pieces of this idea without recognizing it as a unified paradigm. World models for language agents (Gu et al., 2025; Guo et al., 2025) treated state prediction as a separate component for planning, not as a training signal integrated into the policy. Self-reflection at inference time (Shinn et al., 2023; Madaan et al., 2023) used interaction outcomes as prompts but didn't update model parameters. STaR (Zelikman et al., 2022) generated rationales from correct answers but did so without interacting with the environment or exploring alternatives. The paper's key move is to see these as pieces of a larger, unnamed paradigm and to provide the formal vocabulary ($D_{\text{rollout}}$, $D_{\text{refl}}$, the branching factor $K$, the two-stage vs. joint training distinction) that makes the paradigm operational and testable across diverse environments.

The evidence that this reframing is not merely cosmetic comes from the consistency of results across eight environments with fundamentally different action spaces, observation formats, and task structures (Table 2). If the paradigm were vacuous—if the gains came from environment-specific tricks rather than a general principle—the methods would work on some benchmarks and fail on others. Instead, both IWM and SR improve over imitation learning in every environment tested, across three model families and three model sizes. This breadth of evidence supports the claim that the paradigm captures something real: state transitions encode implicit supervision that is extractable across diverse settings.

The significance of this reframing extends beyond the paper's immediate results. By naming the paradigm and providing a formal language for it, the paper opens a research direction. It invites the community to ask: what other forms of supervision can be extracted from $D_{\text{rollout}}$? The paper explicitly notes this in Section 7: "Another direction is to investigate other instances of early experience beyond the two approaches proposed in this paper." The paradigm is the container; IWM and SR are two initial instantiations.

---

Innovation 2: Demonstrating That Interaction-Based Supervision Functions as a Substitute for Expert Data Scale

While the paper's headline gains over imitation learning are substantial (e.g., +18.4 percentage points on WebShop with Llama-3.2-3B for IWM), the deeper insight is about data efficiency as a function of supervision type. The paper shows not just that early experience helps, but that it provides a qualitatively different form of supervision from expert demonstrations—one that teaches the agent about the consequences of suboptimal actions, not just the correct actions themselves.

This is demonstrated most sharply in Figure 4(a), where the paper varies the fraction of expert demonstrations used to seed early experience. With only 1/8 of the expert trajectories on WebShop, IWM trained with Llama-3.1-8B achieves 38.3% success rate, which already surpasses full-dataset imitation learning (45.3% is close, and the trend line suggests further scaling). On ALFWorld, IWM with 1/2 of the demonstrations (56.2%) matches full-dataset imitation learning (80.5% is not reached, but the relative gain from early experience is large). What this means is that a small amount of expert data plus interaction-based supervision can substitute for a much larger amount of expert data alone.

The conceptual significance of this finding is that it reframes the bottleneck in language agent training. The dominant narrative in the field has been that expert demonstrations are the scarce resource—we need more of them, better quality, broader coverage (Deng et al., 2023; Pahuja et al., 2025). The paper's results suggest that this framing is incomplete. The bottleneck is not the quantity of expert data but the type of supervision it provides. Expert data teaches the agent what to do; early experience teaches the agent what not to do and why. Both forms of knowledge are necessary, and they can be acquired through different means with different scaling properties. Expert demonstrations are expensive and scale linearly with human effort; interaction data is cheap and can be generated automatically by the agent itself. The paper's data-efficiency results imply that the optimal allocation of resources may be heavily skewed toward generating interaction data rather than collecting more demonstrations.

Table 4 reinforces this point from the opposite direction. The STaR baseline, which generates rationales from correct answers without environment interaction, degrades performance relative to imitation learning on both WebShop (47.3% → 25.0%) and ALFWorld (80.5% → 74.2%). This is a critical negative result: it shows that the knowledge source (environment interaction vs. model hallucination) matters more than the format (rationale + action vs. action alone). STaR and SR both train on rationale-augmented data, but STaR's rationales are ungrounded in actual state transitions, while SR's rationales are anchored in observed outcomes. The performance divergence demonstrates that grounding, not formatting, is the active ingredient.

This insight connects to a broader challenge in language model training: the problem of distribution shift under compounding errors (Ross et al., 2011). Imitation learning fails because the agent's policy inevitably deviates from the expert trajectory during deployment, entering states not covered in training. The paper's data-efficiency results suggest that early experience mitigates this by exposing the agent to off-expert-path states during training—the alternative actions in $D_{\text{rollout}}$ produce states that the agent would encounter when it makes mistakes. By learning the transition dynamics of these mistake states (IWM) or reasoning about why they represent mistakes (SR), the agent builds robustness to its own errors in a way that additional expert trajectories cannot provide.

---

Innovation 3: The "Warm-Start Compounding" Effect—Reward-Free Supervision Amplifies Subsequent RL

Perhaps the most surprising result in the paper is not that early experience improves over imitation learning (which could be attributed to simply using more data), but that initializing RL with early-experience checkpoints produces higher final performance than initializing with imitation-only checkpoints, even when both receive the identical RL training recipe. This is demonstrated in Figure 3 across three environments: WebShop, ALFWorld, and SearchQA. On WebShop with Llama-3.2-3B, imitation learning + GRPO achieves 82.0% success, while IWM + GRPO achieves 92.2% and SR + GRPO achieves 89.8%. On ALFWorld, the gap between imitation + GRPO (93.8%) and SR + GRPO (98.5%) with Llama-3.1-8B persists after RL.

This finding challenges an implicit assumption in the RL-for-language-agents literature: that the quality of the RL initialization matters only for training stability and convergence speed, not for the asymptotic performance ceiling. If RL were capable of fully recovering from a suboptimal initialization through sufficient exploration and reward optimization, we would expect the post-RL performance of imitation-only and early-experience starts to converge. They do not. The performance gap persists or even widens (e.g., ALFWorld with Llama-3.2-3B: imitation gap of 7.8 points pre-RL becomes 7.5 points post-RL; SearchQA with Qwen-2.5-7B: a 2.1-point pre-RL gap becomes 2.6 points post-RL for SR).

The conceptual implication is that the knowledge acquired through reward-free interaction supervision is not redundant with what RL extracts from reward signals. The two forms of learning are complementary: early experience teaches the agent about environment dynamics and decision principles that help it explore more effectively or assign credit more accurately during RL. The paper does not provide a mechanistic explanation for this complementarity—it does not analyze whether the benefit comes from better exploration, more accurate value function learning, or some other factor—but the empirical result is robust across environments and model sizes.

This has practical significance for the RL infrastructure problem the paper diagnoses. Section 2.1 notes that "scalable RL for language agents is not yet mature" due to missing simulators, reset mechanisms, and reward functions. The warm-start compounding effect means that even when these infrastructure pieces arrive for a given environment, the optimal training pipeline will likely include an early experience phase before RL, not skip directly from imitation learning to RL. The paper thus provides evidence not just for immediate gains in reward-free settings, but for a permanent architectural role of early experience in the training pipeline, even as RL matures.

The negative result of applying GRPO directly from a raw pretrained model—"This performs worst across all tasks and shows unstable training dynamics"—closes the loop on this argument. It shows that some form of supervised warm-start is necessary for RL in these environments, and that early experience provides a better warm-start than imitation learning alone. The paper is essentially arguing for a three-stage pipeline (pretraining → early experience → RL) rather than the two-stage pipeline (pretraining → RL) that the "era of experience" vision might suggest.

---

Innovation 4: The Spectrum from Predictive to Contrastive Supervision and Its Difficulty-Dependent Efficacy

The paper's two methods—implicit world modeling and self-reflection—are not presented as competing alternatives but as points on a spectrum of how to extract supervision from interaction data. IWM extracts predictive knowledge: it teaches the model the mapping $(s, a) \rightarrow s'$ that captures the mechanical regularities of the environment. SR extracts contrastive knowledge: it teaches the model to compare action outcomes and extract decision principles. The paper's results reveal that the relative effectiveness of these two forms of supervision depends on the structure of the environment and the nature of the tasks within it.

This is not presented as a theoretical claim, but it emerges clearly from the pattern of results in Table 2. IWM delivers its largest gains in environments where the state-transition dynamics are systematic and predictable: WebShop (+18.4 with Llama-3.2-3B, +11.3 with Llama-3.1-8B), where clicking a color option predictably filters the product list; ALFWorld (+5.5, +5.4), where picking up an object predictably adds it to inventory. SR delivers its largest gains in environments where tasks require multi-step constraint satisfaction and planning: TravelPlanner (+12.8 to +15.0 across all three models), where selecting a flight, hotel, and restaurant must jointly satisfy budget, date, cuisine, and room-rule constraints; ScienceWorld (+13.3 with Llama-3.1-8B), where scientific experiments require correct sequencing of tool operations.

This pattern makes intuitive sense. When the challenge is primarily about knowing what happens next—what page will load, what state will result—predictive learning directly addresses the knowledge gap. When the challenge is primarily about knowing why one choice is better than another given complex, interacting constraints—why this flight plus this hotel fits the budget while that combination doesn't—contrastive reasoning directly addresses the knowledge gap. The paper does not make this causal claim explicitly, but the environment-level variation in which method dominates provides compelling circumstantial evidence.

The conceptual significance is that it provides a taxonomy for matching supervision type to task structure. Future work building on the early experience paradigm would benefit from analyzing the target environment along this predictive-versus-contrastive dimension to determine which method (or what mixture) to deploy. Environments with stable, deterministic transitions and success criteria that are primarily about correct sequencing (ALFWorld, WebShop) favor IWM. Environments with complex constraints, combinatorially many valid paths, and success criteria that are primarily about constraint satisfaction (TravelPlanner, ScienceWorld) favor SR. The paper's decision to study both methods rather than commit to one reflects an implicit recognition of this spectrum.

The branching factor analysis in Figure 4(b) provides additional evidence for this interpretation. IWM improves monotonically with larger $K$—seeing more state transitions is always better for learning dynamics. SR shows diminishing or negative returns at large $K$—reasoning over too many alternatives reduces the contrast with the expert and exceeds the model's capacity for comparison. This asymmetry is exactly what one would expect from the two methods' different supervision types: predictive learning benefits from more data with no coherence requirement, while contrastive learning requires carefully curated comparisons where the expert action is clearly distinguishable from the alternatives.

---

Innovation 5: The Implicit World Model as a Unifying Architectural Principle

The paper's decision to train the world model within the same parameters as the policy—rather than as a separate module—is a design choice with deeper conceptual implications than might first appear. Prior work on world models for language agents (Gu et al., 2025; Chae et al., 2025; Hao et al., 2023) treats the world model as an external component: the policy queries the world model to simulate outcomes before acting, analogous to model-based RL where a dynamics model is used for planning. The paper explicitly distances itself from this approach: "In contrast, we view the interaction trace itself as an auxiliary prediction task for the agent policy... By training the policy to predict its own future states, the model internalizes coarse environment dynamics without a standalone simulator."

This "implicit" design choice carries several conceptual implications. First, it means there is no architectural distinction between "knowing what will happen" and "knowing what to do"—both forms of knowledge reside in the same parameters and are accessed through the same forward pass. This is a form of representation sharing that forces the model to learn state representations that are useful for both prediction and decision-making, potentially leading to more robust internal representations.

Second, it means the world model is always available at inference time with zero additional computational cost, because no separate model needs to be queried. Traditional model-based approaches require running the world model forward to simulate outcomes before choosing an action, adding latency and computational overhead. The implicit approach amortizes this cost into the training phase: the model learns to use its internalized dynamics knowledge directly in action selection, without explicit simulation.

Third, it suggests a particular view of what a language model is in the context of agent training. The paper treats the language model not just as a policy (a mapping from states to actions) but as a unified representation of environment knowledge—both procedural (what to do) and declarative (what happens if). This aligns with the broader observation in the language model literature that models trained on diverse text corpora acquire implicit world knowledge during pretraining (the "world model in the weights" hypothesis); the paper extends this idea by showing that such implicit knowledge can be deliberately trained into the model through environment interaction, not just absorbed from static text.

The evidence for this interpretation is indirect but suggestive. The OOD generalization results in Table 3 show that IWM consistently recovers a substantial portion of the performance gap between in-domain and out-of-domain settings—sometimes with larger relative gains than in-domain (e.g., SearchQA Llama-3.2-3B: +1.0 in-domain vs. +4.9 OOD). If the model were simply memorizing specific state transitions, we would expect OOD performance to degrade more than it does. The fact that the predictive knowledge transfers suggests that the model is learning something more abstract—regularities about how the environment behaves that generalize across distribution shifts.

This insight connects to a broader question in the field: what is the right way to integrate environment knowledge into language agent policies? The paper's implicit world modeling provides one answer—train it directly into the policy parameters as an auxiliary objective—but this is just one point on a larger design spectrum that includes external world models, retrieval-augmented policies, and tool-based simulation. The paper's success with the implicit approach provides a strong baseline that future work must contend with: if training the policy to predict next states works this well, the bar for justifying a more complex external simulator architecture is raised.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on eight diverse language-agent benchmarks spanning multiple domains: ALFWorld (embodied instruction-following, 3,553 trajectories producing 21,031 state-action pairs), WebShop (e-commerce navigation, 1,571 trajectories / 15,464 state-action pairs), BFCLv3 (multi-turn tool use, 125 trajectories / 1,264 state-action pairs), Tau-Bench (customer-service tool use, 452 trajectories / 5,239 state-action pairs), SearchQA (multi-hop question answering, 2,082 trajectories / 7,691 state-action pairs), ScienceWorld (scientific simulation, 1,000 trajectories / 14,506 state-action pairs), TravelPlanner (long-horizon travel planning, 45 trajectories / 1,395 state-action pairs), and WebArena-Lite (web navigation, 554 trajectories / 7,044 state-action pairs). Each benchmark uses its own native training/evaluation split, with out-of-domain evaluation following the OOD splits defined in the original works. All expert trajectories are collected from either human annotations or high-performing model-generated rollouts.

• Base model(s). Experiments use three instruction-tuned models from two families: Llama-3.2-3B-Instruct, Qwen-2.5-7B-Instruct, and Llama-3.1-8B-Instruct. The 70B scaling experiment additionally uses Llama-3.3-70B-Instruct and Qwen-2.5-72B-Instruct with LoRA fine-tuning. The models are chosen to represent "contemporary LLMs" in a regime where they possess non-trivial base performance but substantial headroom for improvement—zero-shot prompting yields near-zero performance on TravelPlanner and WebShop (Table 2), confirming that the tasks are challenging enough that training is necessary but not so difficult that learning is impossible.

• Metrics. The paper reports each benchmark's native primary metric: success rate (%) for ALFWorld, WebShop, BFCLv3, Tau-Bench, ScienceWorld, TravelPlanner, and WebArena-Lite; F1 score (%) for SearchQA. TravelPlanner additionally reports commonsense constraint satisfaction (Micro and Macro) and hard constraint satisfaction (Micro and Macro) as intermediate metrics, with Final Pass Rate as the primary. ALFWorld reports per-task-type breakdowns (Pick, Look, Clean, Heat, Cool, Pick2). WebShop reports both score and success rate. BFCLv3 reports breakdowns across Base, Long Context, Miss Function, and Miss Parameters sub-splits. All evaluations use official validators released by each benchmark.

• Baselines. (1) Prompting: Zero-shot or few-shot evaluation of the raw instruction-tuned model without any fine-tuning. (2) Imitation Learning (Behavior Cloning): Standard SFT on expert state-action pairs using the loss $\mathcal{L}_{\text{IL}}(\theta) = -\sum_{i=1}^N \log \pi_\theta(a_i \mid s_i)$. This is the primary baseline against which early experience methods are compared. (3) Long CoT (test-time scaling): Inspired by Snell et al. (2024), forces models to reason longer before acting by performing heavier prompt search and truncating delimiter tokens to encourage continued generation (Section 6.1). Applied both to raw instruction-tuned models and imitation-trained models. (4) STaR-style data (Zelikman et al., 2022): The model generates a rationale for the expert action at each state, retaining only cases where the predicted action matches the expert, then fine-tunes on (state, rationale, action) tuples. (5) RL from raw pretrained model: GRPO applied directly to the instruction-tuned model without any supervised warm-start (Section 5.4). (6) GRPO: Reinforcement learning from imitation-trained and early-experience-trained checkpoints, using identical hyperparameters from Feng et al. (2025) and Jin et al. (2025).

• Generation budget / compute accounting. Training is normalized by number of optimization steps rather than FLOPs. For each environment, the paper first determines the optimal number of training steps for the imitation learning baseline (selecting the checkpoint with lowest training loss and best validation performance), then fixes this step budget for all methods. For IWM, the total budget is split into one epoch of world modeling followed by remaining steps of imitation learning, so the total updates match imitation without extra steps. For SR, training runs for the same number of epochs as imitation. The alternative action budget is measured by the branching factor $K$ (number of alternative actions per expert state), swept from 1 to 8. All training uses at most 8 H100 GPUs; evaluation uses either vLLM for efficiency or the native benchmark infrastructure.

• Cross-validation / statistical protocol. The paper does not employ formal cross-validation for hyperparameter selection, but does follow environment-specific protocols: for SearchQA, 500 examples are randomly sampled from the test set to accelerate evaluation (following Goldie et al., 2025; Wu et al., 2024). For TravelPlanner, the validation set of 180 queries is distinct from the 45 training queries. For WebShop and ALFWorld, the paper follows the default splits and evaluation protocols from Feng et al. (2025). For BFCLv3, since no training split exists by default, 75% of the Base category samples are randomly selected for training. Statistical significance tests or confidence intervals are not reported.

Main Quantitative Results

Effectiveness Across Eight Environments (Table 2)

Headline result: Early experience methods improve over imitation learning in all eight environments and across all three model families tested. The magnitude of improvement varies substantially by environment and method, with IWM gains ranging from +0.9 to +18.4 percentage points and SR gains ranging from +0.6 to +15.0 percentage points. The largest absolute gains appear in WebShop and TravelPlanner; the smallest appear in WebArena-Lite (where all methods struggle) and SearchQA.

Web navigation results (WebShop and WebArena-Lite): On WebShop, IWM delivers the largest gains for Llama-3.2-3B (+18.4: 41.8% → 60.2%) and Llama-3.1-8B (+11.3: 47.3% → 58.6%), while SR delivers larger gains for Qwen-2.5-7B (+10.6: 51.6% → 62.2%). The pattern reverses on WebArena-Lite, where IWM consistently outperforms SR across all models: Llama-3.2-3B (+2.4: 6.1% → 8.5%), Qwen-2.5-7B (+3.1: 4.2% → 7.3%), Llama-3.1-8B (+3.6: 4.9% → 8.5%). Both methods show absolute performance far below WebShop, consistent with WebArena being a substantially harder environment (even GPT-4o achieves only 13.9% in prompting mode, per Table 10).

Embodied and scientific simulation results (ALFWorld, ScienceWorld): On ALFWorld, both methods produce consistent but moderate gains across all models (IWM: +4.7 to +5.5; SR: +3.9 to +7.8). The largest single gain is SR on Llama-3.2-3B (+7.8: 78.1% → 85.9%). On ScienceWorld, SR on Llama-3.1-8B produces the largest gain observed for any model-environment pair in this category (+13.3: 54.7% → 68.0%), substantially outperforming IWM on the same model (+2.3). This asymmetry—SR dominating IWM on ScienceWorld but not ALFWorld—is consistent with ScienceWorld's task structure emphasizing multi-step experimental procedures with tool sequencing constraints, where contrastive reasoning about action choices is particularly valuable.

Long-horizon planning results (TravelPlanner): This environment shows the most dramatic and consistent advantage for SR over IWM. On all three models, SR delivers the largest gains in the entire table: Llama-3.2-3B (+12.8: 19.4% → 32.2%), Qwen-2.5-7B (+15.0: 16.7% → 31.7%), Llama-3.1-8B (+15.0: 17.2% → 32.2%). IWM also produces substantial gains (+5.5 to +8.9) but lags behind SR in every comparison. The commonsense and hard constraint metrics in Table 9 reveal that these gains come primarily from improved constraint satisfaction: SR on Llama-3.1-8B improves Hard Constraint Macro from 46.7% (imitation) to 51.1%, and Commonsense Constraint Macro from 25.0% to 42.2%. TravelPlanner has by far the fewest expert trajectories (45) and state-action pairs (1,395) of any environment, making the relative gains from early experience—which generates its own additional supervision—particularly large in percentage-point terms.

Multi-turn tool-use results (BFCLv3, Tau-Bench): On BFCLv3, SR consistently outperforms IWM, with the largest gain on Llama-3.2-3B (+8.0: 21.3% → 29.3%). The detailed sub-split results in Table 7 show interesting patterns: the Base sub-split improves from 21.3% to 29.3% with SR on Llama-3.2-3B, while the challenging Miss Function sub-split (where tools are missing) sees the largest improvement (from 0.0% to 5.3%), suggesting early experience helps the model recognize when available tools are insufficient. On Tau-Bench, both methods produce consistent moderate gains: IWM (+1.8 to +4.9), SR (+4.4 to +5.8). The relatively balanced performance between IWM and SR on Tau-Bench (unlike TravelPlanner where SR dominates) suggests that both predictive and contrastive knowledge contribute roughly equally in structured API-calling environments.

Multi-hop question answering results (SearchQA): The improvements here are the smallest in absolute terms across all environments. IWM gains range from +0.9 to +3.3; SR gains range from +0.6 to +2.1. The F1 metric's finer granularity means that percentage-point gains of 1–3 points represent more modest relative improvements than in other environments, but the consistency across models (11 of 12 model-method pairs show improvement) supports a real effect. The OOD results in Table 8 show substantially larger gains: Llama-3.2-3B IWM improves OOD F1 by +4.9 (40.5% → 45.4%), and Qwen-2.5-7B SR improves OOD by +4.2 (47.0% → 51.2%), suggesting that the benefits of early experience are particularly pronounced under distribution shift in retrieval-based tasks.

Out-of-Domain Generalization (Table 3)

Headline result: Early experience methods consistently improve out-of-domain robustness, with OOD gains often matching or exceeding in-domain gains. On ALFWorld, IWM on Llama-3.1-8B produces a +14.8 OOD gain (63.3% → 78.1%) versus +5.4 in-domain (Table 2). On SearchQA, both methods show larger OOD than in-domain gains across all three models: Llama-3.2-3B IWM (+4.9 OOD vs. +1.0 in-domain), Qwen-2.5-7B SR (+4.2 OOD vs. +2.1 in-domain), Llama-3.1-8B IWM (+2.2 OOD vs. +3.3 in-domain). On BFCLv3, IWM on Qwen-2.5-7B achieves +5.3 OOD (7.6% → 12.9%) versus +2.6 in-domain (Table 2). These patterns suggest that early experience provides knowledge that generalizes beyond the expert demonstration distribution—predictive dynamics and contrastive decision principles learned from alternative actions transfer across domain shifts.

Method-specific OOD patterns: IWM dominates OOD gains on ALFWorld (+14.8 for Llama-3.1-8B vs. +9.4 for SR), consistent with this environment's stable transition dynamics making predictive knowledge especially transferable. SR shows stronger OOD gains on BFCLv3 (+8.5 for Llama-3.2-3B vs. +3.6 for IWM), consistent with shifts affecting tool availability and arguments where contrastive reasoning about action appropriateness matters more. Both methods help on SearchQA OOD, with IWM slightly stronger on Llama-3.2-3B and Llama-3.1-8B, and SR stronger on Qwen-2.5-7B.

Reinforcement Learning Following Early Experience (Figure 3)

Headline result: Initializing GRPO with early-experience checkpoints consistently yields higher post-RL performance than initializing with imitation-only checkpoints, and the performance gap persists after RL training. The paper reports this across three environments with verifiable rewards:

On WebShop (Figure 3a): For Llama-3.2-3B, imitation learning + GRPO achieves 82.0% → 92.2% after RL, while IWM + GRPO reaches 97.7% and SR + GRPO reaches 93.8%. The initial gap between IWM and imitation (60.2% vs. 41.8% = +18.4) narrows but persists after RL (92.2% vs. 82.0% = +10.2 for IWM). For Llama-3.1-8B, imitation + GRPO reaches 93.8%, while IWM + GRPO reaches 97.7% and SR + GRPO reaches 98.5%. On ALFWorld (Figure 3b): The same pattern holds with post-RL ceilings: Llama-3.2-3B imitation + GRPO (92.2%) vs. IWM + GRPO (97.7%) vs. SR + GRPO (99.2%); Llama-3.1-8B imitation + GRPO (93.8%) vs. IWM + GRPO (97.7%) vs. SR + GRPO (98.5%). On SearchQA (Figure 3c): The effect is smaller but consistent. Llama-3.2-3B imitation + GRPO (44.8%) vs. SR + GRPO (46.3%); Qwen-2.5-7B imitation + GRPO (48.6%) vs. IWM + GRPO (51.1%); Llama-3.1-8B imitation + GRPO (47.1%) vs. IWM + GRPO (49.8%).

Direct RL from raw model (no supervised warm-start): The paper notes that "GRPO directly from the raw pretrained model without any supervised stage... performs worst across all tasks and shows unstable training dynamics, highlighting the necessity of a strong initialization." This provides a critical lower bound: RL alone cannot solve these environments without some form of supervised pre-training.

Non-convergence of post-RL performance: The gap between early-experience and imitation-based RL starts does not close—in some cases it widens. On ALFWorld with Llama-3.2-3B, the pre-RL gap of 7.8 points (SR vs. imitation) becomes 7.0 points post-RL. On WebShop with the same model, the 18.4-point IWM pre-RL gap becomes 10.2 post-RL (narrowing but not closing). This demonstrates that RL does not fully recover the knowledge differential encoded in early-experience checkpoints, suggesting the two forms of learning provide complementary rather than redundant benefits.

Ablation Studies and Robustness Checks

Impact of amount of human data (Figure 4a): The paper trains models on WebShop and ALFWorld using Llama-3.1-8B-Instruct with varying fractions of expert trajectories (1/8, 1/4, 1/2, full) while keeping total training budget fixed. On WebShop, IWM with 1/8 of demonstrations achieves 38.3% success, already surpassing imitation learning on 1/4 of demonstrations (33.6%) and approaching the 1/2 imitation baseline (44.6%). SR shows a similar but slightly weaker pattern: 1/8 SR (25.8%) exceeds 1/8 imitation but lags behind IWM at low data fractions. On ALFWorld, IWM with 1/2 demonstrations (67.2%) approaches full-dataset imitation (80.5%), while IWM with 1/4 demonstrations (42.9%) already exceeds 1/2 imitation (46.1%). The key finding is that early experience provides supervision that can substitute for expert data scale: the performance curves for IWM and SR lie consistently above imitation learning at every data fraction, demonstrating that the benefits are not merely additive but represent a form of complementary supervision.

Impact of branching factor (Figure 4b): The number of alternative actions per expert state ($K$) is varied from 1 to 8 on WebShop and ALFWorld. On WebShop, IWM improves steadily from $K = 1$ (45.3%) to $K = 8$ (58.6%), consistent with the interpretation that more transition data improves predictive learning regardless of action quality. SR on WebShop shows a non-monotonic pattern: improving from $K = 1$ (49.6%) to a peak around $K = 2$ to $4$ (55.5–57.0%), then declining to 54.7% at $K = 8$. On ALFWorld, IWM increases from $K = 1$ (78.1%) to a peak at $K = 8$ (85.2%), while SR peaks at moderate $K$ (85.9% at $K = 2$ to $3$, declining to 82.0% at $K = 8$). The paper attributes SR's non-monotonicity to two factors: "comparing many alternatives occasionally includes other success-leading actions, reducing contrast with the expert, and current models have limited capacity to reason over many alternatives and outcomes in a single context." This asymmetry between the methods' sensitivity to $K$ is a key robustness check: IWM's monotonic improvement confirms that the method benefits from broader transition coverage, while SR's peaked curve confirms that contrastive reasoning requires careful curation of comparison difficulty.

Comparison to Long CoT and STaR baselines (Table 4): On WebShop and ALFWorld with Llama-3.1-8B-Instruct, Long CoT applied to the raw instruction-tuned model produces negligible improvement (WebShop: 0.0% → 1.6%; ALFWorld: 25.0% → 28.4%). When applied to the imitation-trained model, Long CoT dramatically regresses performance: WebShop from 47.3% to 0.0%, ALFWorld from 80.5% to 25.8%. The paper explains this as a collapse in reasoning coherence: "once fine-tuned only on expert trajectories lacking inherent rationales, models lose the ability to sustain coherent long-form reasoning, so extended chains often drift or collapse into invalid/off-policy actions." STaR-style data degrades performance on both environments: WebShop from 47.3% to 25.0% (-22.3), ALFWorld from 80.5% to 74.2% (-6.3). The paper attributes this to ungrounded rationales: "The retained rationales are ungrounded, having never been tested in the environment, and frequently hallucinate tools or facts." These negative baselines are critical because they demonstrate that the gains from early experience are not achievable through inference-time scaling or rationale-only training—the environment interaction component is essential.

Model scaling (Figure 5 and Table 10): On WebArena-Lite, the paper compares Llama-3.2-3B, Llama-3.1-8B, and Llama-3.3-70B (with LoRA for the 70B model to constrain compute). Early experience outperforms imitation learning at every scale: for the 70B model, IWM achieves 16.4% vs. 13.3% imitation, and SR achieves 15.2%. Table 10 additionally shows Qwen-2.5-72B-Instruct with the same pattern: IWM (17.6%) and SR (15.8%) exceed imitation (12.7%). The paper states that "early-experience checkpoints consistently occupy the top curve, indicating that the supervision it provides complements model size rather than substituting for it." The sub-split breakdowns in Table 10 reveal environment-specific scaling patterns: on CMS tasks, Llama-3.3-70B SR achieves 23.8% vs. 14.3% imitation, a larger relative gain than at smaller scales; on Map tasks, the 70B IWM achieves 19.4% vs. 17.2% imitation, a more modest gain.

RL from raw model (negative result): As noted in Section 5.4, applying GRPO directly to the instruction-tuned model without any supervised stage produces the worst performance across all tasks with unstable training dynamics. This is not presented as a formal ablation table but is described as a consistent finding that establishes the necessity of supervised warm-start for RL in these environments.

State representation for IWM (Appendix B): The paper experiments with different state summarization strategies across environments. On WebShop, using offline textual summaries (avg. 345 characters) rather than raw DOM makes the prediction task tractable. On SearchQA, predicting summaries of retrieved documents rather than full document text is necessary because "many tokens are not directly relevant to the search query." On WebArena-Lite, using the model itself to generate concise next-state summaries conditioned on the task reduces noise. These design choices are not systematically ablated against raw-state prediction, representing an implicit claim about what level of state abstraction is appropriate for the IWM objective—the paper argues that semantically meaningful summaries enable learning while raw observation text would be too noisy and long.

Action sampling strategy variation (Appendix B): The paper uses different alternative-action sampling strategies across environments without formal ablation: uniform sampling from admissible actions (ALFWorld), model-proposed actions at multiple temperatures (WebShop, SearchQA), exhaustive enumeration of all valid actions (TravelPlanner), and free-form generation (WebArena-Lite). In ALFWorld for SR, actions not in the admissible action space are discarded and replaced with random uniform samples from remaining admissible actions. The paper does not compare these strategies head-to-head within a single environment, leaving open the question of whether the sampling strategy significantly affects performance or whether the gains are robust to this choice.

Critical Assessment

Claim 1: "Both methods consistently improve over imitation learning across eight environments and three model families." This claim is solidly supported by Table 2. The improvements are directionally consistent (all 48 environment-model-method pairs show positive gains) and span a wide enough range of environments to rule out environment-specific effects. However, the absolute magnitude of improvements varies dramatically: from +0.6 (SearchQA, Llama-3.2-3B, SR) to +18.4 (WebShop, Llama-3.2-3B, IWM). The paper does not discuss statistical significance or confidence intervals for any of these numbers. For environments where the absolute gains are small (SearchQA, WebArena-Lite), the practical significance is unclear without variance estimates—a 0.6 percentage point improvement on SearchQA F1 with a 500-example evaluation subset could plausibly fall within noise. The paper would be strengthened by reporting standard errors or performing paired bootstrap tests, particularly for the small-gain environments.

Claim 2: "Early experience enables capabilities unattainable through imitation learning alone, scaling effectively to achieve comparable or superior performance with only half or even one-eighth of the expert data." Figure 4(a) demonstrates that early-experience-trained models with reduced expert data outperform imitation learning with more data, but the "comparable or superior" claim requires careful reading. On WebShop, IWM with 1/8 demonstrations (38.3%) surpasses imitation with 1/2 demonstrations (44.6%? — actually no, the numbers are: 1/8 IWM 38.3% vs. 1/4 imitation 33.6% vs. 1/2 imitation 44.6%). IWM with 1/8 demonstrations does exceed 1/4 imitation but not 1/2 imitation. The claim "comparable or superior performance with only half" is supported on ALFWorld (IWM with 1/2 demonstrations at 67.2% vs. full imitation at 80.5% — approaching but not matching). On WebShop, IWM with 1/4 demonstrations (43.0%) approaches full imitation (45.3%) but does not exceed it. The claim that one-eighth expert data plus early experience "surpasses full-dataset imitation" is true on WebShop for IWM only if we read the trend as eventually crossing, but the measured 1/8 point (38.3%) is actually below full imitation (45.3%). The claim should be qualified: early experience substantially reduces the expert data requirement, but the exact substitution ratio is environment-dependent and not uniformly 8×.

Claim 3: "Initializing RL with early-experience checkpoints leads to substantially stronger performance compared to standard imitation-learning warm starts." Figure 3 supports this across three environments. However, several caveats are warranted. First, the RL training uses fixed hyperparameters and a fixed number of steps from prior recipes (Feng et al., 2025; Jin et al., 2025). It is possible that imitation-only checkpoints would catch up with more RL steps, different learning rates, or different exploration schedules. The paper does not show learning curves over the course of RL training—only pre- and post-RL bars—so we cannot observe whether the gap is closing. Second, the paper states that "in some cases, the performance gap grows during RL training" but does not provide the intermediate trajectory data to support this claim quantitatively. Third, the environments tested (WebShop, ALFWorld, SearchQA) are the three for which RL infrastructure was available; the paper acknowledges that RL "remains difficult to apply effectively" in the other five environments, meaning the warm-start claim cannot be verified across the full benchmark suite. The claim holds on the evidence presented but is limited to environments where GRPO is known to work.

Claim 4: "Early experience applies seamlessly to larger models, preserving its effectiveness across scales." Figure 5 and Table 10 support this with modest evidence. The WebArena-Lite 70B experiment is limited by LoRA fine-tuning, meaning all methods (imitation, IWM, SR) operate under a parameter-efficiency constraint that may interact differently with the training objectives. The 70B IWM (16.4%) and SR (15.2%) do exceed imitation (13.3%), but the absolute gains (3.1 and 1.9 points) are smaller than at the 8B scale for WebArena-Lite (IWM: +3.6, SR: +3.6, from Table 10). The claim of "seamless" application and "preserved effectiveness" is not strongly supported for the 72B model, where IWM gains 4.9 points over imitation (17.6% vs. 12.7%)—a significant improvement but on a very small absolute base. Only one environment (WebArena-Lite) is tested at scale, so the claim of general scaling behavior across diverse environments is extrapolation.

Potential weaknesses:

• No uncertainty quantification. The paper reports point estimates for all metrics without standard errors, confidence intervals, or formal hypothesis tests. Given the small test sets (e.g., WebArena-Lite: 165 tasks split across 5 sub-domains of ~30 each; TravelPlanner: 180 validation queries; BFCLv3: 75 evaluation tasks), the variance of these estimates could be substantial. A difference of 0.6 F1 on SearchQA or 1.2% on WebArena-Lite might not be statistically significant.

• Missing baselines. The paper does not compare against data augmentation baselines that add noise to expert trajectories (e.g., perturbing states) without environment interaction. This would distinguish whether the gains come from simply seeing more diverse states or specifically from causal interaction (executing actions and observing outcomes). The STaR baseline is a partial control for rationale-based augmentation without interaction, but there is no control for state-level data augmentation without interaction.

• State summarization is un-ablated. The IWM method does not predict raw environment states; it predicts human-written or model-generated summaries of those states. It is possible that the act of generating high-quality state summaries (particularly on WebArena-Lite, where the model itself produces summaries) is doing much of the work, and that training to predict these summaries is essentially training on additional expert-curated data (since the summarizer captures task-relevant information). If so, the mechanism of benefit may be data quality improvement rather than causal dynamics learning. A comparison with IWM trained on raw observations would clarify this, but is absent.

• Confounded training budgets. IWM and SR add additional training data that the imitation learning baseline does not see. While the paper equalizes the number of optimization steps, it does not equalize the amount of information per step (IWM and SR steps process more diverse data). A fairer comparison might give the imitation learning baseline additional expert data (if available) or additional epochs on the same data. The STaR baseline provides a partial control (same amount of additional data, but ungrounded), but does not control for data quantity per se. The data-scaling experiment in Figure 4(a) partially addresses this by showing that early experience with less expert data outperforms imitation with more expert data, but this still confounds data type with data quantity.

• Environment-specific design choices are not systematically compared. The alternative action sampling strategy, the state representation format, the number of alternatives $K$, and the quality filtering thresholds all vary across environments without ablation. This makes it difficult to determine which design choices are essential and which are incidental. The paper would benefit from a sensitivity analysis showing that the gains are robust to these choices within at least one environment.

• Single RL algorithm. The RL warm-start experiment uses only GRPO. Different RL algorithms (PPO, AWR, DPO-based approaches) might have different sensitivities to initialization quality. The claim that early-experience initialization provides "substantially stronger performance" under RL is demonstrated for one algorithm family.

• Limited OOD diversity. The OOD experiments in Table 3 test three environments with pre-defined splits (ALFWorld unseen tasks, SearchQA different QA datasets, BFCLv3 missing-function/long-context variants). These are modest distribution shifts. The paper does not test genuinely out-of-distribution scenarios like transferring a web agent trained on WebShop to WebArena or testing SearchQA-trained models on entirely different retrieval corpora.

Despite these limitations, the breadth of the empirical evaluation—eight environments, three model families, two methods, positive results in every setting—is unusually comprehensive for a systems paper proposing a new training paradigm. The weaknesses are primarily about precision of measurement and isolation of mechanisms, not about the existence or direction of the effect. The paper makes a strong case that early experience provides real, practically meaningful improvements over imitation learning across diverse settings; the open questions are about exactly how large the improvements are, exactly which components are necessary, and exactly how general the scaling behavior is.

6. Limitations and Trade-offs

The Cost of Rollout Data Collection Is Unaccounted For in Headline Efficiency Numbers

The assumption or constraint. The early experience paradigm depends on generating $K$ alternative actions per expert state and executing each one in the environment to collect next states, producing $N \times K$ interaction triples in $D_{\text{rollout}}$. For the self-reflection method, a further step queries a language model on each triple to generate contrastive rationales. The paper measures training cost by the number of optimization steps (held equal across methods) but does not account for the environment interaction cost of collecting $D_{\text{rollout}}$ or the inference cost of generating self-reflection data in any budget calculation. The authors acknowledge this implicitly in their training setup (Section 5.1): "For Implicit World Modeling, we begin with one epoch of the WM objective and then continue supervised updates so that the total updates equal the imitation budget without extra steps." The total training steps are equalized, but the data collection cost is not measured or compared.

The consequence. The magnitude of this unaccounted cost varies dramatically across environments, directly affecting the practical applicability of the methods. On TravelPlanner, the paper performs "exhaustive augmentation by executing ALL available valid actions at each state in the expert trajectories," generating over 70,000 state-transition samples from only 1,395 expert state-action pairs (Appendix B.7). This is approximately a 50× expansion—the agent interacts with the environment 50 times more during data collection than during a standard imitation learning pass. On SearchQA, the model generates 30 alternative actions per state (Appendix B.5), producing approximately $7,691 \times 30 = 230,730$ search queries to external retrieval systems. For a practitioner deploying these methods, the total compute budget includes (1) running the policy to propose alternative actions, (2) executing each action in the environment (which may involve web page loads, API calls, or tool invocations), (3) collecting and storing the resulting states, (4) for SR, running an LLM inference to generate contrastive rationales, and (5) the final training step budget. Only item (5) is held equal to the imitation learning baseline. The true cost of early experience—including data collection—could exceed imitation learning by an order of magnitude or more, making the headline gains over imitation learning potentially less impressive when measured against total resource expenditure rather than training steps alone.

What evidence exists in the paper. The paper does not report environment interaction counts or data collection FLOPs for any experiment. Appendix B provides per-environment descriptions of data collection procedures but does not quantify their cost. The branching factor ablation in Figure 4(b) shows that higher $K$ improves performance for IWM, but does not analyze the cost-performance tradeoff—the paper recommends larger $K$ for IWM without discussing the additional collection cost. The training budget equalization is described only in terms of optimization steps (Section 5.1): "we first explore the number of optimization steps for the Imitation Learning baseline in each environment and select the checkpoint with the lowest training loss as well as the performance on the validation set. We then fix this step budget and use it unchanged for our methods to ensure a fair comparison." This is a fair comparison of training efficiency but not of total system efficiency.

Mitigation status. Not addressed. The paper does not discuss the data collection cost, does not include it in any budget calculation, and does not propose methods to reduce it (e.g., sharing rollouts across states, using cached environment responses, or amortizing collection across training runs). The limitation is not acknowledged in Section 7 (Limitations and Future Work), which focuses instead on extending to long-horizon credit assignment and cross-environment transfer.

---

No Evidence on Hardest Problem Regimes—The Method Presumably Fails Where the Base Policy Has Near-Zero Success

The assumption or constraint. Early experience is seeded from expert trajectories and generates alternative actions by sampling from the current policy $\pi_\theta$. The quality and informativeness of the rollout dataset $D_{\text{rollout}}$ depends critically on the base policy having non-degenerate action proposals. If the policy proposes only nonsensical or uniformly poor alternatives, the resulting state transitions may be uninformative—every alternative leads to an error state with no distinguishing information—and the self-reflection comparisons may have no meaningful contrast to extract. The paper does not study this regime systematically, but the evidence from the hardest environments is suggestive. On WebArena-Lite, even the instruction-tuned Llama-3.1-8B-Instruct achieves only 0.6% zero-shot success (Table 10), meaning the base policy is wrong on 99.4% of states. The early experience gains in this environment are the smallest in absolute terms across all benchmarks: +1.2 to +3.6 percentage points (Table 2).

The consequence. The early experience paradigm may provide diminishing returns—or no returns—on problems where the base policy is not already reasonably competent. This is a fundamental capability bound: early experience amplifies existing knowledge by teaching the agent about the consequences of its own (somewhat reasonable but suboptimal) actions, but it cannot create knowledge from scratch. If the agent's proposed alternatives are uniformly terrible, observing that they all lead to error states teaches the agent only that its policy is bad—not what to do instead. The self-reflection method is particularly vulnerable to this failure mode because it requires the contrast between expert and alternative outcomes to be meaningful. If every alternative produces a generic "action failed" state, the generated rationales become vacuous ("this action fails because it doesn't work"), providing no useful training signal. This limitation is analogous to the paper's own finding about RL: "This performs worst across all tasks and shows unstable training dynamics, highlighting the necessity of a strong initialization" (Section 5.4). Early experience itself requires a strong initialization—the base policy must be good enough to produce informative alternatives.

What evidence exists in the paper. The difficulty-dependent pattern across environments is consistent with this limitation but not explicitly analyzed. Table 2 shows the smallest absolute gains on the two hardest environments: WebArena-Lite (+1.2 to +3.6 percentage points) and SearchQA (+0.6 to +3.3 percentage points). The BFCLv3 sub-split analysis in Table 7 provides partial evidence: on the Miss Function sub-split (where the model must recognize when no available tool can fulfill a request), SR on Llama-3.2-3B produces the largest percentage-point gain (+5.3, from 0.0% to 5.3%), but the absolute performance remains extremely low (5.3%). The TravelPlanner results in Table 9 show a different pattern: despite only 45 expert trajectories, the gains are large (+12.8 to +15.0 percentage points). However, TravelPlanner has a structured, finite action space where even a weak policy can enumerate valid actions—the gap is in constraint satisfaction, not action feasibility. The paper does not analyze how the quality of the initial policy's alternative proposals correlates with early experience gains, and does not test the method on versions of the benchmarks where the base policy is deliberately degraded.

Mitigation status. Not addressed in any systematic way. The paper acknowledges in Section 7 that "extending them to address long-horizon credit assignment without explicit rewards remains an open challenge," but this is about horizon length, not about base policy quality. The authors do not discuss the relationship between initial policy strength and early experience effectiveness, nor do they propose methods for bootstrapping from extremely weak initial policies (e.g., using random exploration or human-guided alternatives). This is a significant gap because the environments where early experience would be most valuable—those where imitation learning fails badly—may be precisely those where the base policy is too weak to generate useful rollout data.

---

The Self-Reflection Prompt and Rationale Quality Are Not Evaluated or Controlled

The assumption or constraint. The self-reflection method depends on a language model generating chain-of-thought explanations $c_i^j$ that explain why the expert action $a_i$ is preferable to an alternative $a_i^j$, grounded in the observed next states $s_{i+1}$ and $s_i^j$. The paper provides a prompt template and guidelines (Section 4.3) and applies quality filtering ("We filter out low-quality generations where the explanation incorrectly supports a non-expert action," Appendix B.8, WebArena-Lite). However, the paper never evaluates the quality, accuracy, or usefulness of the generated rationales themselves. There is no measurement of what fraction of generated rationales are factually correct, logically coherent, or contain hallucinations. The only filtering criterion mentioned is whether "the concluded action did not match the expert action" (Appendix B.3, BFCLv3), which is a minimal syntactic check—it ensures the rationale ends by endorsing the correct action but says nothing about whether the intermediate reasoning is valid.

The consequence. If a substantial fraction of the generated rationales are low-quality—containing hallucinated environment facts, incorrect constraint reasoning, or circular justifications—then training on them may be adding noise rather than signal to the policy. The model would learn to produce plausible-sounding but unreliable reasoning patterns. This is particularly concerning given the paper's own finding about STaR-style data: "The retained rationales are ungrounded, having never been tested in the environment, and frequently hallucinate tools or facts, so fine-tuning on them can even degrade performance" (Section 6.1). Self-reflection rationales are grounded in actual environment states (unlike STaR's), but the reasoning about those states is unverified. An LLM tasked with explaining why the expert action is better might invent plausible-sounding but incorrect constraint logic, attribute outcomes to wrong causes, or rely on generic heuristics ("this action is more efficient") rather than task-specific reasoning. Training on such rationales could teach the model to mimic reasoning patterns that do not actually reflect the environment's structure.

This also creates a confound in interpreting SR's benefits: does SR outperform imitation learning because the contrastive reasoning teaches generalizable decision principles, or because the additional training data (regardless of its content) provides beneficial regularization or increased exposure to state descriptions? The paper cannot answer this question because it does not ablate the content of the rationales—for example, by training on scrambled or nonsensical rationales paired with the same expert actions.

What evidence exists in the paper. The quality filtering details vary across environments and are generally minimal. On WebArena-Lite, the filtering reduces 42,264 potential reflections to 3,190 high-quality examples (a 92.5% rejection rate, Appendix B.8). On BFCLv3, the paper mentions "filtering a small number of low-quality samples" (Appendix B.3). On Tau-Bench, "filter out a small number of low-quality reflection samples" (Appendix B.4). No environment reports the actual quality metrics: what fraction of generated rationales were rejected, what the rejection criteria were beyond action mismatch, or what typical failure modes looked like. The paper does not report examples of rejected rationales, does not measure inter-annotator agreement if human filtering was used, and does not analyze whether the filtering threshold affects downstream performance.

The STaR baseline in Table 4 provides indirect evidence that data quality matters: STaR rationales (ungrounded, never tested in the environment) degrade performance by 22.3 points on WebShop. But this does not isolate the effect of rationale accuracy because STaR also differs in not using alternative actions or state outcomes. A direct ablation—training on SR-style data but with scrambled rationales or rationales from a weaker model—is not performed.

Mitigation status. Partially addressed through filtering but not systematically evaluated. The paper's approach to quality control is pragmatic (filter out obviously wrong rationales) but provides no guarantee about the remaining data. Future work options include human evaluation of rationale quality, automated factuality checks against the observed states, or training the reflection generator itself (rather than using a frozen LLM) to produce more accurate reasoning. None of these are explored.

---

Only Three Environments Test the RL Warm-Start Claim, and the RL Training Is Not Optimized Per-Initialization

The assumption or constraint. Section 5.4 claims that "initializing RL with early-experience checkpoints leads to substantially stronger performance compared to standard imitation-learning warm starts." This claim is demonstrated on three environments: WebShop, ALFWorld, and SearchQA. The choice of these three is dictated by infrastructure availability—"we focus on three reward-available benchmarks" (Section 5.4)—and excludes five of the eight evaluated environments (BFCLv3, Tau-Bench, ScienceWorld, TravelPlanner, WebArena-Lite). For the three tested environments, the RL training uses "identical hyperparameters and training steps as established recipes" from prior work (Feng et al., 2025; Jin et al., 2025), with no per-initialization tuning. The paper states: "The only factor that changes across runs is the initialization: Imitation Learning (IL), Implicit World Modeling (IWM), or Self-Reflection (SR)."

The consequence. The fixed RL hyperparameters create a potential confound: the "established recipes" were developed and tuned for imitation-learning initializations. Early-experience checkpoints may have different learning dynamics—different optimal learning rates, different exploration requirements, different susceptibility to forgetting—that the fixed recipe does not account for. If imitation-learning starts are suboptimally tuned relative to their potential, while early-experience starts happen to align better with the default hyperparameters, the comparison is biased in favor of early experience. The paper does not show RL learning curves (only pre- and post-RL bars in Figure 3), so we cannot observe whether imitation-only starts are improving more slowly but would eventually catch up with more steps, or whether early-experience starts saturate earlier. The claim that "the performance gap persists or even widens" during RL training is asserted without quantitative trajectory data.

More fundamentally, the limited environment coverage means the RL warm-start claim—which the paper positions as one of its three main contributions—is demonstrated on only 3 of 8 tested environments. The five missing environments include TravelPlanner (where SR showed the largest gains over imitation: +15.0 percentage points) and WebArena-Lite (the hardest environment, where all methods struggled). If early experience provides better RL initialization on WebShop and ALFWorld but not on TravelPlanner or WebArena-Lite, the claim's generality is substantially weaker than the paper implies. The missing environments may be precisely those where the RL warm-start benefit is most uncertain, since they lack established RL infrastructure (the reason they were excluded).

What evidence exists in the paper. Figure 3 provides pre- and post-RL comparisons for three environments. The post-RL gap between early-experience and imitation starts is clearly positive in most model-environment pairs, but the absolute gaps vary: WebShop Llama-3.2-3B IWM + GRPO (92.2%) vs. imitation + GRPO (82.0%) is a 10.2-point gap; SearchQA Llama-3.1-8B IWM + GRPO (49.8%) vs. imitation + GRPO (47.1%) is only a 2.7-point gap. The paper does not report whether these gaps are statistically significant. The paper also does not compare early-experience + RL against alternatives like "more imitation learning steps" or "imitation learning on more data"—it compares only against the identical RL recipe applied to different initializations. The claim that early experience is a "practical and scalable bridge to reinforcement learning" depends on the assumption that the demonstrated benefits extend to environments and RL algorithms beyond those tested, which is not established.

Mitigation status. The paper acknowledges the environment limitation implicitly by restricting the claim to "three infra-ready environments" (Figure 3 caption: "Reinforcement learning (GRPO) starting from checkpoints trained with different methods on three infra-ready environments"). The hyperparameter confound is not acknowledged. Future work would need to test the warm-start claim across more diverse RL algorithms and environments, and ideally show that early-experience initializations are Pareto-optimal (better performance at equal compute, or equal performance at lower compute) across a range of RL training budgets, not just at a single fixed step count.

---

The Methods Are Studied Independently—No Combined or Adaptive Variant Is Evaluated

The assumption or constraint. The paper presents implicit world modeling and self-reflection as two distinct strategies for extracting supervision from $D_{\text{rollout}}$ under the unified early experience paradigm. However, the two methods are never combined, compared adaptively across environments, or analyzed for complementarity within a single training run. Each experiment trains with either IWM or SR, not both. The paper's claim in Section 7 that "the proposed two methods under this paradigm... improve both in-domain effectiveness and out-of-domain robustness" treats IWM and SR as parallel options but leaves unanswered the question of whether their benefits are additive, redundant, or conflicting.

The consequence. A practitioner reading this paper faces an ambiguous decision: which method should they deploy for their environment? The paper provides informal guidance—IWM excels "where dynamics are stable" and SR "when shifts affect tool availability, arguments, or retrieval distributions" (Section 5.3 takeaway)—but no systematic framework for making this choice. The pattern in Table 2 shows that the relative advantage of IWM vs. SR varies substantially by environment and model: on WebShop with Llama-3.2-3B, IWM dominates (+18.4 vs. +10.9 for SR); on TravelPlanner with the same model, SR dominates (+12.8 vs. +8.9 for IWM); on WebArena-Lite with Llama-3.1-8B, the two methods tie (+3.6 each). Without an adaptive selection mechanism or an understanding of when each method works best, the practitioner must run both methods and compare, doubling the already-unaccounted-for data collection cost.

More critically, the paper cannot answer the question of whether combining the methods would yield additive gains. Since IWM teaches predictive dynamics and SR teaches contrastive decision principles, it is plausible that a model trained on both objectives would outperform either alone—the predictive knowledge from IWM could ground the contrastive reasoning from SR, while the decision principles from SR could guide the model's use of its internalized dynamics. Conversely, the objectives might interfere: the IWM objective trains the model to predict next states from any action (including suboptimal ones), while the SR objective trains the model to reason about why expert actions are preferable—these could pull the model's representations in conflicting directions. The paper provides no evidence either way.

What evidence exists in the paper. The environment-level performance patterns in Table 2 provide suggestive but non-systematic evidence about complementarity. Environments where both methods show strong gains (e.g., TravelPlanner: IWM +7.8, SR +15.0 on Llama-3.1-8B) might benefit from combination; environments where one method dominates (e.g., WebShop Llama-3.2-3B: IWM +18.4 vs. SR +10.9) might not. The branching factor analysis in Figure 4(b) shows that IWM and SR have different optimal $K$ values, suggesting different data requirements, but this is about data volume, not about combining objectives. The paper does not report any experiment where IWM and SR are trained jointly or where an IWM-pretrained model is further fine-tuned with SR (or vice versa). The limitation is not formally acknowledged.

Mitigation status. Not addressed. The paper does not discuss the tradeoff of choosing between methods, does not propose criteria for method selection, and does not flag the lack of combined experiments as a limitation. Section 7 states that "future work will explore combining early experience with richer self-supervised objectives" but does not mention combining IWM and SR specifically. This is a notable gap because it leaves the central architectural question of the early experience paradigm—how to best extract supervision from $D_{\text{rollout}}$—unresolved. The two methods are both valid but their relationship (complementary? substitutive? context-dependent?) is unknown.

---

Performance on the Hardest Benchmark (WebArena-Lite) Remains Very Low Absolut

The assumption or constraint. WebArena-Lite, a realistic web navigation benchmark with 165 hand-selected challenging tasks across five domains, represents the most complex and open-ended environment in the paper's evaluation suite. State observations are noisy accessibility trees containing hundreds of DOM-like elements; actions require fine-grained element selection; tasks span e-commerce, forums, content management, and map navigation. The best zero-shot performance (GPT-4-Turbo) achieves only 17.6% success rate (Table 10). The early experience methods improve over imitation learning—IWM on Llama-3.1-8B raises performance from 4.9% to 8.5%, SR to 8.5%—but the absolute performance remains below 10% for the 3B and 8B models. Even scaling to 70B parameters with LoRA, IWM reaches only 16.4% and SR 15.2% (imitation: 13.3%), still below the zero-shot performance of proprietary models.

The consequence. This result defines a practical ceiling for the early experience paradigm in its current form: on sufficiently complex, open-ended environments, the methods provide statistically significant but practically modest improvements. A web agent that succeeds on 8.5% of tasks (up from 4.9%) is still failing on over 91% of tasks—not a deployable system. The paper's headline narrative of "consistent improvements across all environments" is technically true but masks the fact that on the hardest benchmark, the improvement does not cross the threshold of practical viability. This matters because WebArena-Lite is the environment most representative of real-world deployment scenarios (actual websites, realistic tasks, noisy observations), and it is precisely where the field most needs better training methods. The early experience gains, while directionally positive, are insufficient to close the gap to usable performance.

The sub-split results in Table 10 reveal that the gains are uneven across domains. For Llama-3.1-8B, IWM improves Reddit from 0.0% to 11.1% and CMS from 0.0% to 7.3%, but Map (11.9% → 8.6% for IWM) and OSS (8.0% → 16.1%) show mixed results. The small per-domain sample sizes (~30 tasks each, since 165 tasks are split across 5 domains) mean these numbers have high variance, but the pattern suggests that early experience does not uniformly help across all task types within a single environment.

What evidence exists in the paper. Table 2 reports the aggregate WebArena-Lite success rates; Table 10 provides the per-domain breakdown. The absolute numbers are transparently low, and the paper does not claim otherwise. The scaling experiment in Figure 5 and Table 10 shows that the gap between imitation and early experience persists at the 70B scale but that even the 70B models remain far below usable performance. The paper does not analyze why WebArena-Lite gains are so much smaller than WebShop gains, despite both being web navigation tasks. Possible explanations include: (1) the base policy quality is too low for informative alternatives, (2) the state representations (summarized accessibility trees) lose critical information, (3) the expert data quality is insufficient, or (4) the environment's action space is too large for the rollout data to provide adequate coverage. None of these hypotheses are tested.

Mitigation status. The paper acknowledges this limitation only implicitly through transparent reporting of the numbers. There is no discussion in Section 7 of why performance on the most realistic benchmark remains low, no analysis of failure modes, and no proposals for how to close the gap. The model scaling experiment (Figure 5) provides a partial mitigation by showing that larger models benefit from early experience on this benchmark, but the absolute ceiling remains far below practical utility. Future work options include: combining early experience with methods specifically designed for web navigation (e.g., better state representations, hierarchical action spaces), using stronger base models for rollout generation, or developing adaptive strategies that allocate more interaction budget to states where the policy is uncertain.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual reframing of intermediate scale—not a paradigm shift, but a diagnostic intervention that changes how the field should think about the space between imitation learning and reinforcement learning for language agents. The contribution is best understood as filling a recognized gap with a formal vocabulary and empirical evidence, rather than overturning existing assumptions. Prior to this work, the implicit consensus was that if you couldn't do RL (no rewards), you did imitation learning (expert demonstrations)—and that was the complete menu. The paper demonstrates that this binary view overlooks a third category of supervision that is available in most real-world deployments: the environment's state transitions themselves, collected from the agent's own exploratory actions, encode actionable knowledge about action quality without any scalar reward.

What makes this more than an incremental methods contribution is that it resolves a tension in prior results that had no obvious resolution within the existing taxonomy. Self-correction at inference time had been shown to fail on reasoning tasks (Huang et al., 2024; Valmeekam et al., 2023). World models for language agents existed (Gu et al., 2025; Chae et al., 2025) but were treated as external planning modules, not integrated into policy training. STaR-style rationale bootstrapping (Zelikman et al., 2022) sometimes helped and sometimes hurt, with no clear explanation of when or why. The paper's reframing explains these inconsistencies through a single lens: grounding matters. Inference-time self-correction fails because the model has no environment feedback to anchor its reasoning. STaR's rationales degrade performance (Table 4: -22.3 points on WebShop, -6.3 on ALFWorld) because they are generated without observing actual state transitions. World models help when they capture real dynamics but are expensive to build as separate systems. Early experience provides the grounding—observed next states from executed alternative actions—as a training signal, and both IWM and SR benefit from it because both methods anchor their supervision in actual environment outcomes.

This reframing redirects research attention in several concrete ways:

From "more expert data" to "better interaction data." The dominant response to poor agent performance has been to scale expert demonstrations—collect more human annotations, generate more synthetic trajectories from stronger models. The paper's data-scaling experiment (Figure 4a) provides evidence that this is not the only path, and possibly not the most efficient one. With 1/8 of the expert data augmented by early experience, IWM on WebShop achieves 38.3%, already surpassing imitation learning on 1/4 of the demonstrations (33.6%). The implication is that a marginal dollar of investment in environment interaction infrastructure (making it cheap to execute and collect alternative actions) may yield higher returns than a marginal dollar spent on additional expert annotation. This does not make expert data obsolete—early experience is seeded from expert trajectories and cannot replace them entirely—but it shifts the optimal allocation toward interaction.

From inference-time scaling to training-time interaction learning. Test-time compute scaling (Snell et al., 2024) has emerged as a dominant paradigm for improving LLM outputs without retraining. The paper's Long CoT baseline (Table 4) provides a striking counterpoint: applying longer chain-of-thought reasoning to imitation-trained models dramatically reduces performance (WebShop: 47.3% → 0.0%; ALFWorld: 80.5% → 25.8%). This is not a marginal failure—it is catastrophic collapse. The diagnosis is that "once fine-tuned only on expert trajectories lacking inherent rationales, models lose the ability to sustain coherent long-form reasoning" (Section 6.1). The implication is that inference-time scaling and training-time interaction learning are not substitutes or even orthogonal—inference-time reasoning quality depends on training-time exposure to reasoning patterns, and interaction-based training provides precisely the kind of grounded reasoning practice (through SR) that supports effective inference-time chain-of-thought. This finding should make the field more cautious about claiming that test-time compute can compensate for training deficiencies.

From world models as separate systems to world models as policy-internal representations. The paper's implicit world modeling approach—training the policy itself to predict next states rather than building a separate simulator—is a methodological bet that the boundary between world knowledge and decision knowledge is a training design choice, not an architectural necessity. The success of this approach across eight environments (IWM improves over imitation in all of them, with gains up to +18.4 percentage points on WebShop) suggests that the overhead of maintaining separate world model modules may not be justified for many language agent applications. This could redirect research effort from building more sophisticated external world models toward developing better objectives for integrating predictive and decision-making capabilities within a single model—a convergence toward what the paper calls "unified representation of environment knowledge" (Section 4.2).

More attractive research directions: (1) Methods for extracting additional forms of supervision from interaction data beyond next-state prediction and contrastive rationales (the paper explicitly invites this in Section 7). (2) Adaptive strategies that dynamically choose between IWM, SR, or other objectives based on environment characteristics or training progress. (3) Integration of early experience with continual learning, where interaction data collected during deployment continually refines the policy. (4) Combining early experience with stronger base models for rollout generation, since the quality of alternative actions depends on the policy's competence—a virtuous cycle.

Less attractive research directions: (1) Inference-time self-correction methods that do not involve environment interaction, given the paper's negative Long CoT results and the broader literature's evidence of their unreliability without external feedback. (2) Purely static data augmentation for agent training (e.g., paraphrasing expert trajectories, adding noise to states), since the STaR baseline shows that ungrounded augmentation can be counterproductive—the environment interaction component appears essential. (3) Complex external world model architectures for language agents, unless they can demonstrate benefits over the simpler implicit approach that justify their additional complexity and inference-time cost.

---

Follow-Up Research This Work Enables

Cheap difficulty estimation and adaptive branching. The branching factor $K$—the number of alternative actions per expert state—is a critical hyperparameter with environment-specific and method-specific optimal values (Figure 4b). IWM improves monotonically with larger $K$ (more transition data is always better for predictive learning), while SR peaks at moderate $K$ (2–4) and degrades at larger values because "comparing many alternatives occasionally includes other success-leading actions, reducing contrast with the expert" (Section 6.3). This creates a practical allocation problem: the cost of collecting rollout data scales linearly with $K$, but the benefit function differs by method and environment. A natural follow-up would train a lightweight classifier—operating on the state representation $s_i$ alone—to predict how many alternative actions are worth collecting at that state. States where the policy has high uncertainty (high entropy over actions) or where the expert action has a narrow margin of superiority might benefit from larger $K$; states where the expert action is unambiguously correct might need only $K = 1$. Such a system could allocate a fixed interaction budget adaptively across states, achieving higher performance at equal cost. The paper's existing infrastructure for collecting $D_{\text{rollout}}$ at various $K$ values provides the exact training data needed for this classifier: for a held-out set of states, collect rollouts at $K = 8$, measure the marginal benefit of each additional alternative (in terms of downstream policy improvement), and train the classifier to predict this marginal benefit from the state text alone. A strong result would show that adaptive $K$ selection matches the performance of fixed $K = 8$ at a fraction of the interaction cost.

Combining IWM and SR in a single training pipeline. The paper studies IWM and SR as independent methods, never combining them within a single training run. This is a conspicuous gap because the methods target complementary forms of knowledge: IWM captures mechanical transition regularities, SR captures contrastive decision principles. A combined training pipeline might interleave IWM and SR objectives, use IWM as a pretraining stage before SR (analogous to the two-stage IWM pipeline but with SR as the second stage), or train on both objectives jointly with a mixing ratio. The key experiment would compare four conditions within a single environment: IWM alone, SR alone, IWM → SR (sequential), and IWM + SR (joint training), all at equal total optimization steps. The prediction from the paper's analysis is that sequential IWM → SR would outperform either method alone, because IWM provides a predictive foundation (knowledge of what happens after each action) that SR can then build upon with contrastive reasoning. Joint training might underperform sequential if the objectives interfere. If combined training shows additive gains (e.g., IWM + SR achieves 70% on WebShop where IWM alone achieves 60%), it would establish that the knowledge types are genuinely complementary rather than overlapping. If combined training shows no improvement over the better single method, it would suggest that the two objectives extract largely redundant information from $D_{\text{rollout}}$, which would be an important negative result clarifying the supervision structure.

Stress-testing the base policy quality lower bound. The paper's results suggest but do not establish that early experience effectiveness depends on the base policy being non-degenerate—the model must propose sufficiently informative alternative actions for the rollout data to be useful. This hypothesis can be explicitly tested by systematically degrading the base policy used for rollout generation and measuring the downstream performance of early-experience training as a function of base policy quality. On an environment like WebShop (where the untuned Llama-3.1-8B-Instruct achieves 0.0% zero-shot success but IWM still achieves large gains after training on expert trajectories), one could vary the rollout-generation policy by: (1) using the fully trained imitation-learning checkpoint (strong policy, high-quality alternatives), (2) using an intermediate training checkpoint (medium policy), (3) using the raw instruction-tuned model (weak policy, the default in the paper), and (4) using a deliberately degraded policy with increased temperature or noise. The prediction is that IWM gains should degrade gracefully with rollout policy quality (since even random actions can provide informative state transitions in deterministic environments), while SR gains should degrade sharply (since contrastive reasoning requires alternatives that are plausible enough to create meaningful comparisons). If both methods prove robust to very weak rollout policies, it would significantly expand the applicability of early experience to domains where no reasonable initial policy exists. If SR collapses below some quality threshold, it would identify a critical precondition that practitioners must verify before deploying SR in new environments.

Cross-environment transfer of early-experience-trained representations. The OOD results in Table 3 test generalization within the same environment under distribution shift (different tasks, different tools). A more ambitious question is whether the knowledge internalized through early experience transfers across environments. For example, does a policy trained with IWM on WebShop (e-commerce navigation) perform better when fine-tuned on WebArena-Lite (general web navigation) than a policy trained with imitation learning on the same WebShop data? This tests whether early experience builds generalizable "world interaction" skills—understanding that clicking elements changes pages, that forms require sequential filling, that error messages indicate incorrect actions—versus environment-specific transition memorization. The experiment would pretrain on one environment with IWM or SR, then fine-tune on a target environment with limited expert data, comparing against imitation-only pretraining on the source environment. A positive result (early-experience pretraining on WebShop improves WebArena-Lite fine-tuning more than imitation pretraining does) would position early experience as a general-purpose pretraining objective for language agents, analogous to how language modeling pretraining transfers across downstream NLP tasks. The paper's current results provide suggestive evidence—IWM shows larger OOD gains than in-domain gains on SearchQA (+4.9 vs. +1.0 for Llama-3.2-3B)—but cross-environment transfer has not been tested.

Investigating the mechanism of RL warm-start improvement. The paper demonstrates that early-experience checkpoints serve as better initializations for GRPO than imitation-only checkpoints (Figure 3), but does not explain why. The benefit could arise from several non-mutually-exclusive mechanisms: (1) early-experience models have better exploration during RL (they try more diverse actions because they've seen more diverse transitions), (2) early-experience models have better credit assignment (they can more accurately predict which actions led to good or bad outcomes because they've internalized transition dynamics), (3) early-experience models start closer to a good policy, so RL fine-tunes rather than relearns from scratch, or (4) early-experience models have more robust representations that are less disrupted by RL's policy updates. These mechanisms can be distinguished through targeted experiments. To test the exploration hypothesis, compare the entropy of action distributions and the diversity of states visited during the first N steps of RL for early-experience vs. imitation initializations. To test the credit assignment hypothesis, measure the accuracy of value function predictions early in RL training. To test the representation robustness hypothesis, measure the change in model weights (L2 distance) or representational similarity (CKA) between pre-RL and post-RL checkpoints for different initializations. If early-experience initializations show less representational change during RL while achieving higher final performance, it suggests that the benefit comes from providing a better starting representation that RL refines rather than overhauls. This mechanistic understanding would guide the design of future warm-start strategies and could inform when early experience is most valuable relative to other initialization methods.

Scaling laws for interaction data. The paper's branching factor analysis (Figure 4b) and data fraction analysis (Figure 4a) hint at scaling relationships but are too small-scale to establish functional forms. A systematic scaling study would measure test performance as a function of (1) the number of expert trajectories $N$, (2) the number of alternative actions per state $K$, and (3) the total interaction budget $N \times K$, across multiple environments and model sizes. The goal would be to fit scaling laws of the form $\text{Performance}(N, K) = f(N, K)$ and identify the optimal allocation of a fixed interaction budget between collecting more expert states (larger $N$) and branching more from each state (larger $K$). For IWM, the paper's results suggest that $K$ is highly beneficial (monotonic improvement), implying that for a fixed interaction budget, branching more from fewer states may outperform broader but shallower coverage. For SR, the peaked $K$ curve suggests an interior optimum. Such scaling laws would provide practical guidance for practitioners allocating limited environment interaction budgets, analogous to how Chinchilla scaling laws guide the allocation of pretraining compute between model size and data quantity. The paper's existing infrastructure (eight environments, multiple model sizes, variable $K$ and data fractions) provides a strong foundation for this study, though it would require larger-scale experiments—particularly at higher $K$ values and larger model sizes—to fit reliable functional forms.

---

Practical Applications and Downstream Use Cases

Data-efficient agent training for enterprise tool integration. Large organizations deploying language agents for internal tool use (e.g., navigating proprietary web interfaces, interacting with internal APIs, automating multi-step workflows) face a specific bottleneck: the expert demonstrations needed for imitation learning must be collected for each new tool or workflow, and the cost scales with the number of internal systems. The paper's data-scaling results (Figure 4a) suggest that early experience can reduce this cost substantially. On WebShop, IWM with 1/8 of the expert demonstrations (38.3%) already exceeds imitation learning on 1/4 of the data (33.6%), and approaches 1/2-data imitation (44.6%). In an enterprise context, this means that a team integrating a new internal CRM or ERP system might need to collect expert demonstrations for only a fraction of the typical task coverage, then let the agent explore alternative actions on the remaining states to generate its own supervision. The infrastructure requirement is modest: the environment (the internal tool) must support executing actions and returning states—exactly what it already does during deployment. The expert data savings compound across systems: if an organization maintains 10 internal tools, reducing the annotation requirement by 4× per tool translates to a substantial reduction in total human effort. The SR method provides an additional benefit in this setting: the contrastive rationales generated during training (e.g., "clicking this button would have deleted the record; the expert action saves it instead") serve as interpretable documentation of the agent's learned decision principles, which is valuable for auditing and compliance in enterprise deployments.

Bootstrapping agents for websites without reward APIs. Most public websites do not expose reward signals—a flight booking site does not tell you whether you selected the optimal itinerary, a government form portal does not confirm that all fields are correctly filled. Training agents for such sites today requires either expensive human annotation or brittle heuristic reward functions. The early experience paradigm is directly applicable: an organization wanting to deploy an agent on a target website needs only (1) a modest number of expert trajectories (collected once by a human or a strong model) and (2) the ability to execute alternative actions and observe resulting page states (which any browser automation framework provides). The WebArena-Lite results (Table 10), while modest in absolute terms (8.5% for both IWM and SR on Llama-3.1-8B), represent a 1.7× improvement over imitation learning (4.9%) with no additional human annotation and no reward engineering. For a specific target website with more constrained action spaces than general web navigation—say, a particular airline booking site or a specific government form—the absolute performance would likely be substantially higher. The key practical insight is that early experience converts the agent's own failed attempts into training signal: every time the agent clicks the wrong button and observes the resulting error page, that interaction becomes a data point for IWM (predicting the error state) or SR (reasoning about why the expert's alternative choice was correct). This turns a liability—the agent's inevitable mistakes—into an asset.

Warm-starting RL for newly instrumented environments. As RL infrastructure matures for language agent environments—better simulators, standard reset mechanisms, scalable evaluation platforms—organizations will face the transition from imitation-based to reward-driven training. The paper's RL warm-start results (Figure 3) provide a concrete migration path. When an environment that previously lacked verifiable rewards becomes instrumented with a reward function (e.g., a website adds an API that confirms task completion, or a simulator adds a success metric), the organization should not discard their existing imitation-trained agents and retrain from scratch with RL. Instead, they should first train with early experience on the existing expert trajectories (which requires no rewards), then apply RL starting from the early-experience checkpoint. The post-RL performance gap is substantial: on WebShop with Llama-3.2-3B, IWM + GRPO (97.7%) substantially outperforms imitation + GRPO (92.2%). If the organization had skipped early experience and gone directly from imitation to RL, they would leave approximately 5.5 percentage points of success rate on the table—performance that cannot be recovered by additional RL training at the same step budget (the paper's results show the gap persisting rather than closing). The practical implication is that early experience is not a temporary workaround for the pre-RL era but a permanent component of the optimal training pipeline—it provides a form of pretraining that RL cannot fully replicate.

---

When to Prefer This Method

The paper does not articulate an explicit decision rule for choosing between IWM, SR, or neither. However, the empirical patterns across environments suggest conditions that can be extracted:

• Prefer implicit world modeling when the environment has stable, deterministic transition dynamics and the primary challenge is learning the mapping from actions to outcomes rather than choosing among actions based on complex constraints. Evidence: IWM dominates on WebShop (+18.4 for Llama-3.2-3B vs. +10.9 for SR) and ALFWorld (+5.5 vs. +7.8 for SR, with IWM stronger on two of three models), where clicking a button predictably navigates to a page and picking up an object predictably adds it to inventory.

• Prefer self-reflection when tasks require multi-step constraint satisfaction with combinatorially many valid paths, and the challenge is understanding why one action is preferable given interacting requirements (budget, preferences, sequencing rules). Evidence: SR dominates on TravelPlanner (+15.0 for both Qwen-2.5-7B and Llama-3.1-8B vs. +5.5 and +7.8 for IWM) and ScienceWorld (+13.3 for Llama-3.1-8B vs. +2.3 for IWM), both environments where task success depends on satisfying multiple simultaneous constraints.

• Prefer imitation learning alone when the environment provides comprehensive expert coverage of all relevant states and distribution shift is minimal during deployment. Evidence: the paper provides no environment where imitation learning outperforms early experience, suggesting that early experience is beneficial whenever the prerequisites (expert trajectories + ability to execute alternative actions) are met.

• Consider skipping early experience when the base policy is too weak to generate informative alternatives (near-zero success rate on the target task distribution) and no stronger model is available for rollout generation. Evidence: the paper does not directly test this condition, but the smallest early experience gains appear on WebArena-Lite (the hardest environment, where base policies achieve 0.0–1.2% zero-shot performance), consistent with the hypothesis that early experience benefits scale with base policy quality. This condition is speculative—the paper's data is suggestive but not confirmatory.

• The methods are not interchangeable. The branching factor analysis (Figure 4b) shows that IWM benefits from larger $K$ while SR degrades at large $K$, implying different data collection strategies. The OOD results (Table 3) show IWM dominating on ALFWorld OOD (+14.8 vs. +9.4 for SR on Llama-3.1-8B) while SR dominates on BFCLv3 OOD (+8.5 vs. +3.6 for IWM on Llama-3.2-3B). Practitioners should evaluate both methods on their target environment rather than assuming one is universally superior.