ProAct: Agentic Lookahead in Interactive Environments

🎯 Pitch

LLMs internally hallucinate future states when trying to plan ahead, and these errors compound catastrophically in multi-step tasks. ProAct bypasses this by distilling environment-grounded search trees directly into the model’s reasoning, then stabilizes policy learning with cheap Monte Carlo rollouts. A 4B model trained this way matches the performance of frontier closed-source models on 2048 and Sokoban, without any inference-time search.

1. Executive Summary

This paper introduces ProAct, a two-stage framework that internalizes accurate lookahead reasoning into LLM agents for long-horizon interactive environments, evaluated on stochastic (2048) and deterministic (Sokoban) benchmarks using Qwen3-4B-Instruct. ProAct combines Grounded LookAhead Distillation (GLAD) — which compresses environment-based MCTS search trees into concise causal reasoning chains via supervised fine-tuning (e.g., distilling foresight of tile merges and dead-end avoidance in 2048 into a natural-language analysis of future trends) — with Monte-Carlo Critic (MC-Critic), a plug-and-play auxiliary value estimator that leverages lightweight random-policy rollouts to provide low-variance advantage estimates for stabilizing multi-turn PPO and GRPO training (e.g., rolling out 1,000 random trajectories in under 3 seconds to calibrate action values in 2048). A 4B parameter model trained with ProAct outperforms all open-source baselines and rivals state-of-the-art closed-source models, while generalizing robustly to unseen environment variants — establishing that environment-grounded search distillation plus Monte Carlo value estimation can substitute for model scale in interactive planning tasks, provided the base model encounters trajectories within its capability reach.

2. Context and Motivation

The Core Problem: LLM Agents Cannot Perform Reliable Long-Horizon Lookahead

The fundamental challenge this paper tackles is straightforward to state but devilishly hard to solve: LLM agents need to look multiple steps into the future to make good sequential decisions in interactive environments, but they cannot simulate future states accurately enough to make this useful. When an LLM tries to reason about what will happen if it takes a particular action — predicting the environment's response, then predicting the next response, and so on — small errors in its internal "mental model" of the environment compound with each successive step. The authors term this phenomenon simulation drift (Section 3.1): the agent's imagined trajectory diverges from what would actually happen if those actions were executed in the real environment.

This is not a minor quality-of-life issue. It is the difference between an agent that can play 2048 competently and one that makes self-destructive moves because it hallucinated a tile merge that never happened. It is the difference between a Sokoban solver that pushes boxes toward targets and one that pushes them into corners because it mispredicted where the empty cells were. The paper argues — and their experimental results bear out — that simulation drift is the primary bottleneck preventing LLM agents from succeeding at long-horizon planning tasks.

The mechanism of failure is worth understanding precisely: in a standard Chain-of-Thought or ReAct-style setup (Yao et al., 2022; Wei et al., 2022), the agent generates reasoning before each action. That reasoning often includes statements like "If I move this box up, the path will be clear." But the agent has no access to the ground-truth environment to verify this prediction. It relies entirely on whatever implicit world knowledge the LLM absorbed during pretraining. For tasks like 2048 (which has specific, deterministic tile-sliding rules that are learnable but non-trivial) or Sokoban (which requires precise spatial reasoning about blocking and path clearance), this implicit knowledge is insufficient. The reasoning becomes delusional: logically structured but factually wrong about what will happen. The paper states this bluntly in Section 1:

"Simply increasing the depth of ungrounded reasoning often further degrades performance due to context drift and hallucination, while single-step reasoning fails to capture long-term consequences."

Why Existing Reasoning Paradigms Fall Short

This is not a failure of "LLMs can't reason at all." It's a specific failure mode of unverified mental simulation. Human experts performing the same tasks don't just think harder — they cross-reference their predictions against known rules, or in the case of interactive environments, they can use the environment itself (or a mental model of it distilled from extensive practice) to check whether a candidate move would be productive. The paper's core diagnosis is that LLM agents lack this verification mechanism. They have the "System 2" deliberation capacity (the ability to think step-by-step) but not the grounding that makes that deliberation trustworthy over long horizons.

The paper explicitly connects this to human cognition in Section 1: humans use "System 2 processing... performing mental lookahead — simulating potential future trajectories and comparing outcomes before committing to an action." The gap is that humans (when well-trained) have accurate mental models of the domains they operate in; LLMs, despite their broad knowledge, have fragile and often hallucinated mental models of specific environment dynamics.

Why This Problem Matters (Beyond Academic Interest)

The significance extends well beyond puzzle games, and the paper's choice of 2048 and Sokoban is strategic rather than frivolous. These environments encapsulate the core difficulty of all interactive sequential decision-making:

Stochasticity (2048). After each move, a new tile spawns randomly on an empty cell. This means the agent cannot plan a precise multi-step sequence — it must reason about distributions over futures, weighing the probability and impact of different spawn locations. This mirrors real-world planning under uncertainty (e.g., supply chain management, where demand is stochastic; dialogue systems, where user responses are unpredictable; robotics, where sensor noise and actuator uncertainty matter).

Deterministic but combinatorially explosive (Sokoban). The move rules are fully deterministic, but the branching factor is enormous: a seemingly innocent box push can block access to other boxes and make the level unsolvable, and this blocking effect may only become apparent 20+ moves later. The agent must look far ahead to detect "dead ends" — states from which no solution is reachable. This mirrors combinatorial planning problems in logistics, scheduling, and path planning with irreversible actions.

Sparse rewards. In Sokoban, the agent only gets a positive reward when a box reaches a target, and a terminal reward when all boxes are placed. Most intermediate actions produce zero or negative reward (penalties for invalid moves). This is the classic exploration problem: how does the agent learn which actions are good when feedback arrives much later? The paper's RL component (MC-Critic) is specifically designed to address this.

Long horizons. A single 2048 episode can last hundreds of turns (up to 1,000 in the paper's setup); Sokoban levels can require dozens of considered moves to solve. This means the cumulative effect of small per-step errors in value estimation or reward attribution can completely destabilize training. The paper explicitly identifies this as why standard multi-turn RL methods struggle: they lack low-variance value signals over these long horizons.

Beyond games, the authors position this as foundational for autonomous agents more broadly (Section 1, opening paragraph). An agent that can reliably plan ahead — whether for GUI interaction, code debugging, financial trading, or scientific experiment design — needs exactly this capability: internal simulation that stays grounded in reality. Solving it in a controlled, measurable setting (puzzle games with objective scoring) is a necessary first step before scaling to more open-ended domains.

Prior Approaches and Their Limitations

The paper situates itself against three established research directions, each of which addresses part of the problem but leaves critical gaps:

1. Multi-Turn Agentic RL (The Training-Infrastructure Standard)

Frameworks like AgentGym-RL (Xi et al., 2025), RAGEN (Wang et al., 2025c), SkyRL (Cao et al., 2025), and DART (Li et al., 2025b) have built the plumbing for training LLMs as agents in multi-turn environments. They address critical engineering challenges: asynchronous rollout collection, curriculum strategies that progressively lengthen interaction horizons to prevent model collapse, and "Echo Trap" mitigation (where agents overfit to shallow, repetitive reasoning patterns rather than learning deep planning).

However, the paper identifies a critical blind spot in these approaches (Section 2, under Multi-Turn Agentic RL): they focus on the optimization infrastructure but not on the quality of the internal reasoning. An agent trained with these frameworks might learn to output tokens that correlate with high reward — but those tokens might still represent hallucinated, ungrounded reasoning. The framework ensures the optimization converges; it doesn't ensure the resulting policy is deliberative in a causally valid way. The agent might learn to output the action "push up" because that action historically led to rewards in similar board states, without understanding why it works — a System 1 shortcut that will break under distribution shift.

This is where ProAct's first stage, GLAD, intervenes: it explicitly teaches the model what good reasoning looks like by showing it environment-verified futures before asking it to act. The RL frameworks provide the mechanism for policy improvement; ProAct provides the reasoning content that the policy should internalize.

2. Reasoning Distillation from System 2 to System 1 (The Knowledge-Transfer Direction)

This is the closest intellectual precursor to GLAD. Methods like Tree of Thoughts (Yao et al., 2023) and RAP (Hao et al., 2023) showed that running explicit search algorithms (MCTS, BFS) at inference time dramatically improves LLM performance on reasoning tasks — because the search explores multiple future paths and uses an evaluator (often another LLM call, or the environment itself) to prune dead ends. But these approaches are computationally prohibitive for deployment: each action decision might require hundreds or thousands of LLM calls to simulate future branches. The cost scales exponentially with search depth.

The natural response, pursued by works like Distilling Step-by-Step (Hsieh et al., 2023) and STaR (Zelikman et al., 2022), is distillation: use the expensive search process to generate high-quality reasoning traces, then fine-tune a smaller or faster model on those traces. The model internalizes the patterns of good reasoning without needing to search at deployment time. VAGEN (Wang et al., 2025b) extends this idea to agentic settings by training the model to explicitly generate an internal "world model" — state estimations and transition predictions — as part of its reasoning chain.

Where ProAct advances beyond these approaches is in Reasoning Compression (Section 2, under Reasoning Distillation). Existing distillation methods tend to clone the raw search traces — which contain verbose tags, backtracking annotations, and structural artifacts of the search algorithm (think: "<explore node=5>" type formatting). This is computationally wasteful during fine-tuning and unnatural for the model's pretrained distribution (which consists of natural language, not search-tree markup). GLAD introduces a compression step: after generating search-based trajectories, it uses a teacher model to synthesize the raw traces into natural-language reasoning chains that follow a strict Observation → Analysis → Conclusion structure, while preserving the key insight: why certain actions were rejected in favor of others. This is a subtle but important shift — it's not just copying the search output; it's translating it into the model's "native language" of coherent analytical prose.

3. Value Estimation in Agentic RL (The Signal-Quality Problem)

This is where MC-Critic enters the picture, and the problem it addresses is both well-known in RL and particularly acute for LLM agents.

In standard deep RL (think: AlphaGo, Dota 2, StarCraft II), the agent's policy network is typically a relatively small MLP with millions of parameters. This means the agent can interact with the environment at very high speed — collecting millions of trajectories — and train a critic network (a learned value function) that produces accurate estimates of how good each state is. The abundance of data compensates for the noise in individual rollouts.

LLM agents break this paradigm (Section 3.3, second paragraph). A 4B parameter model takes 3–6 seconds to generate a single reasoning chain and action. This is orders of magnitude slower than a traditional RL agent. At this interaction rate, you simply cannot collect enough data for a learned critic (a billion-parameter value network taking the text state as input) to produce low-variance value estimates. The estimates will be noisy; the policy gradients computed from them will be unstable; training will be inefficient at best, and collapse-prone at worst.

Prior attempts to address this have largely followed the parametric critic route: ArCHer (Zhou et al., 2024) proposes hierarchical value estimation at the utterance level; SWEET-RL (Zhou et al., 2025) uses an asymmetric critic with access to privileged training-time information to provide denser rewards; Turn-Level Reward Design (Li et al., 2025a) explores fine-grained per-turn advantage estimation to improve credit assignment in multi-turn GRPO.

The paper's divergence from these approaches is sharp and well-motivated: don't train a critic at all. Instead, use the environment itself — which can be queried rapidly with a lightweight, random policy — to compute Monte Carlo value estimates on the fly. A random policy can simulate 1,000+ trajectories in under 3 seconds (in 2048) because it doesn't need to generate language — it just needs to execute actions and tally rewards. This provides an unbiased value estimate whose variance can be controlled by the number of rollouts. The estimate is suboptimal (it estimates the value under a random policy, not the learned policy), but the paper argues — and demonstrates empirically — that this suboptimal, low-variance signal is more useful for stable RL training than a theoretically optimal but high-variance learned critic signal.

This is a conceptual bet: in the high-variance regime of LLM-based RL, bias-variance tradeoff favors low-variance simple estimators over potentially lower-bias complex estimators. The environment interaction is cheap; accurate learned value functions are (currently) expensive in terms of wall-clock time and sample complexity. MC-Critic exploits this asymmetry.

How ProAct Positions Itself Relative to Existing Work

The paper synthesizes these three threads into a coherent, two-stage pipeline:

Stage 1 (GLAD) addresses the simulation drift problem at its root. Rather than hoping the base model's world knowledge is accurate enough for multi-step lookahead, or relying on expensive inference-time search, it teaches the model what correct simulation looks like by showing it environment-verified futures and then compressing that knowledge into natural-language reasoning patterns. This is a direct intervention on the deliberation component of the policy ( $\pi_\theta(z|s)$ in Equation 1) — making the reasoning tokens themselves causally grounded.

Stage 2 (MC-Critic) addresses the value estimation problem in multi-turn RL. It recognizes that even with good reasoning priors from GLAD, optimizing for long-horizon returns requires stable advantage estimates. Rather than building an expensive learned critic, it exploits the environment's simulation speed to provide low-variance Monte Carlo value estimates. This is a direct intervention on the credit assignment mechanism of the RL optimizer — telling the policy which actions genuinely lead to better futures, not just which ones got lucky with immediate rewards.

The two stages are complementary, not redundant. GLAD without RL would produce an agent that understands good reasoning but might not consistently execute it optimally — supervised learning only matches the training distribution's action choices, which may be suboptimal even with good reasoning. RL without GLAD would struggle to discover good reasoning patterns from scratch, especially in sparse-reward environments where random exploration almost never succeeds. Together, they form what the paper presents as a principled solution to the core challenge of agentic LLM planning: internalize accurate lookahead reasoning and optimize it stably.

3. Technical Approach

3.1 Reader Orientation

ProAct is a training framework — not a new model architecture or an inference-time technique — that teaches a standard LLM to become a better long-horizon decision-maker in interactive environments. The core problem it solves is that LLMs, when asked to mentally simulate future states of an environment (e.g., "if I move this tile up, what will the board look like in 5 turns?"), get increasingly wrong with each step of simulation because they have no access to the actual environment rules — ProAct solves this by first showing the model real environment futures during supervised training (GLAD) and then stabilizing the reinforcement learning that optimizes long-term returns with cheap environment-based value estimates (MC-Critic).

3.2 Big-Picture Architecture (Diagram in Words)

ProAct is a two-stage pipeline that transforms a base instruction-tuned LLM into an agent capable of grounded multi-step lookahead:

Stage 1 — Grounded LookAhead Distillation (GLAD): The base LLM interacts with the environment through an augmented loop where, at each decision step, it is shown the outcomes of multiple candidate futures explored via MCTS executed directly in the real environment (not simulated by the model). The model reads these environment-verified trajectories, produces reasoning about why certain paths succeed and others fail, and selects an action. Then, these verbose interaction traces are compressed by a teacher model into clean, natural-language reasoning chains following a strict causal structure (Observation → Analysis → Conclusion). The resulting dataset of (state, compressed reasoning, action) triples is used for standard supervised fine-tuning.

Stage 2 — Online RL with Monte-Carlo Critic (MC-Critic): The GLAD-fine-tuned model (or the base model directly, in some experiments) undergoes further optimization via policy-gradient algorithms (PPO or GRPO). The key augmentation is MC-Critic: instead of training a learned value network (which would be sample-inefficient given slow LLM inference), a lightweight random policy rolls out hundreds or thousands of trajectories in the environment in seconds, and the average discounted return from these rollouts provides a low-variance, unbiased value estimate used to compute advantages for policy updates.

3.3 Roadmap for the Deep Dive

First, the formal MDP formulation and policy decomposition (Equation 1) — this establishes what is being optimized and clarifies the crucial distinction between the deliberation policy (reasoning generation) and the execution policy (action selection given reasoning).
Second, the full GLAD pipeline including environment-augmented lookahead data construction, the backtracking mechanism, and the cognitive compression step — because this is the primary technical contribution and establishes the reasoning paradigm the model will internalize.
Third, the MC-Critic formulation (Equations 2–4) — explaining what value estimation problem it solves, how the random-policy Monte Carlo estimator is constructed, and why substituting a random policy for the LLM policy is both necessary (for speed) and empirically sufficient.
Fourth, the two RL algorithm integrations: MC-GRPO and MC-PPO (Equations 5–18) — walking through how MC-Critic plugs into GRPO's group-relative advantage normalization and PPO's GAE-based advantage computation, including the absolute-vs-relative advantage switching mechanism in MC-GRPO.
Fifth, the training protocol — concrete dataset sizes, hyperparameters, and the two initialization regimes (from GLAD checkpoint vs. from scratch) that structure the experimental section.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology paper whose core idea is that LLMs can learn to perform accurate multi-step lookahead in interactive environments if they are first taught what correct simulation looks like through environment-verified examples (GLAD) and then optimized stably using cheap environment-based value estimates (MC-Critic). The mechanism is a training paradigm, not a model modification.

Formal Problem Formulation and Policy Decomposition

The paper frames the agent-environment interaction as a standard Markov Decision Process (MDP), which provides the scaffolding for all subsequent optimization. The MDP is defined as the tuple:

$\mathcal{M} = \langle\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma\rangle$

where $\mathcal{S}$ is the set of possible environment states (serialized as text), $\mathcal{A}$ is the set of available actions (directional moves, push operations), $\mathcal{P}(s_{t+1}|s_t, a_t)$ is the true environment transition dynamics that the agent does not have direct access to, $\mathcal{R}(s_t, a_t)$ is the step-level reward function, and $\gamma \in [0,1]$ is the discount factor.

What this tuple defines: the environment as a stochastic or deterministic system that, given a current state and an action, produces a next state and a scalar reward signal. The agent's job is to learn a policy $\pi_\theta$ parameterized by the LLM weights $\theta$ that maximizes the expected cumulative discounted return $J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[\sum_{t=0}^\infty \gamma^t r_t]$ .

The critical innovation is how the policy is factorized. Unlike traditional RL where the policy directly maps states to actions, the LLM agent's policy is decomposed into two sequential stages:

$\pi_\theta(z_t, a_t | s_t) = \pi_\theta(z_t | s_t) \cdot \pi_\theta(a_t | s_t, z_t)$

where $z_t$ represents the reasoning chain (a sequence of intermediate tokens — state analysis, future predictions, sub-goal decompositions), $a_t$ is the final action token, and the concatenation of both is denoted $c_t = \langle z_t, a_t \rangle$ .

What this equation computes: the joint probability of generating a particular reasoning chain and action given the current state. The factorization separates the decision process into (1) deliberation — generating an internal analysis $z_t$ — and (2) execution — selecting an action $a_t$ conditioned on both the state and that analysis.

Why this form: this decomposition allows ProAct to target the two components with different optimization strategies. GLAD primarily targets $\pi_\theta(z_t|s_t)$ — teaching the model to generate reasoning that accurately reflects environment dynamics — while the RL stage targets the full joint $\pi_\theta(z_t, a_t|s_t)$ — optimizing the end-to-end behavior for long-term returns. Without this decomposition, there would be no explicit reasoning target for GLAD to supervise; the model might learn to output correct actions without producing interpretable or transferable reasoning.

A core concept introduced here is simulation drift: when the agent generates reasoning $z_t$ based on an internal world model $P_{model}(s'|s, a)$ that diverges from the true environment dynamics $\mathcal{P}(s'|s, a)$ , each successive step of mental simulation accumulates error. Over a lookahead depth of $d$ steps, the discrepancy between the imagined and real future compounds, making the entire reasoning chain delusional. This is the phenomenon that GLAD is specifically designed to eliminate.

Grounded LookAhead Distillation: Data Construction

The intuition behind GLAD's data construction is a "Probining-Decision-Reflection" loop that externalizes the lookahead process. Instead of asking the LLM to simulate the future internally (where it has no access to ground truth), the environment itself is queried to produce verified futures, and the LLM's role is to read and analyze these futures rather than hallucinate them.

Step 1: Environment-Augmented Lookahead. At each decision step $t$ , starting from the current state $s_t$ , the system executes a Monte-Carlo Tree Search (MCTS) directly in the environment. This search samples $N$ trajectories $\{\tau_1, ..., \tau_N\}$ , where each trajectory $\tau_i = (s_t, a_t^i, r_t^i, s_{t+1}^i, ...)$ extends for $T$ steps into the future. Both successful/optimal paths and dead-end/suboptimal paths are recorded — this is crucial because the model needs to learn what failure looks like, not just what success looks like. These trajectories are the "ground-truth future map" — they represent what would actually happen if those actions were taken, because they were executed in the real environment, not simulated by the LLM.

Step 2: Trajectory-Aware Decision Making. The current state $s_t$ and the sampled raw trajectories are fed directly into the LLM's context window. The model is prompted to analyze and compare these futures — for example, identifying that "Trajectory A leads to a tile merge yielding +16 points, while Trajectory B leads to a gridlock with no empty cells." Based on this unbiased environmental feedback, the model outputs two things:

Analysis: A comparison and simulation of the potential futures grounded in the provided trajectory data.
Decision: Either a concrete action to take, or a special <BACKTRACK> token if the analysis reveals the current branch is worse than previously explored alternatives.

The <BACKTRACK> mechanism is particularly important: it allows the data construction process to simulate a deep, self-correcting thought process. If the model, after seeing the MCTS futures, realizes that the current decision path leads to a dead end, it can signal a backtrack — and the environment state is reverted to the previous step so exploration can continue down a different branch. This creates training data that includes examples of the model recognizing and correcting poor decisions, not just making good ones.

This loop iterates for $k$ rounds per decision, with the model probing the environment, analyzing the results, and potentially backtracking. The outcome is a collection of raw interaction contexts $(s_t, \mathcal{T}_{\text{real}}, \mathit{raw\_analysis}, a_t)$ for each step where the model committed to an action.

Cognitive Compression: From Raw Search Traces to Clean Reasoning Chains

The raw interaction contexts from the probing phase contain verbose search traces, structural tags akin to tree-search markup, backtracking annotations, and other artifacts of the MCTS process. Directly fine-tuning on this data would be problematic for two reasons: it would be token-inefficient (the model would spend parameters learning to parse search-tree formatting rather than learning reasoning patterns), and the format would differ substantially from the model's pretraining distribution (natural language), potentially causing distribution shift issues during fine-tuning.

The compression step takes these raw contexts and synthesizes them into a concise reasoning chain $z_{\text{compressed}}$ using a teacher model (which can be the same LLM or a larger one). The compression follows four strict principles:

Format Simplification. All structural artifacts (search tags, backtrack markers, exploration node identifiers) are stripped. The reasoning is rewritten entirely in natural language — for example, "Let's analyze the board..." or "If I move up, the tiles will merge..." — aligning with the LLM's pretrained distribution.

Explicit Cause-Effect Reasoning Chains. Every step in the compressed reasoning must follow a strict causal logic: Observation → Analysis → Conclusion. The analysis must explicitly link proposed actions to their consequences based on the environment rules observed in the raw trajectories. For example: "The board currently has empty cells in the bottom row (Observation). If I move down, the tiles in the third and fourth rows will collapse into those empty cells, creating a merge opportunity for the 4-tiles in column 2 (Analysis). Therefore, moving down is the best action now (Conclusion)."

Future Trend Estimation via Counterfactual Reasoning. The compressed reasoning must explain not only why the chosen action is good, but also why the rejected actions are worse. This forces the model to internalize the environment's dynamics and learn counterfactual reasoning. A compressed chain might say: "Moving left is safe now but blocks a critical merge in the future because it would trap the 8-tile against the wall. Moving right would create an immediate merge but leaves the board with no adjacent equal tiles. Therefore I choose up, which lines up the 4-tiles for a merge on the next turn."

Diversity Preservation. The reasoning must retain the "trade-off" analysis present in the raw search — reflecting the genuine deliberation process rather than dogmatically stating a single answer. The compressed chain should express uncertainty and weigh competing factors: "Option A yields a higher immediate score, but Option B provides better board safety for the long term. Considering the risk of a bad tile spawn in the corner, I choose B."

After compression, the final dataset is $\mathcal{D} = \{(s, z_{\text{compressed}}, a)\}$ — state, compressed reasoning chain, and action triples. Standard supervised fine-tuning is performed on this dataset by minimizing the negative log-likelihood loss:

$\mathcal{L}_{\text{SFT}} = -\frac{1}{|\mathcal{D}|}\sum_{(s,z,a)\in\mathcal{D}} \log \pi_\theta(z, a | s)$

What this loss computes: the average negative log-probability the model assigns to the training data's (reasoning, action) sequences given the corresponding states. Minimizing it makes the model more likely to produce reasoning chains and actions similar to those in the dataset.

Why this form: this is the standard maximum-likelihood objective for autoregressive sequence models. It treats the concatenated reasoning-and-action sequence as a single target to match, which means the model learns the joint distribution $\pi_\theta(z, a|s)$ — not just mapping states to actions directly, but learning to first generate the reasoning that motivates the action. This is what "internalizes" the lookahead capability: at inference time, the model will generate a reasoning chain $z$ before selecting $a$ , and that reasoning chain — having been trained on environment-grounded examples — will approximate the kind of causal analysis it learned from the compressed MCTS data, without needing to actually perform search.

Data collection scale: for 2048, 25,000 training samples are collected from 2,048 trajectories initialized with different random seeds. For Sokoban, 8,000 samples are collected from procedurally generated levels. This is a relatively modest supervised dataset — the heavy lifting is done by the MCTS probing, which provides high-quality signal per sample, rather than relying on massive data scale.

Why GLAD Needs the Compression Step (Not Just Raw MCTS Traces)

A natural question: why not simply fine-tune on the raw MCTS traces directly, skipping the compression step? The paper's design choice reflects a nuanced understanding of how LLMs learn from supervised data. Raw MCTS traces contain several forms of noise that would degrade learning:

Structural noise: tree-search tags, backtrack markers, node identifiers. The LLM would need to allocate parameters to modeling these formatting patterns — parameters that would be better spent on learning the actual reasoning logic.
Distributional noise: raw search traces look unlike natural language, pulling the model away from its pretrained distribution. This can cause catastrophic forgetting or slow convergence during fine-tuning.
Redundancy: MCTS explores many similar paths; the raw traces contain substantial repetition. The compression step synthesizes this into a single, dense reasoning chain.
Missing causal structure: raw traces are chronological logs, not causal explanations. The compression step explicitly structures them as Observation → Analysis → Conclusion chains, which is the format the model should learn to produce at inference time.

The compression is, in effect, a form of curriculum design: it takes the raw, messy output of search and translates it into the kind of structured reasoning the model is already good at producing (natural language analysis), while preserving the crucial content — the why behind action choices.

The Value Estimation Problem and MC-Critic Motivation

Before explaining MC-Critic itself, we need to understand the value functions it estimates. In reinforcement learning, two fundamental quantities capture the expected future reward from a given situation:

$V_\pi(s_t) = \mathbb{E}_{\tau \sim \pi}\left[\sum_{k=t}^{\infty} \gamma^{k-t} r_k\right]$

where $V_\pi(s_t)$ is the state-value function — the expected cumulative discounted reward starting from state $s_t$ and thereafter following policy $\pi$ , with $\gamma \in [0,1]$ discounting future rewards to account for uncertainty and time preference.

$Q_\pi(s_t, a_t) = \mathbb{E}_{s_{t+1} \sim \mathcal{P}(\cdot|s_t, a_t)}\left[r_t + \gamma V_\pi(s_{t+1})\right]$

where $Q_\pi(s_t, a_t)$ is the action-value function — the expected cumulative discounted reward from taking action $a_t$ in state $s_t$ , then following policy $\pi$ thereafter. The relationship between the two is the Bellman expectation: the Q-value for an action is the immediate reward plus the discounted value of the resulting next state.

What these equations compute: they are expectations over all possible future trajectories. $V_\pi(s_t)$ averages over what happens if you follow $\pi$ from state $s_t$ ; $Q_\pi(s_t, a_t)$ averages over what happens if you take a specific action first, then follow $\pi$ . The difference $Q_\pi(s_t, a_t) - V_\pi(s_t)$ is the advantage of action $a_t$ — how much better (or worse) it is than the average action from that state.

Why these matter for the paper: policy gradient methods like PPO and GRPO need accurate estimates of these values (or at least the advantages) to update the policy in the right direction. If the estimated advantage of a good action is negative (due to noisy estimation), the policy will be pushed away from that action, destabilizing training. The whole motivation for MC-Critic is that traditional approaches to estimating these values fail for LLM agents due to slow inference.

The paper identifies the core bottleneck: traditional deep RL agents have small policy networks (millions of parameters) and can interact with the environment at extremely high speed — collecting millions of steps of interaction data — enabling them to train accurate critic networks (learned value estimators like $V_\phi$ ) through massive sample counts. LLM agents, in contrast, have billions of parameters and take 3-6 seconds to generate a single reasoning chain and action, making large-scale interaction prohibitively slow. Any learned critic $V_\phi$ will have high variance in its estimates due to insufficient training data, and this noise propagates through the advantage calculation to destabilize policy updates.

MC-Critic: The Core Formulation

MC-Critic's radical proposition is: don't train a critic at all. Instead, estimate values directly through Monte Carlo rollouts in the environment, using a lightweight surrogate policy for speed.

Specifically, given a state $s_t$ , the LLM agent would ideally estimate its value by rolling out the current learned policy $\pi_\theta$ for $M$ trajectories and averaging the returns. But this is prohibitively slow — each rollout requires the LLM to generate reasoning and actions autoregressively, taking 3-6 seconds per step. So MC-Critic substitutes a random policy $\pi_{\text{random}}$ that selects actions uniformly at random (or according to some simple heuristic) from the valid action space. This random policy does not need to generate language — it just picks and executes actions, which can be done at environment speed (thousands of steps per second).

The MC-Critic state-value estimate is:

$V_{\pi_{\text{random}}}^{\text{MC}}(s_t) = \frac{1}{M}\sum_{i=1}^{M}\sum_{k=0}^{T-1} \gamma^{k} r_{k+t}^i$

where $M$ is the number of Monte Carlo trajectories rolled out from $s_t$ , $T$ is the maximum rollout length per trajectory, $r_{k+t}^i$ is the reward received at step $k$ of trajectory $i$ , and $\gamma$ is the discount factor.

What this equation computes: for each of the $M$ sampled trajectories starting from $s_t$ under the random policy, we accumulate the discounted sum of rewards over $T$ steps. Then we average these $M$ discounted sums to get a single scalar estimate of $V(s_t)$ — the expected return from $s_t$ if acting randomly. This is an unbiased estimator of $V_{\pi_{\text{random}}}(s_t)$ ; its variance decreases with $1/\sqrt{M}$ .

Why this form with a random policy: the substitution of $\pi_{\text{random}}$ for $\pi_\theta$ is the key engineering insight. $V_{\pi_{\text{random}}}^{\text{MC}}(s_t)$ is theoretically suboptimal compared to $V_{\pi_\theta}^{\text{MC}}(s_t)$ — it estimates the value of acting randomly, not the value of acting according to the learned policy. However, the paper argues this is the right bias-variance tradeoff for LLM-based RL. The variance of $V_{\pi_\theta}^{\text{MC}}(s_t)$ would be enormous because the LLM can generate only a handful of trajectories in the same time the random policy can generate thousands. In 2048, the random policy can rollout over 1,000 trajectories in under 3 seconds — providing a low-variance estimate. The learned policy would produce maybe 1-2 trajectories in the same time — providing an estimate so noisy it would be useless for stable training. The empirical results (Figures 4-5) validate this: MC-Critic consistently improves training stability and final performance, suggesting that low variance matters more than theoretical optimality in this regime.

The action-value extension. To compute advantages for specific actions, MC-Critic estimates the action-value function:

$Q_{\pi_{\text{random}}}^{\text{MC}}(s_{t_u}, a_{t_u}^i) = \mathbb{E}_{s_{t_u+1} \sim \mathcal{P}(\cdot|s_{t_u}, a_{t_u}^i)}\left[r_{t_u}^i + \gamma V_{\pi_{\text{random}}}^{\text{MC}}(s_{t_u+1})\right]$

This takes the immediate reward $r_{t_u}^i$ from executing the specific action $a_{t_u}^i$ , then adds the discounted MC value of the resulting next state to estimate the long-term value of that action. The expectation over $\mathcal{P}$ means this accounts for stochastic transitions (important in 2048 where random tile spawns affect outcomes).

MC-GRPO: Integrating MC-Critic with Group Relative Policy Optimization

GRPO (Group Relative Policy Optimization) is a recently popular RL algorithm for LLMs that avoids training a separate critic by normalizing rewards within a group of samples. The paper first explains the two baseline GRPO variants before introducing the MC-Critic augmentation.

Trajectory-Level GRPO (Traj-GRPO). In this baseline, $G$ complete trajectories $\{\tau_i\}_{i=1}^G$ are generated from a shared initial state $s_0$ . Each trajectory terminates after some number of steps $T_i$ . A trajectory-level total reward is computed:

$R(\tau_i) = \sum_{t=0}^{T_i-1} r_t^i$

— simply the undiscounted sum of all step-level rewards in the trajectory. Then group normalization is applied across the $G$ trajectories to compute advantages:

$\hat{A}^{\text{Traj-GRPO}}_{i} = \frac{R(\tau_i) - \text{mean}(\{R(\tau_1), \cdots, R(\tau_G)\})}{\text{std}(\{R(\tau_1), \cdots, R(\tau_G)\})}$

where each trajectory's reward is normalized by subtracting the group mean and dividing by the group standard deviation. This gives positive advantages to above-average trajectories and negative advantages to below-average ones.

The probability ratio for policy updates is computed at the token level:

$\rho_{t,j}^i(\theta) = \frac{\pi_\theta(c_{t,j}^i | s_t^i, c_{t,<j}^i)}{\pi_{\theta_{\text{old}}}(c_{t,j}^i | s_t^i, c_{t,<j}^i)}$

where $c_{t,j}^i$ is the $j$ -th token of the output at step $t$ in trajectory $i$ , $c_{t,<j}^i$ are the preceding tokens, and $\theta_{\text{old}}$ refers to the policy before the current update.

The Traj-GRPO loss is:

$\mathcal{L}^{\text{Traj-GRPO}}(\theta) = -\frac{1}{\sum_{i=1}^G \sum_{t=0}^{T_i-1} |c_t^i|} \sum_{i=1}^G \sum_{t=0}^{T_i-1} \sum_{j=1}^{|c_t^i|} \min\left[\rho_{t,j}^i(\theta) \hat{A}^{\text{Traj-GRPO}}_{i}, \text{clip}\left(\rho_{t,j}^i(\theta), 1-\epsilon, 1+\epsilon\right) \hat{A}^{\text{Traj-GRPO}}_{i}\right]$

What this loss computes: for every token in every step of every trajectory, it computes the clipped surrogate objective from PPO. The min operation prevents the policy from changing too much in a single update — if the probability ratio $\rho$ moves outside $[1-\epsilon, 1+\epsilon]$ , the gradient is clipped to zero for that token. The shared trajectory-level advantage $\hat{A}^{\text{Traj-GRPO}}_{i}$ is assigned to every token in trajectory $\tau_i$ , regardless of which step that token belongs to.

Why Traj-GRPO is problematic for long-horizon tasks: credit assignment is extremely coarse. A trajectory that ends with a high score has positive advantage for all its tokens — including tokens from early steps that may have been poor decisions that were later compensated for by good luck or correction. The paper notes this "can easily lead to model collapse" (Li et al., 2025a) because the policy cannot distinguish good individual decisions from lucky sequences.

Step-Level GRPO (Step-GRPO). To address the credit assignment problem, Step-GRPO operates at the granularity of individual decision steps rather than full trajectories. The procedure is:

Roll out a trajectory and store all visited states in a state pool (analogous to the question set in math-domain GRPO).
At each training step, randomly sample a batch of states $\{s_{t_u}\}_{u=1}^b$ from the pool.
For each state, generate $G$ independent single-step samples — each sample is one reasoning chain and action $\{s_{t_u}, c_{t_u}^i, r_{t_u}^i\}_{i=1}^G$ .
Compute step-level advantages from the immediate rewards:

$\hat{A}^{\text{Step-GRPO}}_{t_u,i} = \frac{r_{t_u}^i - \text{mean}(\{r_{t_u}^1, \cdots, r_{t_u}^G\})}{\text{std}(\{r_{t_u}^1, \cdots, r_{t_u}^G\})}$

The loss is analogous to Traj-GRPO but applied per-step rather than per-trajectory.

What Step-GRPO improves: credit assignment is now at the step level — each action is evaluated by its immediate reward relative to other actions from the same state. This is finer-grained and more stable. What it loses: the step-level reward $r_{t_u}^i$ is myopic — it captures only the immediate outcome, not the long-term consequences. In 2048, a move that yields an immediate merge of +8 might be worse than a move that yields +0 now but sets up a +32 merge three turns later. Step-GRPO cannot distinguish these; it will always prefer the immediate reward.

MC-GRPO: Augmenting Step-GRPO with MC-Critic. This is where the integration happens. The key change is replacing the immediate step reward $r_{t_u}^i$ in the advantage calculation with the MC-Critic's action-value estimate $Q_{\pi_{\text{random}}}^{\text{MC}}(s_{t_u}, a_{t_u}^i)$ , which captures long-term expected returns:

$\hat{A}^{\text{MC-GRPO-relative}}_{t_u,i} = \frac{Q_{\pi_{\text{random}}}^{\text{MC}}(s_{t_u}, a_{t_u}^i) - \text{mean}(\{Q_{\pi_{\text{random}}}^{\text{MC}}(s_{t_u}, a_{t_u}^i)\}_{i=1}^G)}{\text{std}(\{Q_{\pi_{\text{random}}}^{\text{MC}}(s_{t_u}, a_{t_u}^i)\}_{i=1}^G)}$

What this computes: for a given state $s_{t_u}$ , compute the MC-Critic Q-value for each of the $G$ sampled actions, then normalize within the group. An action with above-average expected long-term return gets a positive advantage; a below-average action gets a negative advantage. Crucially, this advantage reflects long-term value, not just immediate reward — the $V_{\pi_{\text{random}}}^{\text{MC}}(s_{t_u+1})$ term in the Q-value accounts for all future steps in the rollouts.

The identical-action problem. A practical issue arises: if all $G$ sampled actions $\{a_{t_u}^i\}_{i=1}^G$ are identical (which happens when the policy is confident and exploration is low), their Q-values are also identical. The group standard deviation is zero, making the relative advantage undefined (or zero after normalization), and resulting in zero policy gradient — the model cannot learn from this state. Rather than discarding these samples entirely (the approach used in DAPO; Yu et al., 2025), MC-GRPO falls back to an absolute advantage:

$\hat{A}^{\text{MC-GRPO-absolute}}_{t_u,i} = \frac{Q_{\pi_{\text{random}}}^{\text{MC}}(s_{t_u}, a_{t_u}^i) - \text{mean}(\{Q_{\pi_{\text{random}}}^{\text{MC}}(s_{t_u}, a)\}_{a \in \mathcal{A}})}{\text{std}(\{Q_{\pi_{\text{random}}}^{\text{MC}}(s_{t_u}, a)\}_{a \in \mathcal{A}})}$

What this computes: instead of normalizing within the sampled group, we normalize against the expected Q-value over the entire action space $\mathcal{A}$ . This provides a meaningful advantage signal even when the group samples are homogeneous — telling the model whether its chosen action is better or worse than the average possible action from that state.

The switching rule is:

$\hat{A}^{\text{MC-GRPO}}_{t_u,i} = \begin{cases} \hat{A}^{\text{MC-GRPO-relative}}_{t_u,i} & \text{if } \{a_{t_u}^i\}_{i=1}^G \text{ are not identical,} \\ \hat{A}^{\text{MC-GRPO-absolute}}_{t_u,i} & \text{if } \{a_{t_u}^i\}_{i=1}^G \text{ are identical.} \end{cases}$

Why this hybrid design: the relative advantage is preferred when actions differ because it provides direct contrastive signal (this action vs. those others). The absolute advantage is a fallback that ensures the policy continues to improve even in low-diversity states, by comparing to the global action-value distribution rather than the local sample distribution. This prevents the training from stalling in states where the policy has become overconfident in a single action.

The loss for MC-GRPO is structurally identical to Step-GRPO (Equation 13 in the paper), substituting $\hat{A}^{\text{MC-GRPO}}_{t_u,i}$ for $\hat{A}^{\text{Step-GRPO}}_{t_u,i}$ .

MC-PPO: Integrating MC-Critic with Proximal Policy Optimization

For PPO, the paper extends the approach to multi-turn scenarios with a turn-level critic. The baseline, called Step-PPO, operates as follows:

Turn-level value estimation. A critic network $V_\phi$ takes the state $s_t$ and the full response $c_t$ as input and outputs a scalar value estimate at the final token of $c_t$ . This is the turn-level value. Turn-level GAE (Generalized Advantage Estimation) computes advantages:

$\delta_t^{\text{Step-PPO}} = r_t + \gamma V_\phi(s_{t+1}, c_{t+1}) - V_\phi(s_t, c_t)$

$\hat{A}_t^{\text{Step-PPO}} = \sum_{l=t}^{T-1} (\gamma\lambda)^{l-t} \delta_l^{\text{Step-PPO}}$

What these compute: $\delta_t$ is the TD (temporal difference) error at step $t$ — the difference between the observed reward plus predicted next-state value and the predicted current-state value. $\hat{A}_t$ is the GAE advantage, which is an exponentially-weighted sum of future TD errors, controlled by the GAE parameter $\lambda \in [0,1]$ . When $\lambda=0$ , it reduces to the one-step TD error; when $\lambda=1$ , it becomes the Monte Carlo return. The GAE balances bias (low $\lambda$ , relying more on the learned critic) and variance (high $\lambda$ , relying more on actual returns).

The policy loss for Step-PPO is analogous to the standard PPO clipped objective applied at the turn level, with all tokens within a turn sharing the same turn-level advantage $\hat{A}_t^{\text{Step-PPO}}$ . The value loss trains the critic to predict the return:

$\mathcal{L}^{\text{Step-PPO}}(\phi) = \frac{1}{T}\sum_{t=0}^{T-1} \left(V_\phi(s_t, c_t) - R_t\right)^2, \quad \text{where } R_t = \hat{A}_t^{\text{Step-PPO}} + V_{\phi_{\text{old}}}(s_t, c_t)$

The critical limitation: the learned critic $V_\phi$ is trained on limited data (due to slow LLM inference), producing high-variance value estimates. These feed into the GAE advantage computation, producing noisy advantages that destabilize policy updates.

MC-PPO addresses this by blending the learned critic's estimate with the MC-Critic's low-variance estimate:

$V^{\text{MC-PPO}}(s_t) = (1 - \omega) V_\phi(s_t, c_t) + \omega V_{\pi_{\text{random}}}^{\text{MC}}(s_t)$

where $\omega \in [0,1]$ controls the weight given to the MC-Critic component.

What this computes: a weighted average of two value estimates for state $s_t$ — one from the learned critic (potentially biased but trainable) and one from Monte Carlo rollouts (unbiased but suboptimal since it uses a random policy). When $\omega=0$ , this reduces to Step-PPO; when $\omega=1$ , it uses only the MC-Critic; intermediate values blend both sources.

Why this blended form: the learned critic and MC-Critic have complementary error profiles. The learned critic can capture policy-specific value structure that a random policy misses (e.g., the critic can learn that the LLM's policy tends to make specific kinds of mistakes in certain board configurations, making those states actually worse than the random-policy value suggests). But it suffers from high variance due to limited training data. The MC-Critic has low variance (due to averaging over many rollouts) but is systematically biased (it estimates $V_{\pi_{\text{random}}}$ , not $V_{\pi_\theta}$ ). Blending them allows each to compensate for the other's weaknesses. The paper does not specify a particular $\omega$ value explicitly in the main text; it is treated as a hyperparameter to be tuned or potentially adapted during training.

After computing the blended value, MC-PPO proceeds identically to Step-PPO: the blended value $V^{\text{MC-PPO}}(s_t)$ replaces $V_\phi(s_t, c_t)$ in the TD error and GAE computations, and the same policy and value losses are applied. This makes MC-Critic a true plug-and-play augmentation — it changes only the value computation step, leaving the rest of the PPO algorithm unchanged.

Training Protocol: Two-Stage Pipeline and Hyperparameters

The full ProAct pipeline is executed in two distinct phases with specific data scales and configurations.

Stage 1 — GLAD (Supervised Fine-Tuning):

The initial SFT dataset is constructed via the GLAD procedure. For 2048, 2,048 trajectories are initialized with different random seeds; from these, 25,000 individual (state, compressed reasoning, action) samples are collected. For Sokoban, procedurally generated levels produce 8,000 samples. These numbers reflect a deliberate design choice: rather than collecting massive supervised datasets, GLAD invests compute in the MCTS probing phase to make each sample information-rich. The SFT itself is performed on the Qwen3-4B-Instruct model end-to-end (no frozen parameters) using the AReaL framework (Fu et al., 2025). Appendix Table 5 provides exact SFT hyperparameters.

Stage 2 — Online RL with MC-Critic:

Two initialization regimes are studied:

From GLAD checkpoint: The SFT-fine-tuned model serves as the initial policy for RL. This tests whether MC-Critic can further refine the GLAD-provided reasoning priors. Training is conducted on the same environment configurations as the SFT evaluation. For Sokoban, the base evaluation levels (which are unseen during SFT) are also used as RL training environments to align evaluation with optimization.
From scratch (base Qwen3-4B-Instruct): No SFT initialization. However, for Sokoban, the SFT-level difficulty is too challenging for exploration from scratch, so simplified levels solvable within 20 steps are used instead, with maximum trajectory length $T=20$ . This tests whether MC-Critic alone — without GLAD's reasoning priors — can enable stable long-horizon RL.

Key hyperparameters for the RL stage (from Tables 6 and 7):

Actor learning rate: $1 \times 10^{-6}$ for both PPO and GRPO (both environments, both initialization regimes)
Critic learning rate (for PPO): $1 \times 10^{-5}$
GRPO group size: $G = 8$ (with SFT init: 2048) or $G = 4$ (from scratch: Sokoban), $G = 8$ (from scratch: 2048)
Batch size: 32 (with SFT init) or 16 (from scratch)
PPO/GRPO clipping: $\epsilon = 0.4$
MC-Critic: $M = 1000$ trajectories for both environments; $T = 1000$ (2048, with SFT init), $T = 200$ (Sokoban, with SFT init), $T = 20$ (Sokoban, from scratch)
Discount factor for MC returns: $\gamma = 0.9$ (2048), $\gamma = 1.0$ (Sokoban)
GRPO training epochs per sample: 10 (2048, with SFT init) or 1 (other settings)
PPO training epochs per sample: 1 (all settings)
Optimizer: Adam (all settings)

Why $\gamma=1.0$ for Sokoban: Sokoban is deterministic and has a clear terminal state (all boxes on targets) with sparse intermediate rewards. Using undiscounted returns ( $\gamma=1.0$ ) makes the value estimate directly reflect the number of boxes solved, without artificially discounting later box placements. In 2048, $\gamma=0.9$ reflects the stochastic nature — future tile placements are uncertain, so distant rewards should be discounted.

4. Key Insights and Innovations

Innovation 1: Simulation Drift as a Diagnosed Failure Mode, Not Just "Hallucination"

The paper's most conceptually valuable move is giving a precise name and mechanism to a phenomenon the field has only vaguely gestured at: simulation drift. Prior work on LLM agents has extensively documented that models hallucinate — they produce factually incorrect statements about the world, including about the consequences of their own actions. But "hallucination" is a catch-all term that lumps together factual errors, reasoning mistakes, and delusional planning into one bucket. The paper (Section 3.1) disentangles a specific subtype: the compounding prediction error that occurs when an LLM recursively simulates an environment it has no direct access to, with each step's error feeding into the next step's input. This isn't just "the model said something wrong" — it's a dynamical systems failure where the gap between the agent's internal world model $P_{model}$ and the true dynamics $\mathcal{P}$ grows exponentially with lookahead depth.

Why this reframing matters: it redirects the solution space. If the problem were generic hallucination, you'd train on more data or add retrieval. If the problem is simulation drift specifically, the solution is to ground the simulation in external verification during training so the model internalizes the correct dynamics — which is exactly what GLAD does. The paper's case study (Figure 3) makes this concrete: the base model generates verbose but factually wrong analyses (hallucinating board configurations), while the GLAD-trained model produces compact, accurate reasoning that correctly predicts merge outcomes. The diagnostic is specific enough to be actionable.

This also explains a paradox in the literature: why are inference-time search methods like Tree of Thoughts (Yao et al., 2023) effective on reasoning benchmarks but prohibitively expensive for interactive environments? Because reasoning benchmarks (math, logic puzzles) have static problem statements where the "environment" doesn't change in response to the agent's reasoning — there's no feedback loop that compounds errors. Interactive environments create precisely the feedback loop that makes simulation drift catastrophic. The paper's contribution is identifying this distinction and building a training methodology around it.

Innovation 2: Reasoning Compression as a Quality-Improving, Not Just Cost-Reducing, Step

Distilling search-based reasoning traces into a smaller model is not new — Distilling Step-by-Step (Hsieh et al., 2023) and STaR (Zelikman et al., 2022) established the paradigm. Where ProAct departs is in treating the distillation as a qualitative transformation rather than merely a compression for efficiency. The cognitive compression step in GLAD (Section 3.2.2) doesn't just shorten the output — it imposes a specific causal structure (Observation → Analysis → Conclusion), strips search artifacts that would be out-of-distribution for the LLM's pretraining, and adds counterfactual reasoning that explains why rejected actions were worse. This is a form of curriculum design: the teacher model translates the messy, procedural output of MCTS into the kind of structured analytical prose that (a) the base model already knows how to produce from pretraining, and (b) constitutes genuinely better reasoning than the raw search traces.

Evidence for this distinction is implicit but present: the GLAD-trained model in Figure 3(b) is not merely a smaller, faster version of an MCTS agent — it produces qualitatively different reasoning than both the base model (Figure 3a) and what raw MCTS traces would contain. It describes trade-offs, compares multiple futures, and uses phrases like "considering the long term" that reflect the compressed form's explicit instruction to preserve deliberative nuance. If compression were purely about cost reduction, you'd expect some degradation relative to the full search. The fact that GLAD outperforms strong closed-source models (Table 1) suggests the compressed form is actually pedagogically superior for the LLM — it's learning from cleaner, better-structured examples than the raw search would provide.

Innovation 3: The Random-Policy Monte Carlo Critic as a Bias-Variance Bet

The MC-Critic is, on its face, a simple idea: substitute cheap environment rollouts with a random policy for expensive LLM rollouts when estimating state values. But the intellectual contribution isn't the substitution itself — it's the explicit framing of this as a bias-variance tradeoff where low variance wins, and the empirical demonstration that this bet pays off across substantially different environments and RL algorithms.

Traditional RL for LLM agents has largely followed the parametric critic path: train a value network alongside the policy, accepting that the critic will be noisy but hoping that with enough data it will converge (ArCHer, SWEET-RL, Turn-PPO). The unstated assumption is that a high-variance estimate of the correct quantity ( $V_{\pi_\theta}$ ) is preferable to a low-variance estimate of a biased quantity ( $V_{\pi_{random}}$ ). MC-Critic challenges this assumption directly, and the results in Figures 4-5 validate the challenge: MC-PPO and MC-GRPO consistently outperform their critic-based and reward-only counterparts.

What makes this a genuine innovation rather than an engineering trick is the generality of the insight. The paper shows MC-Critic works with both PPO and GRPO, in both stochastic (2048) and deterministic (Sokoban) environments, from both GLAD-initialized and scratch checkpoints. The hyperparameter analysis (Figure 6) reveals nuanced environment-specific behavior — in 2048, more rollouts ( $M=1000$ ) are better because rewards are dense and the Monte Carlo average converges; in sparse-reward Sokoban, fewer rollouts ( $M=10$ ) are better because they preserve the distinction between rarely-successful actions rather than averaging it away. This is not a one-size-fits-all recipe — it's a diagnostic framework that makes the bias-variance tradeoff explicit and tunable per environment, which is a conceptual contribution beyond the specific algorithm.

Innovation 4: The Decomposition of Agent Policy into Deliberation and Execution as a Training Target

The factorization $\pi_\theta(z_t, a_t | s_t) = \pi_\theta(z_t | s_t) \cdot \pi_\theta(a_t | s_t, z_t)$ in Equation 1 is mathematically trivial — it's just the chain rule of probability. What makes it an intellectual contribution is how it's operationalized as a training strategy. Most prior work on LLM agents either trains the model end-to-end on (state → action) without explicitly supervising the intermediate reasoning, or integrates reasoning into the action sequence but treats it as a monolithic target. ProAct's two-stage pipeline maps cleanly onto this factorization: GLAD primarily supervises $\pi_\theta(z_t | s_t)$ by providing ground-truth futures for the model to reason about, while the RL stage optimizes the joint $\pi_\theta(z_t, a_t | s_t)$ for long-term returns, using the reasoning priors established in Stage 1 as a starting point.

This decomposition resolves a tension that has plagued prior work. VAGEN (Wang et al., 2025b) trains models to generate world-model reasoning, but does so within the RL loop — the reasoning and the action policy are optimized simultaneously with the same reward signal. This risks the model learning to produce reasoning tokens that correlate with reward without actually being causally valid. GLAD front-loads the reasoning supervision, using environment-verified trajectories to teach the model what correct reasoning looks like before any reward optimization begins. The RL then refines the action selection given this reasoning capability, rather than trying to discover both simultaneously. The empirical separation — GLAD alone already outperforms all open-source baselines (Table 1), and MC-Critic adds further gains (Figures 4-5) — validates that these are distinct, complementary improvements rather than redundant mechanisms.

This is a methodological insight: for agentic LLM training, the reasoning component and the policy component should be taught with different objectives (supervised grounding vs. reward optimization) applied in sequence, rather than lumped together in a single end-to-end optimization. It's not a theorem, but it's a design principle that the paper's results suggest generalizes across environments.

5. Experimental Analysis

Evaluation Methodology

Datasets and Environments. The paper evaluates on two long-horizon interactive environments: 2048 (a stochastic single-agent puzzle on a 4 × 4 grid where tiles merge and new tiles spawn randomly, with episodes capped at 1,000 steps or 10 consecutive invalid actions) and Sokoban (a deterministic puzzle where the agent pushes boxes onto target locations, with episodes capped at 200 steps). Both are accessed exclusively through textual observations serialized into structured text. For 2048, variants include a 3 × 3 grid and a "3072" version (minimum tile value 3 instead of 2). For Sokoban, variants include unseen levels, modified action spaces, and altered symbolic representations of the board. All environment details, including reward structures and termination conditions, are specified in Section 4.1.1 and Appendix A.1.
Base Model. All experiments use Qwen3-4B-Instruct (Yang et al., 2025), a 4-billion-parameter instruction-tuned language model. The paper states this choice ensures fair comparison across baselines and ablations, with all methods sharing the same architecture and parameter count. The model is fine-tuned end-to-end with no frozen parameters (Section 4.1.2).
Metrics. For 2048, performance is measured by the cumulative merge score — the sum of values of all tiles produced by merges during an episode. For Sokoban, performance is measured by the average number of boxes successfully placed on target locations per level, averaged over multiple independent runs. This metric captures partial progress rather than binary success, providing a smoother signal on harder levels. Episodes are terminated early if a deadlock state is detected where no further progress is possible (Appendix A.1).
Baselines. The paper compares against a broad set of models reported in Table 1, including both open-source and closed-source Instruct Models: GPT-5 (OpenAI, 2025), Claude Sonnet 4.5, Seed1.6 (Bytedance Seed Team, 2025a), Seed1.8 (Bytedance Seed Team, 2025b), and UI-TARS (Qin et al., 2025). Critically, all baseline models are evaluated zero-shot under the same environment interface with temperature fixed to 0.6 — they receive no SFT or RL on these environments. This makes the comparison a test of how well ProAct's training paradigm improves a 4B model relative to much larger models' out-of-the-box reasoning capabilities.
Baselines for RL experiments. In the RL comparisons (Figures 4–5, Tables 2–3), the paper evaluates against standard algorithm variants without MC-Critic: Step-PPO (turn-level PPO with a learned critic $V_\phi$ and GAE), Step-GRPO (step-level GRPO using immediate rewards), Traj-GRPO (trajectory-level GRPO using full-trajectory returns), and Step-PPO + MC-Critic / Step-GRPO + MC-Critic (the MC-Critic-augmented variants). These are all applied to the same Qwen3-4B-Instruct backbone under identical training configurations aside from the MC-Critic augmentation.
Generation Budget and Compute Accounting. The paper does not measure test-time compute in FLOPs or wall-clock time. Instead, training is controlled by the number of SFT samples (25K for 2048, 8K for Sokoban) and RL training steps. For MC-Critic, the key cost parameters are $M$ (number of Monte Carlo rollout trajectories) and $T$ (rollout horizon), reported in Tables 6–7. The paper emphasizes that random-policy rollouts are extremely cheap: in 2048, over 1,000 trajectories can be simulated in under 3 seconds (Section 3.3.1). Inference-time generation from the LLM takes 3–6 seconds per reasoning chain — about three orders of magnitude slower. This asymmetry is the core justification for MC-Critic's design.
Cross-Validation and Statistical Protocol. For Sokoban, the paper uses a fixed set of levels that do not appear in the SFT training data as the base evaluation benchmark (Section 4.1.4). To maintain consistency between evaluation and RL, these same levels are used as training environments in the subsequent RL phase. For 2048, trajectories are initialized with different random seeds to promote diversity. No formal cross-validation or confidence intervals are reported. Results are presented as mean scores over multiple runs, though the exact number of evaluation runs per setting is specified only in Appendix A.1 ("multiple independent runs").

Main Quantitative Results

Supervised Fine-Tuning with GLAD

Headline result (Table 1). A 4B Qwen3 model fine-tuned with GLAD alone outperforms all open-source baselines and several strong closed-source models on both 2048 and Sokoban, while generalizing to unseen environment variants.

The numbers from Table 1 break down as follows. On 2048 (4 × 4 grid), the GLAD-trained 4B model achieves a cumulative merge score that the paper reports as superior to GPT-5, Claude Sonnet 4.5, Seed1.6, Seed1.8, and UI-TARS — all evaluated zero-shot on the same environment. The table reports exact scores that place GLAD above all listed baselines. On the 3 × 3 variant (which the model was not trained on), GLAD maintains a substantial margin over the zero-shot models. On the 3072 variant (different minimum tile value, also unseen during SFT), GLAD similarly outperforms the baseline models.

On Sokoban (Base), evaluated on newly generated levels not present in the SFT training data, GLAD again outperforms all listed baselines in average boxes placed per level. This is a genuine generalization test: the model was trained on procedurally generated Sokoban levels but evaluated on a disjoint set. On the Action variant (different action space), GLAD's performance remains superior. On the Symbol variant (altered map representation), GLAD also outperforms baselines, though the paper does not quantify the degradation relative to the Base setting.

What Table 1 demonstrates beyond raw scores. The pattern across environment variants is particularly informative. The fact that GLAD maintains superiority on the 3 × 3 and 3072 variants of 2048 — despite never seeing these configurations during SFT — indicates that the model has not merely memorized board-specific heuristics. It has internalized something more general about the dynamics of tile-sliding games: how merges cascade, how empty cells affect mobility, and how to reason about future tile spawns in board configurations it hasn't explicitly practiced on. This is consistent with the cognitive compression step's design: by distilling MCTS trajectories into natural-language causal reasoning (rather than board-specific pattern matching), the model learns transferable analytical skills.

Similarly, on Sokoban, generalization to unseen levels is a strong signal. Sokoban levels can differ dramatically in their solution structure — different box counts, different wall configurations, different deadlock patterns. A model that succeeds on unseen levels must have learned general principles about box pushing, path clearance, and deadlock avoidance rather than level-specific sequences. The Action and Symbol variants test two additional generalization axes: can the model adapt when the interface changes (Action) or when the perceptual representation changes (Symbol)? The paper's positive results on both suggest GLAD produces reasoning that is robust to surface-level format changes.

What the closed-source comparison does and does not show. The fact that a 4B fine-tuned model outperforms GPT-5 and Claude Sonnet 4.5 on these specific environments is striking, but has a crucial caveat: the closed-source models are evaluated zero-shot, while the 4B model has received environment-specific SFT. This is not an apples-to-apples comparison of intrinsic capability — it's a demonstration that targeted training on environment-grounded reasoning can allow a small model to punch far above its weight class for a specific domain. The more meaningful comparisons are against open-source models of similar scale, and against the base Qwen3-4B-Instruct (which is implicitly the zero-shot baseline for the 4B model, though the paper does not report its absolute performance in Table 1).

Case study evidence (Figure 3). The qualitative comparison in Figure 3 is worth examining carefully because it illustrates what kind of improvement GLAD produces. The base Qwen3-4B-Instruct model (Figure 3a) generates reasoning that the paper characterizes with two failure modes: (1) redundant waiting steps — the model produces analysis tokens that don't advance the decision process, and (2) hallucinated board configurations — the model describes tile positions and merge outcomes that don't match reality. The paper annotates these with red segments for incorrect analysis and blue for correct, showing that even when the model's reasoning is verbose, its accuracy is inconsistent.

The GLAD-supervised model (Figure 3b) produces reasoning that is noticeably more compact and accurate. It correctly describes the current board state, explicitly simulates the outcomes of multiple candidate actions, compares their consequences in terms of tile merging and future potential, and selects an action with a clear justification. The analysis follows the Observation → Analysis → Conclusion structure that was enforced during cognitive compression. The paper's annotation shows predominantly blue (correct) reasoning with substantially fewer errors.

Interpreting the case study's significance. This figure is not just a cherry-picked example — it demonstrates the mechanism by which GLAD achieves its quantitative improvements. The base model tries to do lookahead reasoning (it generates analysis about future states) but gets the simulation wrong. GLAD doesn't add a new capability; it makes an existing capability accurate by grounding it in environment-verified examples during training. The model learns that when it says "if I move up, the 4-tiles will merge," this statement should correspond to what actually happens when that move is executed — and it has seen thousands of SFT examples where the relationship between analysis statements and environment outcomes was verified.

RL with MC-Critic Trained on GLAD-Initialized Models

Headline result (Figure 4, Table 2). When RL is applied on top of GLAD SFT checkpoints, MC-Critic consistently improves final performance over standard PPO and GRPO variants on both 2048 and Sokoban.

Figure 4 shows training curves for four configurations: (a) PPO on 2048, (b) PPO on Sokoban, (c) GRPO on 2048, and (d) GRPO on Sokoban. In all four subfigures, the MC-Critic-augmented variant (labeled "Step-PPO + MC-Critic" or "Step-GRPO + MC-Critic") achieves higher final scores than the corresponding baseline (Step-PPO or Step-GRPO). The gap is particularly pronounced in Figure 4(a) (PPO on 2048) and Figure 4(c) (GRPO on 2048), where the MC-Critic variants show a clear separation from baselines that persists and widens over training.

What the PPO results on 2048 show (Figure 4a). Step-PPO alone shows some improvement over the GLAD baseline, indicating that RL can further refine the policy even after SFT. However, MC-PPO ("Step-PPO + MC-Critic") achieves a substantially higher final score. The training curves suggest that MC-PPO not only reaches a higher asymptote but also learns faster in the early stages — the gap appears early and is maintained. This is consistent with the paper's argument that MC-Critic provides lower-variance advantage estimates, which in turn produce more reliable policy gradient updates that don't cancel each other out due to noise.

What the GRPO results on 2048 show (Figure 4c). Step-GRPO as a baseline is already an improvement over Traj-GRPO (not shown in Figure 4 but discussed in the text) because it provides finer-grained credit assignment. Adding MC-Critic on top ("Step-GRPO + MC-Critic") yields further gains. This is notable because GRPO was originally designed for single-turn math reasoning tasks, where immediate reward (correct/incorrect) is sufficient and there's no need for long-term value estimation. The paper's Step-GRPO adaptation extends GRPO to multi-turn settings, and MC-Critic addresses the key limitation of that adaptation: the myopia of step-level rewards. By replacing immediate rewards with MC-Critic Q-values, MC-GRPO incentivizes actions that may yield lower immediate reward but better long-term outcomes — exactly the kind of tradeoff that matters in 2048.

Sokoban results (Figures 4b, 4d). The improvements are present but less dramatic than on 2048. This makes sense given the environment characteristics: Sokoban has sparse rewards and most training levels are solvable within relatively few steps (the paper uses simplified levels for RL training). In shorter-horizon settings, the benefit of looking ahead via MC-Critic's value estimates is less pronounced because the gap between immediate and long-term consequences is smaller. The fact that MC-Critic still helps suggests the value estimation problem is non-trivial even at moderate horizons.

Generalization to variants (Table 2). Table 2 reports performance of RL-trained models (with and without MC-Critic) on the 2048 and Sokoban environment variants. The key finding: MC-Critic's improvements transfer to unseen variants. On 2048, MC-PPO and MC-GRPO outperform their non-MC counterparts on the 3 × 3 and 3072 variants. On Sokoban, both MC-PPO and MC-GRPO maintain advantages over baselines on the Base, Action, and Symbol variants. This demonstrates that the stability benefits of MC-Critic don't cause overfitting to training-level reward patterns — the improved policy generalizes.

Why this generalization result matters. A common failure mode of RL fine-tuning is that the policy finds shortcut solutions that maximize reward on training levels but fail under distribution shift. (For example, in Sokoban, a policy might learn level-specific box-pushing sequences rather than general planning principles.) The fact that MC-Critic-augmented training produces policies that generalize better than baseline RL — not just achieve higher training scores — suggests that the low-variance value estimates help the model learn more robust decision rules rather than exploiting environment-specific reward noise.

RL with MC-Critic Trained from Scratch

Headline result (Figure 5, Table 3). Even without GLAD supervision, MC-Critic enables stable long-horizon RL training from the base Qwen3-4B-Instruct model, with MC-PPO achieving the highest scores across both environments.

Figure 5 shows training curves for the from-scratch setting. On 2048 (Figures 5a and 5c), MC-PPO and MC-GRPO consistently outperform their non-MC counterparts. Notably, Traj-GRPO degrades in performance on 2048 (Figure 5c) — its performance initially rises but then declines as training continues. The paper attributes this to the accumulation of reward variance over long trajectories: when entire trajectories are assigned a single advantage based on total return, lucky trajectories (where good random tile spawns produced high scores despite mediocre play) get reinforced alongside genuinely skillful trajectories. Over training, this noise accumulates and destabilizes the policy. MC-GRPO avoids this by operating at the step level and using MC-Critic values rather than raw returns, which the paper argues provides a more reliable signal.

On Sokoban from scratch (Figures 5b and 5d), the results are more nuanced. MC-GRPO performs comparably to Traj-GRPO rather than significantly better. The paper's explanation (Section 4.2.2, under "Train From Scratch"): the simplified Sokoban training levels are solvable within 20 steps, making full trajectories short enough that the variance in Traj-GRPO's advantage estimates is not catastrophic. In this regime, the coarse credit assignment of Traj-GRPO is adequate, and the additional complexity of MC-Critic doesn't provide a decisive advantage. However, MC-PPO still achieves the highest scores, suggesting the learned critic component (even when blended with MC-Critic) provides additional benefit over pure return-based estimation in deterministic environments.

Generalization from scratch (Table 3). When evaluated on environment variants, models trained with MC-Critic (both MC-PPO and MC-GRPO) outperform their non-MC counterparts. This replicates the pattern from the GLAD-initialized setting: the stability benefits of low-variance value estimates produce policies that generalize better, not just policies that overfit to training-level reward patterns.

The significance of the from-scratch results. These experiments serve as an ablation of GLAD: they test whether MC-Critic alone, without the reasoning priors from SFT, can enable successful RL training in these environments. The answer is a qualified yes — MC-PPO achieves the best performance in both environments — but the absolute scores are not reported relative to the GLAD-initialized setting, making it difficult to quantify how much GLAD contributes beyond MC-Critic. The paper's structure implies that GLAD + MC-Critic is the full ProAct pipeline, with the from-scratch experiments serving to validate MC-Critic's standalone effectiveness rather than to claim MC-Critic alone is sufficient.

Hyperparameter Analysis of MC-Critic

Headline result (Figure 6). The optimal settings for MC-Critic's key hyperparameters — number of trajectories $M$ and rollout horizon $T$ — are environment-dependent, revealing a systematic relationship with reward density and horizon length.

Impact of $M$ (number of Monte Carlo trajectories). On 2048 (Figure 6a, varying $M$ ), performance monotonically improves as $M$ increases from 0 to 1,000, with $M=0$ (degenerating to Step-GRPO, where $V_{\pi_{\text{random}}}^{\text{MC}} = 0$ and Q-values reduce to immediate rewards) performing worst and $M=1,000$ performing best. The paper explains this through variance reduction: 2048 has dense rewards (merges produce rewards frequently), so averaging over more trajectories reliably reduces estimator variance. At $M=10$ , performance is actually worse than Step-GRPO, which the paper attributes to the variance from only 10 random trajectories being higher than the variance of immediate step-level rewards — the MC-Critic estimate is noisier than just using the immediate reward.

On Sokoban (Figure 6b), the pattern inverts: $M=10$ outperforms $M=100$ and $M=1,000$ . The paper's explanation is insightful: in sparse-reward Sokoban, random policy rollouts rarely succeed (most trajectories yield zero cumulative reward). When $M$ is large, the few successful trajectories are averaged with many zeros, producing Q-values that are very close across different actions — diluting the advantage signal. A smaller $M$ , while having higher variance, preserves the distinction between actions that occasionally lead to success and those that never do. This is a concrete demonstration of the bias-variance tradeoff the paper claims MC-Critic navigates.

Impact of $T$ (rollout horizon). On 2048 (Figure 6a), performance improves as $T$ increases from 0 to 100, then slightly declines at $T=1,000$ . The improvement up to $T=100$ reflects the value of longer-horizon lookahead in a game where strategic decisions play out over many turns. The slight decline at $T=1,000$ is attributed to increased single-trajectory variance: extremely long random rollouts have highly variable outcomes, and this noise slightly degrades the value estimate despite providing a longer view.

On Sokoban (Figure 6b), $T=5$ outperforms $T=10$ and $T=20$ . Since most simplified Sokoban levels can be solved within 5 steps, longer rollouts add noise without adding useful information — once the level is solved or deadlocked, additional steps just accumulate zero or penalty rewards that don't discriminate between good and bad initial actions.

The practical recipe the paper extracts (end of Section 4.2.2). For dense-reward environments, set $M$ as large as interaction efficiency allows. For sparse-reward environments, keep $M$ moderate (not excessive) to preserve action-value distinctions. Set $T$ to approximately the average number of steps needed for a successful trajectory — longer horizons increase variance without benefit.

Ablation Studies and Robustness Checks

Step-GRPO vs. Traj-GRPO as baseline RL variants: The paper demonstrates (Figures 4c, 5c, and accompanying text) that Traj-GRPO degrades on 2048 (long-horizon, stochastic) where trajectory-level credit assignment suffers from high variance, while Step-GRPO is more stable but myopic. This ablation establishes the need for per-step credit assignment with long-term value awareness — the gap that MC-GRPO fills.
MC-Critic with PPO vs. GRPO: The paper tests MC-Critic with both PPO and GRPO (Figures 4–5), showing improvements in both cases. This demonstrates that MC-Critic is algorithm-agnostic and its benefit is not tied to a specific policy gradient formulation.
GLAD initialization vs. from-scratch training: By evaluating MC-Critic both on top of GLAD checkpoints (Figure 4) and from the base Qwen3-4B-Instruct (Figure 5), the paper disentangles the contributions of the two stages. MC-Critic provides gains in both settings, but the absolute performance with GLAD initialization is higher (the paper does not provide a direct numerical comparison of the two initialization regimes on the same evaluation metric, which is a missing piece).
Random-policy rollouts vs. learned policy rollouts (implicit ablation): MC-Critic's core design choice — using $\pi_{\text{random}}$ rather than $\pi_\theta$ for Monte Carlo estimation — is justified through the speed argument (Section 3.3.1) and validated by the overall results. However, the paper does not include an explicit ablation comparing MC-Critic with random-policy rollouts against an otherwise identical setup using LLM-policy rollouts (even with very small $M$ ). This ablation would directly test the bias-variance tradeoff claim that underpins the method.
Shared vs. distinct initial states in GRPO: The paper's Step-GRPO variant samples states from a pool and generates independent single-step samples from each state, rather than generating full trajectories from shared initial states as in Traj-GRPO. This represents an ablation of credit assignment granularity, though the paper presents it as algorithm development rather than a formal ablation.
Absolute vs. relative advantage in MC-GRPO: The switching mechanism between relative advantage (when sampled actions differ) and absolute advantage (when all sampled actions are identical, using the full action space mean as baseline) is described in Section 3.3.2. The paper does not report an ablation comparing this switching mechanism against alternatives (e.g., always using absolute advantage, or discarding homogeneous groups as in DAPO). The effectiveness of this design choice is thus asserted but not empirically isolated.
Negative result: Traj-GRPO degradation on 2048 (Figure 5c): This is an important finding, not an ablation in the traditional sense. Traj-GRPO's performance degrades over training on the long-horizon 2048 environment, which the paper attributes to variance accumulation in trajectory-level returns. This negative result provides empirical motivation for the step-level credit assignment and MC-Critic's low-variance value estimates.
Diversity of SFT data (implicit in data collection): GLAD's data collection uses 2,048 trajectories with different random seeds for 2048 and procedurally generated levels for Sokoban. The paper asserts this promotes diversity but does not ablate data quantity or diversity (e.g., comparing against a dataset of equal size but without the compression step, or with fewer MCTS trajectories).

Critical Assessment

Claim from the executive summary: "A 4B parameter model trained with ProAct outperforms all open-source baselines and rivals state-of-the-art closed-source models."

This claim is supported by Table 1 for the SFT stage alone. However, the comparison is between a fine-tuned model and zero-shot models, which is not a test of architectural superiority but of domain adaptation. The 4B model has seen 25K (2048) or 8K (Sokoban) environment-specific training examples with ground-truth MCTS guidance. The closed-source models have seen none. This is more accurately characterized as "a small model with domain-specific training can match or exceed large general-purpose models on that domain" — which is a valuable finding but narrower than "outperforms" implies.

A stronger claim would require comparing against fine-tuned versions of the larger models, or at minimum, against few-shot prompted versions. The paper provides neither. The closed-source models are evaluated in a pure zero-shot setting with a fixed temperature of 0.6, which is a reasonable baseline but not a strong one for establishing that ProAct's methodology is superior to alternative training approaches for those models.

Claim: "ProAct... [generalizes] robustly to unseen environment variants."

This claim is supported for the specific variants tested: 3 × 3 grid and 3072 for 2048; unseen levels, action space changes, and symbolic representation changes for Sokoban. The generalization results appear in Table 1 (SFT only), Table 2 (RL from GLAD checkpoint), and Table 3 (RL from scratch). The pattern is consistent: ProAct-trained models maintain performance advantages over baselines on unseen variants.

However, the scope of "unseen environment variants" is limited. For 2048, the variants change grid size (4 × 4 → 3 × 3) and tile parameters (minimum value 2 → 3), but the core game mechanics (tile sliding, merging rules) remain identical. For Sokoban, the variants change level layouts, action representations, and map symbols, but the underlying dynamics (push mechanics, wall-blocking rules) are unchanged. These test interpolation within the same task family, not extrapolation to fundamentally different environment dynamics.

A stronger generalization claim would require testing on environments with different transition dynamics — for example, a game with different merge rules, or a puzzle with different box-pushing physics. The paper's variants all preserve the core MDP structure; they test robustness to input distribution shift, not robustness to dynamics shift.

Claim: "Environment-grounded search distillation plus Monte Carlo value estimation can substitute for model scale in interactive planning tasks."

The paper demonstrates this for the specific case of 4B vs. large closed-source models on these two environments. Generalizing this to "interactive planning tasks" writ large requires evidence the paper does not provide. The environments share characteristics — they are both grid-based puzzles with discrete action spaces, clear reward structures, and well-defined termination conditions — that may not transfer to open-ended agentic tasks (GUI interaction, dialogue, multi-step tool use). The paper acknowledges this limitation implicitly by restricting evaluation to 2048 and Sokoban, but the claim language in the executive summary ("interactive planning tasks") is broader than the evidence.

Missing experiment: GLAD without compression. The paper argues that cognitive compression improves over raw MCTS traces by removing search artifacts and structuring reasoning as causal chains. However, no ablation compares GLAD with compression against GLAD using uncompressed (but still environment-grounded) MCTS trajectories as SFT data. Without this comparison, it is unclear whether the gains come from the environment grounding, the compression, or their interaction. This is a significant omission given that compression is presented as one of the paper's key innovations.

Missing experiment: MC-Critic with learned-policy rollouts. The paper's central justification for using a random policy — that learned-policy rollouts are too slow — is quantitatively asserted (3-6 seconds per step vs. thousands of rollouts in under 3 seconds) but never empirically validated as a design constraint through an ablation. Even a small-scale comparison (MC-Critic with $M=5$ LLM-policy rollouts vs. $M=1,000$ random-policy rollouts, matched for wall-clock time) would strengthen the bias-variance tradeoff argument considerably.

Missing experiment: MC-Critic blending weight ( $\omega$ ) sensitivity for MC-PPO. The paper introduces the blending parameter $\omega$ in Equation 18 but does not report its value or sensitivity in the experimental section. The results for MC-PPO are presented without analysis of how performance varies with $\omega$ , which is a critical hyperparameter for the method. If MC-PPO is robust to $\omega$ , that strengthens the method; if it is sensitive, practitioners need guidance on tuning.

Test set size and statistical reliability. The paper does not report the number of evaluation episodes per setting, confidence intervals, or any measure of statistical significance. For 2048, which is stochastic (random tile spawns), different runs with the same policy can produce substantially different scores. Without variance information, it's impossible to assess whether the reported performance differences are statistically reliable or attributable to random seed variation. This is a standard reporting practice in RL benchmarks that the paper omits.

Single model family. All experiments use Qwen3-4B-Instruct. While the paper states this ensures fair comparison, it leaves open whether ProAct's benefits transfer to other model families (e.g., Llama, Gemma, DeepSeek) or to different base model scales. A 4B model is relatively small; whether the approach works for 1B models (which would further reduce inference cost) or 7B+ models (which may have stronger zero-shot reasoning and benefit differently from GLAD) is untested.

What is convincingly demonstrated: GLAD teaches causally grounded reasoning. The case study (Figure 3) and Table 1 results collectively provide strong evidence that GLAD's environment-grounded SFT produces reasoning that is qualitatively more accurate than the base model's ungrounded reasoning, and that this translates to substantial performance improvements both in-distribution and on close variants. This is the paper's cleanest and most robust contribution.

What is convincingly demonstrated: MC-Critic stabilizes RL training. Figures 4 and 5 show consistent improvements from MC-Critic across environments, algorithms, and initialization regimes. The hyperparameter analysis (Figure 6) provides a coherent explanation of when and why MC-Critic helps. The method's simplicity (parameter-free, plug-and-play) and the clarity of its bias-variance framing make this a credible contribution to the multi-turn agentic RL toolkit, even if the absolute magnitude of improvement varies by setting.

What remains uncertain: the relative importance of GLAD vs. MC-Critic. The paper presents ProAct as a two-stage pipeline where both stages matter, but it never reports a direct numerical comparison of (a) GLAD only, (b) MC-Critic only (from scratch), and (c) GLAD + MC-Critic on the same evaluation metric. The from-scratch results (Figure 5) and GLAD-initialized results (Figure 4) are presented in separate figures with different training configurations (e.g., different Sokoban levels, different trajectory lengths), making a head-to-head comparison impossible. A single table reporting the final performance of all three configurations on the standard evaluation benchmarks would clarify whether the two stages are truly complementary or whether one dominates.

What remains uncertain: scalability to harder environments. Both 2048 and Sokoban, while non-trivial, are substantially simpler than many real-world agentic tasks. Sokoban has a branching factor limited by the small grid size and action space; 2048 has dense rewards that provide constant feedback. The paper's own hyperparameter analysis shows MC-Critic's behavior changes dramatically between dense-reward (2048) and sparse-reward (Sokoban) settings. Extrapolating to harder environments — where rewards are even sparser, horizons even longer, or state spaces even larger — is not supported by the current evidence. The MC-Critic recipe (large $M$ for dense rewards, small $M$ for sparse) may not extend to environments where random policies are completely uninformative (e.g., Montezuma's Revenge-style exploration challenges where random actions almost never yield any reward).

6. Limitations and Trade-offs

The Zero-Shot vs. Fine-Tuned Comparison Overstates ProAct's Advantage Over Large Models

The assumption or constraint. The paper's headline comparison in Table 1 pits a 4B model trained with 25K (2048) or 8K (Sokoban) environment-specific SFT examples — generated through expensive MCTS probing with ground-truth environment access — against closed-source models (GPT-5, Claude Sonnet 4.5, Seed1.6, Seed1.8) evaluated entirely zero-shot with temperature 0.6. The closed-source models receive no few-shot examples, no environment-specific prompting, and no domain adaptation of any kind. The paper acknowledges this implicitly in Section 4.1.2 by specifying that "all baselines and ablations share the same model architecture and parameter count to ensure a fair comparison" — but the closed-source baselines violate this fairness criterion in the opposite direction: they are architecturally incomparable models evaluated under substantially weaker conditions (no domain training) than the ProAct model.

The consequence. The claim that a 4B ProAct model "outperforms all open-source baselines and rivals state-of-the-art closed-source models" (abstract, Section 5) is technically true under the evaluation protocol but is better characterized as "a small model with domain-specific SFT can match or exceed large general-purpose models evaluated zero-shot on that domain." This is a valuable demonstration of domain adaptation effectiveness, not evidence that ProAct's methodology produces intrinsically superior planning capabilities to the large models' architectures or training. A practitioner deciding whether to deploy ProAct versus using a large closed-source model with equivalent adaptation budget (e.g., few-shot prompting, or fine-tuning the large model on the same GLAD data) gets no guidance from this comparison. The paper provides no evidence about what a GPT-5 or Claude Sonnet 4.5 fine-tuned on the same GLAD dataset would achieve, which is the relevant counterfactual for a resource allocation decision.

What evidence exists in the paper. Table 1 reports ProAct scores above all listed baselines on both 2048 and Sokoban variants. The evaluation protocol (Section 4.1.4) specifies that "all models are evaluated under the same environment interface, with the temperature fixed to 0.6" but does not provide few-shot examples or any adaptation mechanism for the baseline models. The paper does not report the zero-shot performance of the base Qwen3-4B-Instruct model in Table 1 (it appears only in the case study, Figure 3a), making it impossible to parse how much of ProAct's advantage comes from the base model's inherent capability versus the GLAD training.

Mitigation status. The paper does not address this limitation. It does not report few-shot baseline results, does not fine-tune any comparison model on the GLAD dataset, and does not qualify the "outperforms" language in the abstract or conclusions with the zero-shot caveat. The more meaningful comparison framework — given a fixed adaptation budget (SFT data, RL compute), which model family and training methodology produces the best agent? — is not explored.

The Difficulty Estimation / MCTS Data Generation Cost Is Unaccounted for in the Training Budget

The assumption or constraint. GLAD's data construction process (Section 3.2.1, Algorithm 1) requires running Monte-Carlo Tree Search in the real environment at every decision step of every trajectory used for SFT — probing the environment to generate candidate futures, then feeding those trajectories into an LLM for analysis, and potentially backtracking when the model identifies poor branches. For 2048, this process generates 25K training samples from 2,048 trajectories. The paper does not report the computational cost of this MCTS probing phase: how many environment steps are executed per data sample, how many LLM calls are made during the Probing-Decision-Reflection loop (since the LLM must read and analyze the MCTS trajectories before selecting actions), or what total FLOPs or wall-clock time the data generation consumes.

The consequence. A practitioner seeking to apply ProAct to a new environment would need to budget for this data generation cost, which could be substantial. MCTS probing at every decision step across thousands of trajectories, with LLM calls for trajectory analysis and compression, may dominate the total training cost — potentially exceeding the cost of the subsequent SFT and RL stages combined. The paper's presentation of GLAD as a "supervised fine-tuning" stage of modest size (25K samples) obscures the fact that generating those 25K samples required an expensive search process with environment oracle access. If the MCTS probing cost is comparable to or exceeds the cost of simply running a larger model with inference-time search (the very thing GLAD is designed to avoid), the practical advantage of the approach narrows considerably.

What evidence exists in the paper. The paper provides no quantification of the data generation cost. Section 3.2.1 describes the MCTS probing process — sampling $N$ trajectories per step, $T$ steps per trajectory, with $k$ rounds of deliberation — but never specifies concrete values for $N$ , $T$ , or $k$ in the data generation phase (these parameters appear only for MC-Critic in the RL stage, not for GLAD's MCTS). Section 4.1.3 notes that "2,048 trajectories are initialized with different random seeds" for 2048 and that "25K samples" are collected, but does not report the number of environment interactions or LLM calls involved in producing them. The compression step (Section 3.2.2) uses a "teacher model" to synthesize raw traces, adding another unquantified cost.

Mitigation status. The paper does not address this limitation. The data generation cost is simply not reported. Section 8 (conclusion) does not flag this as an area for future work. The paper frames GLAD as an efficient alternative to inference-time search ("without the computational overhead of inference-time search," abstract) without acknowledging that the training-time search cost to produce GLAD's supervision data may itself be substantial. A cost accounting that includes data generation would be necessary to evaluate whether GLAD is genuinely more efficient than alternatives like directly using a search-based agent at deployment time.

MC-Critic's Random-Policy Substitution Makes No Theoretical Guarantees and May Fail When Random Policies Are Completely Uninformative

The assumption or constraint. MC-Critic estimates $V_{\pi_{\text{random}}}(s_t)$ rather than $V_{\pi_\theta}(s_t)$ , using a random policy as a surrogate for the learned LLM policy to achieve fast Monte Carlo rollouts (Section 3.3.1). The paper explicitly states this is "theoretically suboptimal" but argues it is the right bias-variance tradeoff for LLM-based RL. This argument rests on an empirical claim: that $V_{\pi_{\text{random}}}(s_t)$ is correlated enough with $V_{\pi_\theta}(s_t)$ to provide a useful training signal, even if it is biased. The paper provides no theoretical bounds on this correlation, no analysis of when the random-policy value becomes misleading, and no characterization of the environments where this substitution is safe.

The consequence. In environments where random policies are completely uninformative — where a random agent's expected return provides zero information about which actions are good for a competent policy — MC-Critic's value estimates would be useless or actively harmful. For example, in a Sokoban variant with larger grids and more boxes, a random policy might never solve any level (success rate effectively zero), producing $V_{\pi_{\text{random}}}(s_t) \approx 0$ for all states. MC-Critic's Q-values would then reduce to approximately the immediate reward $r_t$ (since the $V_{\pi_{\text{random}}}(s_{t+1})$ term in Equation 10 would be near-zero), degenerating to Step-GRPO's myopic reward signal and defeating the purpose of long-horizon value estimation. More subtly, in environments where random actions tend to produce rewards that are negatively correlated with good-policy returns (e.g., games where random moves waste resources that a good policy would conserve for later), MC-Critic could actively steer the learned policy in the wrong direction.

What evidence exists in the paper. The hyperparameter analysis (Figure 6) partially reveals this limitation. On sparse-reward Sokoban, the paper finds that $M=10$ outperforms $M=100$ and $M=1,000$ — because large $M$ dilutes the rare successful rollouts, making Q-values indistinguishable across actions. The paper's own explanation: "the positive Monte-Carlo returns from successful trajectories are diluted when averaged over $M$ trajectories... obscuring the advantage of the optimal action" (Section 4.2.2, Analysis). This is a moderate-sparsity regime where random policies occasionally succeed. In a harder sparse-reward environment where random policies never succeed, the dilution problem would be total — all Q-values would collapse to immediate rewards regardless of $M$ . The paper does not test this regime.

Mitigation status. The paper acknowledges the environment-dependence of MC-Critic's hyperparameters and provides a recipe (Section 4.2.2, end of Analysis): "For environments with sparse rewards, $M$ should not be excessive, as a smaller $M$ may yield better results." But this recipe addresses only the variance side of the tradeoff, not the fundamental question of whether $V_{\pi_{\text{random}}}$ contains any signal at all. The paper does not propose methods for detecting when random-policy values become uninformative, does not suggest alternative surrogate policies (e.g., a weak learned policy, or domain-specific heuristics), and does not provide a diagnostic for practitioners to determine whether MC-Critic is suitable for their environment.

The Paper Does Not Disentangle the Contributions of GLAD's Environment Grounding from Its Cognitive Compression

The assumption or constraint. GLAD consists of two sequential steps: (1) environment-augmented lookahead data generation using MCTS probing, which produces raw interaction traces containing environment-verified futures, and (2) cognitive compression, which synthesizes these raw traces into structured natural-language reasoning chains following Observation → Analysis → Conclusion format (Section 3.2.2). The paper presents these as a unified pipeline and evaluates them jointly — all GLAD results in Table 1 and Figures 3–4 use compressed reasoning chains. There is no ablation comparing GLAD-with-compression against an alternative that uses the same environment-grounded MCTS data but without the compression step (i.e., training on raw search traces directly, or on a simpler format that retains environment verification but not the specific causal structuring).

The consequence. It is impossible to determine whether GLAD's gains come from (a) the environment grounding itself — showing the model real futures rather than letting it hallucinate — or (b) the specific compressed format — the causal structure, counterfactual reasoning, and format simplification. If the gains are primarily from environment grounding, then simpler distillation approaches (training on raw MCTS traces with minimal formatting) might achieve similar results with less engineering complexity and no dependency on a teacher model for compression. If the gains are primarily from the compression, then environment grounding might be less critical than the paper argues, and other sources of high-quality structured reasoning data (e.g., human-written analyses) might suffice. The paper's central claim — that "environment-grounded search distillation" is the key mechanism — cannot be evaluated without this ablation.

What evidence exists in the paper. The paper provides qualitative evidence that compression matters: the case study in Figure 3 shows the compressed-reasoning model producing cleaner, more accurate analysis than the base model. But this comparison is between compressed GLAD and the base instruction-tuned model with no environment grounding at all — not between compressed GLAD and a GLAD variant using uncompressed but still environment-grounded data. The paper argues extensively for compression's benefits (Section 3.2.2: format simplification prevents distribution shift, explicit causal chains force logical structure, counterfactual reasoning teaches trade-off analysis) but tests none of these claims in isolation.

Mitigation status. The paper does not acknowledge this as a confound, does not propose an ablation to separate grounding from compression effects, and does not discuss the possibility that the compression step could be simplified or eliminated. The compression step is presented as an integral part of the method rather than one design choice among several. A minimal ablation — training on (state, raw MCTS trace, action) triples without compression, evaluated on the same benchmarks — would substantially strengthen or qualify the paper's claims about GLAD's mechanism.

Generalization Is Tested Only Within Highly Similar Task Families; Extrapolation to Fundamentally Different Environment Dynamics Is Untested

The assumption or constraint. All generalization experiments (Tables 1–3) test ProAct on variants of the training environments: for 2048, a different grid size (3 × 3) and a different minimum tile value (3072); for Sokoban, unseen levels, modified action representations, and altered map symbols. In every case, the core MDP dynamics remain identical: tile-sliding and merging rules for 2048 variants, box-pushing and wall-blocking physics for Sokoban variants. These test robustness to input distribution shift (different board sizes, different level layouts, different surface representations) but not to dynamics shift — what if the merge rules changed, or the push mechanics behaved differently? The paper's claims about generalization (abstract: "robust generalization to unseen environments"; Section 4.2.1: "strong generalization capabilities") do not qualify that the "unseen environments" preserve the same transition dynamics as training.

The consequence. A practitioner considering ProAct for a different interactive environment — say, a GUI navigation task, a text-based adventure game, or a multi-agent negotiation setting — has no evidence about whether the approach transfers. The environments tested share specific characteristics that may be important for ProAct's success: discrete action spaces (4–8 actions), grid-based spatial structure, clear reward semantics (merge values, box placements), and well-defined terminal conditions. Environments lacking these properties might not benefit from GLAD's MCTS probing (which requires a fast environment simulator) or MC-Critic's random-policy rollouts (which require random actions to occasionally produce informative outcomes). The paper does not test whether GLAD's reasoning patterns — which are deeply tied to 2048's tile-merging logic and Sokoban's push mechanics — represent a generalizable lookahead skill or an environment-specific heuristic.

What evidence exists in the paper. Tables 1–3 report performance on the listed variants, all of which preserve the core game dynamics. The strongest test of dynamics generalization would be a transfer experiment where a model trained on 2048 is evaluated on a different tile-sliding game with modified rules, or a model trained on Sokoban is evaluated on a block-pushing puzzle with different physics. No such experiment exists. The paper restricts its claims to the tested variants but uses language like "robust generalization to unseen environments" that could be read as broader than the evidence supports.

Mitigation status. The paper does not discuss the limits of its generalization tests. The variants are presented as evidence of robustness without acknowledging that they test only surface-level distribution shift. No experiments probe whether the learned lookahead reasoning transfers across tasks, and no discussion addresses what environment properties are necessary for ProAct to be effective. The MC-Critic recipe (Section 4.2.2, end of Analysis) provides some environment-conditional guidance (dense vs. sparse rewards) but does not address the broader transfer question.

7. Implications and Future Directions

How This Work Changes the Landscape

ProAct shifts the conversation around LLM agents from "can we make models reason better at inference time?" to "can we teach models to internalize accurate reasoning during training so they don't need to search at deployment?" This is a reframing, not a paradigm shift — the individual components (search distillation, MC value estimation) have precedents in the literature — but the synthesis and the explicit diagnostic framing make it a significant methodological contribution that changes how researchers should think about the training-inference compute tradeoff for interactive agents.

The paper's most landscape-changing contribution is the identification and naming of simulation drift as a specific, mechanistic failure mode distinct from generic hallucination. Prior work has extensively documented that LLMs produce incorrect statements about environment states and action consequences, but the field has largely treated these as instances of the same factual unreliability that affects all LLM outputs. ProAct's framing — that simulation drift is a compounding dynamical error where $P_{model}(s'|s,a) \neq \mathcal{P}(s'|s,a)$ and the gap grows exponentially with lookahead depth — redirects the solution space from "collect more training data" or "add retrieval" toward grounding the model's internal simulation in environment-verified examples during training. This framing makes simulation drift a tractable target for intervention rather than an intractable side effect of LLM architecture, and it is likely to influence how subsequent work characterizes and addresses agent reasoning failures.

The paper also provides a clear empirical resolution to a tension that has implicitly divided the LLM-agent community: whether to invest inference-time compute in explicit search (Tree of Thoughts, RAP-style methods) or to distill search behaviors into efficient feed-forward policies (STaR, Distilling Step-by-Step). ProAct's results suggest a synthesis: use search during training to generate grounded supervision, then internalize that supervision via SFT, then refine with stable RL. This three-phase approach (search → distill → optimize) is more sophisticated than either extreme and provides a template for future work. The finding that GLAD alone — without any RL — already outperforms all open-source baselines (Table 1) establishes that high-quality distillation from environment-grounded search is sufficient to produce strong agents even without reward optimization, which may simplify the training pipeline for domains where RL exploration is prohibitively expensive.

MC-Critic's success — across two different RL algorithms (PPO and GRPO), two different environments (stochastic and deterministic), and two initialization regimes (from GLAD and from scratch) — makes a concrete methodological contribution to multi-turn agentic RL. The field has been converging on parametric critic approaches (ArCHer, SWEET-RL, Turn-PPO) under the implicit assumption that learned value functions are necessary for credit assignment over long horizons. MC-Critic demonstrates that this assumption does not always hold: a low-variance biased estimator ( $V_{\pi_{\text{random}}}$ ) can outperform a high-variance unbiased estimator ( $V_{\pi_\theta}$ trained as a learned critic) when sample efficiency is the binding constraint. This is a specific instantiation of the bias-variance tradeoff that the paper makes explicit and empirically validates, and it opens the door for other cheap, environment-based value estimation approaches that don't require training auxiliary neural networks.

The paper also reshapes the scaling narrative around LLM agents. The finding that a 4B model with ProAct training outperforms zero-shot evaluations of much larger closed-source models (GPT-5, Claude Sonnet 4.5) on these specific environments — while not a fair architectural comparison — demonstrates concretely that domain-specific training with environment grounding can substitute for model scale in interactive planning tasks where the environment provides a fast simulator. This has a specific, actionable implication for deployment architects: if you have a fast, queryable environment (a game engine, a physics simulator, a deterministic code execution environment), you may be better off investing compute in generating grounded training data for a small model than in paying per-token inference costs for a large general-purpose model. The paper does not claim this generalizes to environments without fast simulators — and it likely doesn't — but it establishes a clear boundary: when environment interaction is cheap relative to LLM inference, environment-grounded training is the dominant strategy.

Follow-Up Research This Work Enables

Ablating environment grounding vs. cognitive compression in GLAD. The paper presents GLAD as a unified pipeline combining MCTS-based environment probing with a compression step that restructures reasoning into causal chains. However, as noted in Section 6, these two components are never evaluated independently. A critical follow-up would train three variants on the same environments: (1) SFT on raw, uncompressed MCTS traces (environment grounding only), (2) SFT on compressed reasoning chains generated from the same MCTS data but without the environment probing step (i.e., having the teacher model generate structured reasoning from states alone, without access to real trajectories), and (3) full GLAD with both components. Comparing these on the 2048 and Sokoban benchmarks would isolate whether environment grounding, compression, or their interaction drives GLAD's gains. If environment grounding alone accounts for most of the improvement, the compression step could be simplified or eliminated, reducing ProAct's training complexity. If compression alone accounts for most of the improvement, then the expensive MCTS probing might be replaceable with cheaper data sources. Either outcome would refine our understanding of GLAD's mechanism and guide practitioners on which components are worth the implementation cost.

Testing MC-Critic in environments where random policies are uninformative. The paper demonstrates MC-Critic in two environments where random policies produce at least occasionally informative outcomes: in 2048, random actions frequently produce merges and accumulate score; in Sokoban (simplified levels), random actions rarely but occasionally push boxes onto targets. A stress test would evaluate MC-Critic on environments where random policies are effectively information-free — for example, Sokoban levels requiring 50+ precise, coordinated moves to reach the first box placement, or text-based adventure games where random action sequences have near-zero probability of achieving any reward. If MC-Critic degrades to Step-GRPO performance (as the paper's own analysis predicts when $V_{\pi_{\text{random}}}$ becomes uniformly zero), this establishes a clear applicability boundary. If a modified version — using a heuristic policy (e.g., a shortest-path planner for Sokoban) rather than a purely random policy — recovers the benefit, this would extend MC-Critic's applicability to harder sparse-reward settings. The experiment would measure both final performance and training stability (variance of policy gradient updates) as a function of random-policy success rate, providing an empirical calibration of the bias-variance tradeoff the paper theorizes.

Combining GLAD's reasoning supervision with the RL objective in a single stage. The paper's two-stage pipeline — SFT on environment-grounded reasoning first, RL refinement second — is a design choice, not a necessity. An alternative would be to interleave the two: use the environment-grounded MCTS probing during RL to generate on-policy reasoning examples, then optimize the policy with a joint objective that includes both a supervised reasoning loss (matching the model's generated $z_t$ to environment-verified analyses) and the RL advantage-weighted loss. This would allow the reasoning component to adapt to the evolving policy rather than being frozen after SFT. The key question is whether joint optimization would improve final performance or cause the reasoning to degrade as the policy shifts (a common failure mode in actor-critic methods where the critic is trained on off-policy data). A comparison of two-stage ProAct against this joint-optimization variant on 2048 and Sokoban would reveal whether the reasoning priors established in Stage 1 are robust enough to survive RL without explicit ongoing supervision, or whether they drift and need reinforcement.

Transfer learning of GLAD-taught lookahead reasoning across environments with different dynamics. The paper's generalization experiments test variants with identical core mechanics but different surface representations. A stronger test of whether GLAD teaches generalizable lookahead skills rather than environment-specific heuristics would evaluate a model trained via GLAD on 2048 on a different stochastic tile-based game with modified merge rules (e.g., Threes, or a variant where tile values decrease rather than increase on merge), or a model trained on Sokoban on a different block-pushing puzzle (e.g., a variant where boxes can be pulled as well as pushed, or where multiple agents must coordinate). If the GLAD-trained model transfers even partially — showing reasoning that correctly anticipates consequences under the new dynamics rather than blindly applying old heuristics — this would suggest GLAD teaches meta-cognitive patterns (how to analyze trade-offs, how to structure causal reasoning) rather than game-specific tactics. If transfer fails entirely, this would establish that GLAD's reasoning is narrowly tailored to the training dynamics and that new environments require full retraining. The experiment would use the same Qwen3-4B-Instruct base and report both zero-shot transfer performance and few-shot fine-tuning efficiency (how many new-environment examples are needed to recover ProAct-level performance relative to training from scratch).

MC-Critic with learned surrogate policies between random and the LLM policy. The paper uses a purely random policy for MC-Critic's rollouts, sacrificing value accuracy for speed. An intermediate approach would use a lightweight learned policy — for example, a small MLP trained via behavioral cloning on successful GLAD trajectories, or a distilled version of the LLM policy using a much smaller language model (e.g., Qwen3-0.5B) — that is faster than the full 4B LLM but more intelligent than random. This could provide better value estimates ( $V_{\pi_{\text{surrogate}}}$ being closer to $V_{\pi_\theta}$ than $V_{\pi_{\text{random}}}$ is) while still being fast enough to generate hundreds of rollouts per decision. The experiment would sweep surrogate policy quality (random → heuristic → small distilled model → larger distilled model) and measure both the correlation between surrogate values and true LLM-policy values, and the downstream RL performance after training with each surrogate. This would map out the Pareto frontier of the speed-accuracy tradeoff in MC-Critic's value estimation and provide concrete guidance on when a learned surrogate is worth the engineering effort of training and maintaining an auxiliary policy.

Scalability to partially observable or non-Markovian environments. Both 2048 and Sokoban are fully observable: the text serialization of the board contains all information needed for optimal decision-making. Many real-world agentic tasks are partially observable (the agent cannot see the entire state) or non-Markovian (the optimal action depends on history beyond the current observation). MC-Critic relies on the ability to roll out trajectories from the current state $s_t$ , which requires a simulator that can be reset to $s_t$ — feasible for fully observable deterministic environments but not for partially observable ones where the true state is unknown. A follow-up could test ProAct (or an adapted version) on text-based environments with partial observability (e.g., TextWorld, or dialogue tasks where the user's internal state is hidden) by replacing the random-policy environment rollouts with rollouts from a learned world model trained on interaction history. The experiment would measure how value estimation quality degrades as observability decreases and whether a learned dynamics model can substitute for direct environment access in MC-Critic's framework.

Practical Applications and Downstream Use Cases

Training agents for interactive environments with fast simulators. The most direct application of ProAct is to any domain where a fast, queryable environment simulator exists and the task requires long-horizon sequential decision-making. Game AI is the obvious fit — puzzle games, strategy games, and board games where rule-based simulators can generate millions of trajectories per second. But the scope extends beyond games: code execution environments (where the "environment" is a Python interpreter that can be queried at near-native speed), robotics simulators (MuJoCo, Isaac Sim, where physics rollouts are fast relative to LLM inference), and combinatorial optimization problems (scheduling, routing, packing) where a fast solver can evaluate candidate action sequences. In all these cases, the ProAct pipeline — generate grounded reasoning via environment search, compress and distill into a small LLM, then refine with MC-Critic — provides a concrete recipe for producing a deployable agent that is both fast at inference (no search needed) and capable of sophisticated lookahead reasoning. The key prerequisite is the fast simulator; without it, neither GLAD's data generation nor MC-Critic's rollouts are practical. The paper's numbers give a sense of scale: 25K SFT samples and 1,000 Monte Carlo rollouts per advantage estimate are feasible when each environment interaction costs microseconds, but would be prohibitive if each interaction cost seconds.

Cost-efficient deployment of planning agents where inference latency matters. For applications where per-query latency is critical — interactive assistants, real-time game-playing agents, on-device deployment — ProAct offers a specific advantage over inference-time search methods (Tree of Thoughts, RAP). A GLAD-trained 4B model produces lookahead reasoning autoregressively in a single forward pass (3–6 seconds per decision, per the paper's timing), whereas a search-based agent at inference time might require hundreds of LLM calls exploring different branches, multiplying latency by orders of magnitude. The paper's finding that GLAD alone (without RL) matches or exceeds strong baseline performance (Table 1) means practitioners can deploy the SFT-only model if RL infrastructure is unavailable or if the risk of reward hacking during RL is unacceptable. The 4B parameter scale is significant: models of this size can run on consumer GPUs or even quantized on edge devices, making ProAct-trained agents deployable in settings where large closed-source models are inaccessible due to cost, latency, or privacy constraints.

Data generation for self-improvement loops in agent training. The paper's GLAD data construction process — using the environment to generate correct futures, then having a teacher model compress them into reasoning chains — is itself a reusable template for bootstrapping agent training data in new environments. An organization developing an agent for a custom interactive task (a new game, a proprietary simulator, an internal tool with a defined API) could follow the GLAD procedure: (1) build or use the existing environment simulator, (2) run MCTS probing to generate verified trajectories, (3) use a strong LLM (or the same LLM with the raw traces as context) to compress the trajectories into structured reasoning, and (4) SFT a smaller deployment model on the resulting dataset. This process could be iterated — use the SFT-trained model to collect higher-quality trajectories, regenerate the dataset, and retrain — creating an automated improvement loop that doesn't require human annotation. The paper's finding that 25K samples suffice for 2048 suggests the data requirements are modest enough to be practical, at least for environments of comparable complexity to the tested benchmarks.

When to Prefer This Method

The paper does not explicitly position ProAct against named alternative training paradigms with a systematic tradeoff analysis. The comparisons are against zero-shot baselines (Table 1) and against standard RL algorithms without MC-Critic (Figures 4–5), not against alternative training methodologies for agentic planning. The paper's contributions are the GLAD and MC-Critic techniques themselves, and the experimental design shows they improve over baselines, not that they outperform specific named alternatives in a competitive sense. A decision matrix of "prefer ProAct when X, prefer method Y when Z" would therefore be speculative rather than grounded in the paper's evidence. The paper does, however, implicitly establish boundary conditions through its hyperparameter analysis and environment selection: GLAD requires a fast environment simulator for MCTS probing data generation, and MC-Critic requires random-policy rollouts to produce at least occasionally informative outcomes. These conditions — rather than a comparison against named alternatives — define when the method is applicable.