A Subgoal-driven Framework for Improving Long-Horizon LLM Agents

ArXiv: 2603.19685

🎯 Pitch

Open web agents get stuck in loops on 42–49% of failures—not because they click wrong buttons, but because they lose track of what they’ve done. The authors fix this with dynamic milestoning (self-verified subgoal checklists at inference) and a dense RL reward that shoots Gemma-3-12B from 6.4% to 43.0% success, leaving GPT-4o at 13.9%.

---

1. Executive Summary

This paper proposes a subgoal-driven framework that improves long-horizon web agents through two complementary mechanisms: inference-time planning via dynamic milestoning (decomposing tasks into verifiable subgoal checklists that the agent self-assesses against its action history at each step) and an RL training procedure called MiRA (Milestoning your Reinforcement Learning Enhanced Agent) (using a learned potential critic that provides dense, milestone-based shaped rewards to mitigate credit assignment sparsity). Applied to the WebArena-Lite benchmark, Gemini-2.5-pro with dynamic milestoning improves by ~10% absolute success rate over the base model (23.0% → 32.1%), while MiRA boosts the open-source Gemma-3-12B from 6.4% to 43.0% — surpassing proprietary systems like GPT-4-Turbo (17.6%) and the prior open-source state-of-the-art WebRL (38.8%). The framework's gains derive primarily from reducing "stuck midway" failures — the dominant error mode identified via automated failure analysis — establishing that explicit milestone reasoning resolves the planning bottleneck that causes agents to loop or stall, but only shifts the remaining errors toward wrong-termination mistakes rather than eliminating failure entirely.

2. Context and Motivation

The Core Problem: Web Agents Cannot Sustain Reasoning Over Long Task Horizons

The fundamental issue this paper tackles is that LLM-based web agents — autonomous systems that navigate websites through multi-step sequences of clicks, types, and scrolls — systematically break down when tasks require more than a handful of interaction steps. This isn't a marginal weakness; the authors' automated failure analyzer reveals that across models ranging from proprietary giants (Gemini-2.5-pro) to fine-tuned open-weight models (Gemma-3-12B-SFT), approximately 42–49% of all failed trajectories end because the agent gets stuck midway — it enters repetitive action loops, revisits already-visited pages, or cycles through unproductive behaviors without recognizing its lack of progress (Figure 3, Section 4.5). These are not failures of low-level action execution (misclicking a button or misreading a form field); they are failures of planning and state awareness — the agent loses track of where it is in the task and what it should do next.

This problem is distinct from the more commonly studied short-horizon failures where an agent clicks the wrong link or types an incorrect query. In those cases, the agent typically makes a discrete error and then continues or terminates, producing a clear "wrong termination" or "failure to make reasonable attempt." Mid-task stagnation, by contrast, represents a subtler and arguably more frustrating failure mode: the agent keeps trying but has no mechanism to recognize that its efforts have become circular. The paper's failure analysis tool detects this by scanning for "identical last N actions or a short action-state sequence repeating M times" (Table 1), and then retroactively pinpoints the entry point into the loop — the moment the agent failed to notice it was stuck — rather than the loop itself. This diagnostic precision is important because it reveals that the agent doesn't just need better action selection; it needs a mechanism for introspective progress monitoring that operates continuously throughout execution.

Why This Matters: Real-World Deployment and the Long-Horizon Gap

The practical significance of long-horizon planning failure extends well beyond benchmark scores. Real-world web tasks — filing expense reports across multiple pages, researching and comparing products across tabs, managing project workflows in tools like GitLab or Jira — routinely span 10, 20, or 30+ interaction steps. These tasks are not just "harder" versions of short tasks; they impose qualitatively different demands on an agent. In a 5-step task, the agent can often succeed through reactive decision-making (see form → fill form, see submit button → click submit) because the full context needed for each decision is present in the immediate observation. In a 20-step task, the relevant context is distributed across many previous pages and actions: the agent must remember that it opened a specific tab three steps ago, that it already filtered results by a particular criterion, or that it needs to return to a previous page after completing a subtask.

The paper quantifies this challenge through its own empirical landscape. While state-of-the-art Gemini 2.5 Computer Use achieves 75% on aggregate UI control tasks, its performance drops to 36% on open-ended benchmarks like WebWorld (Google DeepMind, 2025, cited in Section 1). On WebArena-Lite itself, even after supervised fine-tuning (SFT) on human demonstrations, Gemma-3-12B-SFT still exhibits "get stuck midway" failures in over 30% of trajectories. The failure analysis in Section 4.5 breaks this down by model type:

• Base proprietary models (Gemini-2.5-pro): ~49% of failures are mid-task stagnation
• Base open-weight models (Gemma-3): ~42% of failures are mid-task stagnation
• SFT-fine-tuned open models (Gemma-3-SFT): mid-task stagnation remains the dominant category, though at a somewhat lower rate

The persistence of this failure mode across model scales and training paradigms — from 0-shot prompting to task-specific fine-tuning — signals that the problem is not insufficient model capability per se, but a missing architectural mechanism for maintaining and verifying progress state during extended interactions.

There's also a deployment-level motivation: if web agents are to move from research benchmarks to production systems (customer support automation, enterprise workflow agents, accessibility tools), they must be reliable in ways that go beyond aggregate success rates. An agent that succeeds 60% of the time but loops endlessly on 30% of the remaining tasks is far worse for user trust and system integration than one that fails quickly with an identifiable error. The paper's focus on reducing stagnation specifically — not just improving overall accuracy — is motivated by this reliability concern.

Where Prior Approaches Fall Short

The paper identifies several families of prior work that attempt to address long-horizon reasoning, but each exhibits specific limitations that the proposed framework is designed to overcome.

Prompting- and Imitation-Based Agents (SFT). The baseline approach in much of the LLM agent literature is to either prompt a frozen model with task descriptions and action formats, or to fine-tune the model on human/synthetic demonstrations via supervised learning. These methods achieve reasonable zero-shot or few-shot performance but inherit critical brittleness. Imitation learning, as the paper notes (Section 2.1), "depends on static data and fails to teach recovery from errors." When an SFT-trained agent encounters a state that deviates from its training distribution — a search result page that looks slightly different, a popup that blocks the expected element — it has no mechanism to recognize the deviation and adapt. It simply continues executing its learned action patterns, which in the worst case leads directly to the repetitive looping behavior the paper diagnoses. This limitation is baked into the training objective: SFT maximizes the likelihood of training actions given training states, but provides no signal about which deviations are recoverable and which are catastrophic.

Reinforcement Learning Methods with Sparse Rewards. Reinforcement learning (RL) addresses the core limitation of SFT by allowing agents to learn from outcomes (rewards) rather than only from demonstrations. In web navigation, the standard reward signal is binary: success (1) or failure (0) at task completion. This is the formulation the paper adopts in Section 3.1, where the reward is defined as $r_t(s_t, a_t, g) = \mathbb{1}[s_{t+1} \in \mathcal{S}_g]$.

The problem is that binary terminal rewards create a credit assignment desert: when an agent performs 20 actions and then succeeds or fails, which of those 20 actions were responsible? Standard RL algorithms struggle to propagate sparse signals backward through long action sequences, particularly when the relationship between early actions and final success is non-obvious (e.g., opening a specific tab on step 3 makes a filter available on step 15). The paper explicitly connects this to the failure analysis: "rewards in web navigation are often binary (success/failure) after many interactions, making credit assignment difficult; consequently, RL-based agents still show steep performance drops as task length increases" (Section 2.1).

Prior RL-based web agents like WebRL (Qi et al., 2024) attempt to mitigate this through self-evolving curricula — progressively training on harder tasks as the agent improves — but they still operate fundamentally with sparse terminal rewards. The paper's experiments show that WebRL reduces stagnation errors compared to SFT (Figure 13, from ~33% to ~25%), but a substantial fraction remains. The sparse reward signal, even with curriculum learning, simply doesn't provide enough intermediate guidance to teach the agent when it has lost its way.

Hierarchical and Subgoal-Conditioned Approaches. Several prior works have recognized the need for intermediate structure and have proposed hierarchical decomposition strategies. These fall into two broad categories, both of which the paper positions itself against.

Latent or learned subgoals. Methods like VSC-RL (Wu et al., 2025, cited in Section 2.2) and HIQL (Park et al., 2023, cited in Section 2.3) learn subgoal representations or high-level policies over latent states. VSC-RL uses variational subgoal-conditioned learning to boost sample efficiency in vision-language agents, while HIQL learns a high-level policy over latent states to decouple strategic planning from low-level control. The paper identifies a fundamental limitation of these approaches for web agents: latent subgoals lack semantic interpretability. You cannot ask a learned latent vector "has the agent navigated to the correct repository page yet?" because the subgoal exists only in the model's internal representation. This makes it "impossible to explicitly verify the agent's intermediate progress" (Section 2.3), which is precisely the capability the paper's failure analysis shows agents need most. The paper also criticizes world-model-based approaches (Duan et al., 2024), where agents simulate future outcomes to explore sparse-reward environments, as "computationally expensive and prone to compounding errors in dynamic, open-ended web environments" (Section 2.3).

Static plan decomposition. Other methods use LLMs to decompose tasks into plans upfront (Zhou et al., 2022; Wang et al., 2023), but these decompositions are typically static — generated once at the beginning of the task and then executed without revisiting. The paper notes that even architectures like CUGA (Shlomov et al., 2025, discussed in Section 2.2) which implement hierarchical planner-executor frameworks with task decomposition and reflective re-planning, observe that "tasks exceeding ~10 interaction steps often fail due to impaired sub-goal coherence, poor re-planning, and drift from the original aim." The implication is that static decomposition alone is insufficient; agents need dynamic, continuous progress verification that checks whether subgoals have actually been achieved given the current state, not just whether they were planned.

Process Reward Models (PRMs). A more recent line of work addresses sparse rewards by training Process Reward Models that provide step-by-step correctness signals. In web agent contexts, Web-Shepherd (Chae et al., 2025, cited in Section 2.2) uses checklist-style sub-goal verification to monitor trajectories and reduce error propagation, while AgentPRM (Xi et al., 2025) introduces a dual-scoring mechanism measuring both "promise" (likelihood of success) and "progress" (inter-step advancement). These approaches move in the right direction by densifying the reward signal, but the paper identifies a critical vulnerability: learned PRMs produce soft, noisy signals susceptible to over-optimization. When an RL agent learns to maximize a learned reward model's output rather than true task success — a phenomenon well-documented in the RLHF literature — it can discover adversarial behaviors that score highly under the PRM but don't correspond to genuine progress. The paper positions its own approach as achieving a "best-of-both-worlds balance by replacing soft rewards with hard objectives" — using explicit, semantically verifiable milestones rather than learned scalar estimates (Section 2.2).

Goal-Conditioned RL with HER. The paper also discusses Hindsight Experience Replay (HER; Andrychowicz et al., 2017), a standard technique in goal-conditioned RL where failed trajectories are relabeled as successful for alternative goals that were actually achieved. While HER provides denser learning signals, the paper notes that "standard HER assumes Markovian rewards, which often limits its applicability to the non-Markovian, long-horizon nature of web navigation" (Section 2.3). Web navigation is non-Markovian in a practical sense: whether clicking a link is "correct" depends on what was searched for and which pages were previously visited, information not contained in the current page state alone. This is why the paper explicitly encodes state as the combination of current observation and full action history: $s_t = [a_1, a_2, ..., a_{t-1}, o_t]$ (Section 3.1).

How This Paper Positions Itself

The paper positions its contributions along a clear thesis: long-horizon web agent failures are fundamentally planning failures, not execution failures, and they can be addressed by introducing explicit, verifiable milestones that serve dual roles — as runtime progress checkpoints during inference AND as dense training signals during RL fine-tuning.

This is more than a combination of existing ideas. The paper argues that prior work has treated inference-time planning and training-time reward shaping as separate problems addressed by separate mechanisms (static plans + sparse RL rewards; or PRMs for search; or hierarchical policies with latent subgoals). The key insight is that the same subgoal structure can serve both functions, creating a unified framework where:

1. At inference time, the agent's own reasoning capability (self-reflection against a subgoal checklist) provides runtime state awareness and error recovery — addressing the "where am I?" blindness that causes looping.
2. At training time, the same subgoals generate a learned potential function that densifies the reward landscape — addressing the credit assignment problem that prevents RL from learning which intermediate actions matter.

The paper explicitly frames this as following a simple principle: "If the final goal is difficult to reach directly, increasing the probability of reaching meaningful intermediate milestones helps" (Section 1). This is a statement about both optimization (dense rewards help gradient-based learning find better policies) and execution (state awareness helps the deployed policy avoid getting lost).

Crucially, the paper's motivation is empirically grounded rather than purely theoretical. The automated failure analysis in Section 4 doesn't just identify that agents get stuck — it provides the diagnostic machinery to show where and why they get stuck, using differential analysis against teacher demonstrations and golden paths from peer agents (Figure 2). This analysis reveals that failures cascade from specific decision points: the agent selects a wrong link on a search results page (Step 3 in Figure 2's example), and that single deviation cascades into an entire trajectory's failure. By identifying these "first points of significant divergence," the paper makes a compelling case that the solution must operate at the level of recognizing and correcting deviation in real-time, not just improving average action quality.

The paper also positions itself as pragmatic about limitations in ways that strengthen its claims. It acknowledges that subgoals are not perfect binary gates — the Exact Equivalence F1 score between "all subgoals completed" and "task success" is only 0.68 (Section 5.1) — but shows that subgoal completion serves as a continuous progress signal with AUROC of 0.84 and monotonic relationship to success probability. This measured claim — subgoals are good progress estimators, not necessary-and-sufficient conditions — distinguishes the approach from more brittle decomposition strategies that assume completing all subgoals guarantees success.

Finally, the paper explicitly connects its contributions to three concrete challenges (C.1–C.3 in Section 1) that any milestoning framework must solve: where subgoals come from, how to integrate them at inference time without prohibitive overhead, and how to embed them in RL training without distorting the final objective. The structure of the technical approach (Section 5) directly mirrors these challenges, with §5.1 addressing subgoal generation, §5.2 addressing inference-time integration, and §5.3 addressing RL training. This design transparency allows the reader to evaluate whether each challenge is adequately resolved rather than treating the framework as a monolithic black box.

3. Technical Approach

3.1 Reader Orientation

The paper presents MiRA (Milestoning your Reinforcement Learning Enhanced Agent), a training-plus-inference framework that teaches LLM-based web agents to navigate long-horizon tasks by decomposing goals into explicit, verifiable subgoals that serve double duty — as runtime progress checkpoints and as dense training signals for reinforcement learning. The framework solves the problem of agents getting "stuck midway" in multi-step web tasks (looping, losing track of progress) by giving the agent both a mechanism to check its own progress during execution and a learned reward landscape during training that rewards forward movement toward intermediate milestones rather than only rewarding final success.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five interconnected components that operate across two phases (inference and training):

1. Subgoal Generator (Section 5.1): A teacher model (Gemini-2.5-pro) that takes a task instruction and initial webpage screenshot, and produces a fixed-length sequence of natural-language subgoals — coarse-grained milestones like "navigate to the correct repository page" or "search for the college on the map." These subgoals are generated once per task (not during execution) using few-shot prompting with 12 curated examples per website domain.

2. Dynamic Milestoning Framework (Section 5.2): An inference-time mechanism where the agent, at every step, reflects on its action history and current webpage state to answer three questions: "What milestones have I achieved?", "Have I completed the current subgoal?", and "What future milestones should I achieve?" This self-assessment produces a binary subgoal completion vector $\mathbf{z}_{i,t} = [z_{i,1}, ..., z_{i,K}]$ that serves as explicit state awareness, preventing the agent from hallucinating progress and enabling dynamic re-planning when a subgoal is not yet satisfied.

3. Potential Critic $P_\psi$ (Section 5.3, Section 5.3): A learned neural network (built by augmenting a pretrained LLM with an MLP regression head and sigmoid activation) that predicts a continuous progress score $P_\psi(s_t, g) \in [0, 1]$ given the current state and goal. It is trained via supervised regression on interpolated progress labels derived from the subgoal completion vectors of successful trajectories, creating a dense, monotonic shaping landscape.

4. Value Critic $V_\phi$ (Section 5.3): A separate learned network trained on binary terminal rewards (success/failure) that estimates the probability of eventual task success from the current state. It provides the baseline for advantage computation and ensures the final optimization target remains the true task objective, not the auxiliary progress signal.

5. MiRA-RL Training Loop (Section 5.3, Section 5.4, Algorithm 1): An offline RL procedure that combines the potential critic's dense shaping reward with the value critic's success estimate, computes shaped advantages via a doubly-robust estimator, and updates the actor policy using a KL-regularized regression objective (MSE on log-probability ratios against a reference policy). This is wrapped in an outer curriculum loop (Section 5.4, Algorithm 2) that iteratively generates new task distributions from failure analysis.

Information flows through the training pipeline as follows: the actor interacts with web environments on a task distribution → trajectories are annotated with subgoal completion vectors by a SubGoal Checker (LLM-as-Judge) → success/failure labels are assigned by an Outcome Reward Model → progress labels are interpolated from subgoal completions → the potential critic is trained to regress progress labels → the value critic is trained on binary outcome labels → shaped rewards $r'_t = r_t + \alpha(P_\psi(s_{t+1}, g) - P_\psi(s_t, g))$ are computed → shaped advantages are estimated via doubly-robust mixing → the actor is updated by regressing log-probability ratios toward these advantages → failed trajectories seed the next phase's task distribution via semantic similarity resampling.

3.3 Roadmap for the Deep Dive

The explanation follows the paper's own decomposition of the three challenges (C.1–C.3) and then adds the training mechanics:

• Subgoal Generation (C.1): First, I explain where subgoals come from — the teacher model prompting strategy, the few-shot demonstration design, and most importantly, how subgoal quality is validated through correlation analysis with actual task success (Section 5.1). This is foundational because everything downstream depends on subgoal reliability.

• Inference-Time Dynamic Milestoning (C.2): Second, I walk through the runtime mechanism — the self-reflection loop, the SubGoal Checklist with AutoRater, the binary progress vector, and how this transforms opaque execution into structured state awareness (Section 5.2). Understanding this first clarifies what the training procedure is ultimately trying to teach.

• Training with Subgoals — Progress Labeling: Third, I explain how discrete subgoal completions are converted into continuous, smooth progress labels through linear interpolation between key steps, with the critical "gap anchoring" mechanism for the final subgoal segment (Section 5.3). This is the bridge between the symbolic subgoal checks and the differentiable reward shaping.

• Training with Subgoals — Potential Critic and Shaped Rewards: Fourth, I detail the potential critic's architecture, training objective, and role in generating dense shaping rewards $\Delta P_\psi$. I explain the auxiliary nature of these rewards — they densify but do not distort the optimization target — and the restriction to positive trajectories only (Section 5.3).

• Actor Optimization — Policy Mirror Descent and Doubly-Robust Advantages: Fifth, I cover the RL mechanics: the KL-regularized optimal policy form (Equation 5), the MSE regression objective for off-policy learning (Equation 8), the doubly-robust advantage estimator mixing TD-error and Monte-Carlo returns (Equation 10), and the gradient interpretation showing how updates are scaled by both advantage magnitude and KL-constraint (Equation 13).

• Outer Curriculum Loop: Finally, I describe how the training phases chain together — environment interaction, perplexity filtering, replay buffer management, and failure-driven task resampling (Section 5.4, Algorithm 2).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems paper whose core idea is that long-horizon web navigation failures are planning failures that can be addressed by introducing explicit, verifiable subgoals that serve simultaneously as runtime progress monitors and dense RL training signals. The technical approach divides cleanly into three sub-problems matching the three challenges posed in the introduction: (C.1) generating reliable subgoals, (C.2) integrating them at inference time, and (C.3) embedding them in RL training.

---

3.4.1 Subgoal Generation (Addressing C.1 — "Where do subgoals come from, and how reliable are they?")

The paper generates subgoals using a teacher model (Gemini-2.5-pro) prompted with the task instruction and an initial screenshot of the target webpage, producing a fixed-length sequence of natural-language milestones. The generation is done offline, once per task — not dynamically during execution — which separates the planning cost from the interaction cost and allows the same subgoals to serve both inference-time checking and training-time reward shaping.

Methodology and prompting strategy. The teacher model is prompted with the task instruction and initial webpage screenshot, and is "strictly instructed to decompose the task into a fixed number of milestones (e.g., exactly 4 steps)" (Section 5.1). The fixed-size decomposition standardizes temporal granularity across diverse tasks — every task gets the same number of subgoals regardless of its actual length, which matters for the interpolation scheme in Section 5.3. The prompt (Figure 19 in Appendix A.4) provides two example decompositions as few-shot demonstrations:

• Example 1 (GitLab): "Create a private JEKYLL repository called 'awesome_project'" decomposes into: (1) Access GitLab and locate 'Create New', (2) Select the JEKYLL template from the template menu, (3) Enter repo name 'awesome_project' and set it private, (4) Create repository and confirm landing on its homepage.
• Example 2 (Map/Wiki): "Find the page of the college(s) where The Chair was filmed in Pennsylvania other than those in Pittsburgh on the map" decomposes into: (1) Identify correct Wikipedia page for The Chair and locate filming colleges in Pennsylvania (excluding Pittsburgh), (2) Copy the college name from the Wikipedia page, (3) Open the map website and search the copied name, (4) Navigate to the college's location on the map.

These examples demonstrate the desired granularity — subgoals should be coarse enough to represent meaningful progress (not "click button 7") but fine enough to provide checkpoint-level guidance (not "complete the entire task"). The paper uses an iterative in-context learning strategy: "We curate a diverse set of few-shot demonstrations mapping intents to logical intermediate milestones" and "inherently randomize the distribution of these few-shot examples during the prompting phase" to "mitigate positional bias and overfitting to specific action sequences" (Section 5.1). The key design choice is that subgoals are generated from task semantics and visual grounding (the screenshot), not from execution traces — they represent what should happen, not what a particular agent did on a particular attempt.

Curated demonstrations per domain. The paper uses "12 examples for each website category" as few-shot demonstrations (Section 5.1), covering the five WebArena-Lite domains: Reddit, GitLab, Shopping Admin (CMS), Map, and Shopping (OSS). Table 5 in Appendix A.4 provides representative examples across all five domains, showing the diversity of subgoal types:

• Reddit: "Ask for product recommendations for running shoes within a budget of $100 in r/sports" → (1) Navigate to "r/sports" subreddit, (2) Initiate creation of new post, (3) Write title and body asking for shoes under $100, (4) Submit the post.
• GitLab: "Create a new public project 'awesome-llms' and add primer, convexegg, abishek as members" → (1) Navigate to new project creation page, (2) Enter 'awesome-llms', set visibility to public, (3) Finalize project creation, (4) Add primer, convexegg, and abishek in members page.
• Shopping Admin: "Tell me the grand total of invoice 000000002" → (1) Navigate to sales/invoices section, (2) Search for invoice 000000002, (3) Open details for the invoice, (4) Identify grand total amount.
• Map: "Min travel time by car from Animal Rescue League of Pittsburgh to Schenley park?" → (1) Navigate to map website, access directions, (2) Enter "Animal Rescue League" as start, (3) Enter "Schenley park" as destination, (4) Select car mode, find min travel time route.
• Shopping (OSS): "Show least expensive switch card holder with capacity of 15+ cards" → (1) Search for "switch card holder", (2) Filter for capacity of 15 or more, (3) Sort results by price (low to high), (4) Select first item.

These examples reveal the paper's subgoal philosophy: each subgoal is a semantically meaningful, verifiable milestone that could in principle be checked by looking at the current webpage state. "Navigate to the subreddit" is verifiable (is the URL correct?), "Initiate creation of a new post" is verifiable (is the post-creation form visible?), "Select first item" is verifiable (is a specific product detail page displayed?). This verifiability is what enables the SubGoal Checker (the LLM-as-Judge used during training and inference) to determine subgoal completion status.

Validation of subgoal quality — why this matters. The paper does not simply assert that the generated subgoals are good; it quantitatively validates their utility through two types of correlation analysis against ground-truth task success on a validation set of agent traces (Section 5.1). The traces used for validation come from "Agent is trained with MiRA and the base model is Gemma3-12b" (footnote 2), meaning the subgoals are evaluated on the very type of agent that will use them, not on human demonstrations. Let $i$ index a specific trace, $K$ be the total number of generated subgoals (typically 4), and $\mathbf{z}_i \in \{0, 1\}^K$ be the subgoal completion vector indicating which of the $K$ subgoals the agent achieved.

Exact Equivalence analysis. The first analysis tests whether completing all $K$ subgoals is synonymous with task success — i.e., does $\mathbf{z}_i = [1, 1, ..., 1]$ imply $y_i = 1$ (success) and does $y_i = 1$ imply $\mathbf{z}_i = [1, 1, ..., 1]$? The paper reports:

"Under the Exact Equivalence regime, we observe an Equivalence-F1 score of 0.6847. While the Precision is high (0.7917), indicating that completing all subgoals is a strong indicator of success, the Recall is moderate (0.6032). This suggests that strict adherence to every generated subgoal is not a necessary condition for success; the agent occasionally finds valid alternative paths that bypass specific milestones (false negatives)."

In plain language: if all subgoals are completed, the task is very likely successful (precision = 0.79), but there are successful trajectories that don't complete every generated subgoal (recall = 0.60). This is a crucial finding because it means subgoals cannot be used as hard constraints — a policy that terminates as soon as all subgoals are marked complete would miss valid successful paths. Instead, subgoals are better treated as progress indicators on a continuous scale.

Graded Agreement analysis. The second analysis defines a progress score $s_i = \frac{1}{K}\sum_{k=1}^K z_{i,k}$ — simply the fraction of subgoals completed — and tests whether this score discriminates between successful and failed trajectories and whether it increases monotonically with success probability. The results:

"The Area Under the ROC Curve (AUROC) for $s_i$ is 0.84 (Figure 4, left). This high value indicates that the generated subgoals effectively rank-order traces, distinguishing promising trajectories from failures with high reliability."

"As shown in Figure 4 (right), we observe a strictly monotonic relationship between the number of completed subgoals and the empirical probability of success. This is statistically confirmed by a Kendall's $\tau$ rank correlation of 0.4585 ($p < 0.001$)."

This is the validation that licenses using subgoal completion as a dense reward signal: if completing more subgoals reliably correlates with higher success probability, then rewarding the agent for completing subgoals should, on average, steer it toward successful behaviors. The monotonic relationship (Figure 4, right) shows that $P(y=1 \mid m)$ — the probability of success given that $m$ out of $K$ subgoals are completed — strictly increases with $m$. This means there is no "valley" where completing 2 subgoals is worse than completing 1; each additional subgoal shifts the probability distribution rightward.

The paper's conclusion that subgoals "function best as a continuous progress indicator rather than a strict binary gate" (Section 5.1) directly shapes the downstream design: the potential critic is trained to regress continuous progress values, not to classify subgoal completion, and the shaped rewards are continuous increments rather than discrete bonuses for subgoal achievement.

---

3.4.2 Enhancing Online Inference with Dynamic Milestoning (Addressing C.2 — "How can subgoal reasoning be integrated at inference time without prohibitive latency or contextual overhead?")

Standard web agents execute actions reactively: observe the current page state, select an action, execute it, observe the new page state, repeat. This reactive loop has no explicit memory of subgoal progress — the agent might "know" it needs to find a specific page, but it has no structured mechanism to check whether it has already found that page or whether it is making progress toward finding it. The paper's Dynamic Milestoning Framework replaces this opaque execution with a self-reflective loop where the agent, at each time step $t$, queries its own reasoning capability (the same Gemini-2.5-pro model that generates actions) to assess its progress against the pre-generated subgoal checklist.

The three reflective questions. At each step $t$, the agent examines its historical interaction traces (comprising screenshots and action logs) and answers three questions (Section 5.2, Figure 5):

1. "Looking into the historical traces, what milestones have I achieved?" — This requires the model to map its past actions and observed page states onto the specific subgoals in the checklist. For example, if the subgoal was "navigate to the r/sports subreddit" and the action history shows a click on a link followed by a page load with URL containing "r/sports", the model should recognize this as subgoal completion.

2. "Have I completed the current subgoal?" — This focuses attention on the immediate next milestone. The model compares the current webpage state against the expected state for the next pending subgoal, determining whether to advance the progress pointer or continue working on the current subgoal.

3. "What future milestones should I achieve?" — Given the current progress state, this plans forward — identifying the next actionable subgoal and translating it into concrete next actions.

Figure 9 in the main paper provides a concrete walkthrough of this introspection for a GitLab task: "Check out my todos on GitLab, review them and open the first item for details." The subgoals are {g1: Open GitLab and login, g2: Navigate to To-Do list, g3: Review the list of active to-do items, g4: Open the first to-do item detail view}. At a specific timestep, the model reflects: "Reviewing the action log: I opened 'http://10.146.0.3:8023' (satisfies g1), then clicked the nav bar's To-Do icon and landed at '/dashboard/todos' (satisfies g2). g3 and g4 remain undone. Thus, milestones achieved so far: {g1, g2}, and the progress vector is [1,1,0,0]." It then reasons about whether g3 is complete: "The page clearly shows a list of items (titles, links). Since my last reasoning step enumerated them, I mark g3 as completed. Update progress vector: [1,1,1,0]." And plans forward: "Only g4 remains: open the first to-do item. I inspect the DOM and find element_id = 20... Clicking it should bring me to the detailed view."

The AutoRater (LLM-as-Judge) and progress vector. The self-reflection is operationalized through an "AutoRater (LLM-as-Judge) module instantiated within the same thinking model" (Section 5.2). This module compares the current visual state and action history against the pre-generated SubGoal Checklist and produces a binary subgoal completion vector:

$\mathbf{z}_{i,t} = [z_{i,1}, z_{i,2}, ..., z_{i,K}]$

where $z_{i,k} \in \{0, 1\}$ indicates whether the $k$-th subgoal has been achieved for trace $i$ at timestep $t$. The vector is binary — each subgoal is either complete or not — which provides a clean, interpretable progress signal unlike the soft scalar scores produced by learned PRMs.

This vector becomes explicit context injected into the reasoning loop for the next action decision. The paper emphasizes two distinct benefits depending on subgoal status (Section 5.2):

• When a required milestone is not yet satisfied ($z_{i,k} = 0$): The enhanced context "prevents the agent from hallucinating progress." Without the explicit checklist, an agent might "believe" it has completed a step when it actually navigated to a wrong but similar-looking page, or it might skip a step entirely. The binary $z_{i,k} = 0$ forces the model to recognize the gap and "dynamically re-plan, generating actions specifically aimed at fulfilling the pending requirement."

• When a milestone is confirmed ($z_{i,k} = 1$): The completed subgoals "act as contextual anchors, allowing Gemini-2.5-pro to focus its planning capacity on the immediate next step rather than re-processing the entire task history." Instead of holding the full task context in its attention window (which becomes increasingly unreliable over long horizons), the agent can compress its history into a compact abstraction: "I've done g1→g2→g3, now I need g4."

This is the mechanism that directly addresses the "mid-task stuck" failure mode: if the agent enters a loop, the AutoRater will observe that no new subgoals are being completed ($\mathbf{z}_{i,t} = \mathbf{z}_{i,t-1}$ for several steps), recognize this as stagnation, and trigger re-planning rather than continuing to repeat the same actions.

The Gemini-SGO agent. The paper implements this dynamic milestoning using "the thinking capabilities of Gemini-2.5-pro" to create what they call "Gemini-SGO" (SubGoal-Oriented). This is an inference-only enhancement — the base model weights are not modified, only the action selection loop is augmented with self-reflection. The model still generates actions (clicks, types, scrolls) as before, but now conditions those actions on the explicit progress vector and the answers to the three reflective questions.

Inference efficiency and compute trade-offs. A natural concern with adding self-reflection at every step is latency: asking the model to process its full interaction history and reason about progress against a checklist adds computation to every action cycle. The paper addresses this through an analysis of "Thinking Budgets" (Appendix A.2, Figure 14) that compares static compute allocation against dynamic, adaptive allocation.

The static budgets fix a maximum token count per reasoning step (256, 512, 1024, 2048, 4096, 8192, 16384 tokens). The results:

• At 256 tokens per step: ~24.3% success rate, ~6.5 seconds inference time per step
• At 8192 tokens per step: ~32.5% success rate, ~19 seconds inference time per step
• At 16384 tokens per step: success rate drops to ~26%, inference time rises further

The non-monotonic relationship (peak at 8192, decline at 16384) is explained by "diminishing returns" and potentially "unnecessary deliberation without improving decision quality" — the model over-thinks simple decisions.

The dynamic ("Auto") strategy used by Gemini-SGO adaptively allocates compute only when milestone verification is ambiguous. The paper reports: "The Auto (Dynamic) strategy employed by our Gemini-SGO agent effectively resolves this optimization problem. By adaptively allocating compute only when milestone verification is ambiguous, the dynamic model achieves success rates statistically comparable to the maximum static budget (~32.1%) but with significantly lower latency." The reported inference time for Auto mode is 16.74 seconds per step — higher than the minimum static budget but lower than the 8192-token static budget that achieves comparable accuracy.

Why dynamic allocation works better: The paper argues that this is not simply "using more compute" but "intelligently shifting the burden of planning between the amortized cost of offline training and the targeted application of online reasoning" (Section 6.2.1). Most steps in a web navigation task are routine (clicking an obvious "Submit" button, scrolling to view results) and don't require deep reflection. The dynamic strategy spends compute only when the agent needs to verify ambiguous progress — has it reached the right page? Did the search return the expected results? This is the practical resolution to challenge C.2: the overhead is managed by being selective about when to invoke the full self-reflection mechanism.

Relationship to the training component. The Dynamic Milestoning mechanism and the MiRA-RL training procedure are designed to be complementary but independent: "the offline RL phase (MiRA) allows the model to internalize subgoal dependencies into its weights, effectively 'compiling' planning into intuition for common web navigations. The inference-time mechanism (SGO), conversely, serves as a runtime guardrail that is independent of the training phase" (Section 6.2.1). A MiRA-trained open-weight model (like Gemma-3-12B + MiRA) does not perform explicit self-reflection at inference time — the subgoal awareness is baked into its learned policy. The SGO mechanism is only used with proprietary models that cannot be fine-tuned. The paper does not experiment with combining both (training with MiRA and also using dynamic milestoning at inference time), which the discussion section briefly flags as a potential future direction.

---

3.4.3 Progress Labeling via Subgoal Completion (Bridging C.1 and C.3)

The subgoals generated in Section 5.1 are symbolic — they are natural-language descriptions of milestones, and the SubGoal Checker produces binary completion judgments. But RL training needs continuous, differentiable reward signals — discrete "subgoal achieved" bonuses would create a sparse reward problem similar to the terminal reward sparsity the framework is trying to solve. The paper bridges this gap through linear interpolation that converts discrete subgoal completion events into smooth progress labels.

Detecting key steps. For each trajectory and task goal $g$, the system has access to a binary subgoal completion vector at each step: $\mathbf{z}_t = (z_{t,1}, ..., z_{t,K}) \in \{0, 1\}^K$. The cumulative completion count is $c_t = \sum_{k=1}^K z_{t,k}$. Key steps are defined as those timesteps where the cumulative count increases: $c_t > c_{t-1}$ (Section 5.3). These are the moments when a new subgoal was objectively verified as complete.

For example, in a trajectory with $K = 3$ subgoals where subgoal 1 completes at $t_1 = 2$ and subgoal 2 completes at $t_2 = 4$, the key steps are $t = 2$ and $t = 4$. Before $t_1$, $c_t = 0$; between $t_1$ and $t_2$, $c_t = 1$; after $t_2$, $c_t = 2$.

Linear interpolation for smooth labels. Rather than assigning discrete progress values (0 at step 1, $1/K$ after first subgoal, $2/K$ after second subgoal, etc.), the paper interpolates linearly between key steps:

$p_t^* = (1 - \alpha_t)\frac{j}{K} + \alpha_t\frac{j+1}{K}$

where $j$ indexes the subgoal that completed at time $t_j$, $j+1$ indexes the next subgoal that will complete at time $t_{j+1}$, and $\alpha_t = \frac{t - t_j}{t_{j+1} - t_j}$ is the fraction of the interval between the two completion events that has elapsed at step $t$.

What it computes: For any timestep $t$ in the interval $[t_j, t_{j+1}]$, the progress label $p_t^*$ is the linear blend between the progress value at $t_j$ (which is $j/K$) and the progress value at $t_{j+1}$ (which is $(j+1)/K$). At $t = t_j$, $\alpha_t = 0$ so $p_t^* = j/K$; at $t = t_{j+1}$, $\alpha_t = 1$ so $p_t^* = (j+1)/K$; at the midpoint, $p_t^* = (j + 0.5)/K$. The output is a real number in $[0, 1]$ that increases smoothly over the trajectory as the agent progresses through subgoals.

Why this form: The alternative — discrete, step-function progress labels — would create sharp discontinuities in the regression target, making the supervised learning problem harder and creating artificial "cliffs" in the reward landscape that the RL agent might exploit. Continuous interpolation produces a smooth potential function that provides nonzero gradient at every step (not just at the moments when a subgoal flips from 0 to 1), which is crucial for credit assignment: the agent receives positive shaping reward for any action that moves it closer to the next subgoal boundary, not just for the action that crosses the boundary. The linear form (rather than, say, sigmoidal or exponential) is chosen for simplicity and to maintain a constant "rate" of progress between subgoals, reflecting the intuition that actions taken between subgoal completions contribute equally to forward progress.

Gap anchoring for the final segment. A subtle but important design choice concerns the interval after the last subgoal completion. Consider a trajectory where the final subgoal ($j = K-1$ in 0-indexed form, i.e., the $K$-th subgoal) is detected as complete at timestep $t_K$, but the agent continues executing actions (verification, scrolling, final submission, calling exit()) until the true termination at $T$. If the interpolation simply stopped at $t_K$, the progress label would be flat at 1.0 for all remaining steps, providing zero shaping signal during the final administrative actions.

The paper addresses this through gap anchoring: "To provide dense signal during the final administrative steps, we anchor the completion of the last subgoal to the trajectory end $T$ rather than $t_K$. The progress ramps from $(K-1)/K \rightarrow 1.0$ over the interval $[t_{K-1}, T]$" (Section 5.3). In the worked example with $K=3$:

• Subgoal 1 completes at $t_1 = 2$
• Subgoal 2 completes at $t_2 = 4$
• Final subgoal 3 logically completes at $t_3 = 6$ but anchored to $T = 9$

The progress for $t \in [4, 9]$ interpolates from 0.66 to 1.0 over the full interval, not the shorter interval $[4, 6]$. For a step at $t = 7$, the interpolation gives $\alpha_7 = \frac{7-4}{9-4} = 0.6$, so $p_7^* = 0.4 \times 0.66 + 0.6 \times 1.0 \approx 0.86$. This ensures the potential critic learns a monotonically increasing value function that "drives the agent through the final verification steps to the sparse reward at termination" (Section 5.3), rather than plateauing prematurely.

Trajectory selection for label computation. Critically, progress labels are computed only from positive trajectories — those that ultimately reach the final goal. The paper states: "We further restrict shaping signals to those derived from positive traces — i.e., trajectories that ultimately achieve the final goal — ensuring that subgoal completion is statistically correlated with task completion" (Section 5.3, emphasized in the "Auxiliary Nature of Subgoals" breakout box). This is an important design constraint: if progress labels were computed on failed trajectories that happened to complete some subgoals but then went off-track, the learned potential would be contaminated by patterns that don't actually lead to success. The restriction to positive traces guarantees that the progress signal always points toward genuinely successful behavior.

Summary of the labeling pipeline: (1) Rollout trajectories from the current policy or from pretrained checkpoints; (2) Filter to trajectories verified as successful by the Outcome Reward Model; (3) Apply the SubGoal Checker (LLM-as-Judge) to annotate subgoal completion at each timestep, producing binary vectors $\mathbf{z}_t$; (4) Identify key steps where cumulative completion increases; (5) Interpolate linearly between key steps with gap anchoring to produce smooth progress labels $p_t^* \in [0, 1]$ for every timestep. These labels become the supervised regression targets for the potential critic, as described next.

---

3.4.4 Potential Critic for Dense Shaping (Addressing C.3 — "How can intermediate rewards be embedded in RL training to improve credit assignment and stability without hindering final goal completion?")

The potential critic $P_\psi$ is the mechanism that converts the static progress labels from Section 5.4.3 into a learned, differentiable reward landscape that provides dense training signals during RL. It is a separate neural network trained via supervised regression to predict progress, and its outputs are used to construct auxiliary shaping rewards that densify the otherwise-sparse terminal reward signal.

Architecture. The potential critic is built by "augmenting a pretrained LLM with a multilayer perceptron (MLP) head and applying a sigmoid activation to constrain the output range to $[0, 1]$" (Section 5.3). The pretrained LLM provides the base representation — the paper uses Gemma-12B as the backbone — and the MLP head projects the final hidden state to a scalar. The sigmoid ensures the output is a valid probability/progress value between 0 and 1, matching the range of the interpolation-derived labels $p_t^*$.

State construction. The input to the potential critic is the full state $s_t = [a_1, a_2, ..., a_{t-1}, o_t]$ concatenated with the natural-language goal instruction $g$, forming $[h_{s_t}; e_g]$. The paper emphasizes that "successful trajectories solving the same task goal tend to exhibit a stable longest-common subsequence of actions: even though surface-level behaviors vary across agents and tasks, the core semantic steps — such as 'open menu', 'locate repository', 'click clone', 'verify output' — recur with strong regularity" (Section 5.3). By conditioning on the full action-observation history, the potential critic can learn these procedural invariants from data, assigning higher potential values to states that lie along the common semantic paths of successful trajectories.

Training data and procedure. The training data is collected through exploratory rollouts: "We employed Llama3-8b (WebRL) and a vanilla-RL agent — operating without any milestone-based architectural modifications — to perform exploratory rollouts across 1,237 tasks" (Section 5.3). These agents are deliberately NOT the MiRA agent itself, which avoids distribution shift issues (the potential critic should provide useful shaping even before the policy itself has learned). The rollouts generate "a diverse set of execution traces ranging from early failures to complete successes." The successful traces are annotated with progress labels via the interpolation procedure, and the potential critic is trained via standard mean squared error regression:

$\mathcal{L}_P(\psi) = \mathbb{E}_{(s_t, g, p_t^*)}\left[ \lVert P_\psi(s_t, g) - p_t^* \rVert^2 \right]$

where $P_\psi(s_t, g) \in [0, 1]$ is the critic's predicted progress value, $p_t^* \in [0, 1]$ is the interpolated ground-truth progress label, and the expectation is taken over timesteps from successful trajectories.

What it computes: For each state-goal pair $(s_t, g)$, the squared difference between the critic's prediction and the ground-truth progress label. Minimizing this loss teaches the critic to estimate how far the agent has progressed toward task completion, outputting values near 0 for early states, near 0.5 for intermediate states, and near 1 for states close to completion.

Why this form: Mean squared error is the natural regression loss for continuous targets and produces smooth gradients that are well-suited for the critic's role as a differentiable progress estimator. The paper does not use a classification-based loss (as it does for the value critic) because progress labels are continuous, not binary. The choice to train on positive trajectories only (rather than all trajectories) is motivated by the statistical correlation requirement: states from failed trajectories might exhibit high progress under some subgoal metric but not lead to final success, which would create a misleading shaping landscape.

Warm start and online fine-tuning. The potential critic is "utilized as a warm-up model" — trained once on the initial dataset and then "continue[d] to fine-tune the potential critic using fresh online rollouts generated by the current policy" during the RL phases (Section 5.3). This allows the potential landscape to "evolve alongside the agent's capabilities" — as the agent learns new behaviors, the types of states it visits change, and the critic must adapt to remain calibrated on the current distribution.

Auxiliary reward shaping. During RL training, the potential critic's outputs are converted into shaping rewards using the PBRS (Potential-Based Reward Shaping) formulation:

$r'_t = r_t + r_t^{\text{(aux)}} = r_t + \alpha\left(P_\psi(s_{t+1}, g) - P_\psi(s_t, g)\right)$

where $r_t$ is the original sparse environmental reward (0 at all intermediate steps, 1 or 0 at termination), $P_\psi(s_{t+1}, g) - P_\psi(s_t, g)$ is the estimated progress made by the last action, and $\alpha \in [0.1, 0.8]$ is a scaling factor selected via grid search on a held-out validation set (optimal value $\alpha = 0.3$ from Appendix A.6, Table 7).

What it computes: The shaped reward $r'_t$ at each step $t$ is the original sparse reward plus a bonus proportional to how much "forward progress" the potential critic estimates was achieved by the most recent action. If $P_\psi(s_{t+1}, g) > P_\psi(s_t, g)$, the bonus is positive (the agent moved closer to completion); if the potential decreased, the bonus is negative (the agent moved backward or entered a less promising state). The $\alpha$ parameter controls how much influence the shaping signal has relative to the sparse terminal reward.

Why this form: The PBRS formulation $\gamma \Phi(s_{t+1}) - \Phi(s_t)$ is theoretically attractive because it preserves the optimal policy when the potential function $\Phi$ depends only on state (see proof in Appendix A.1). The paper acknowledges that "exact PBRS is rarely tractable" in practice — the learned potential critic $P_\psi$ is not a perfect potential function and may depend on aspects of the trajectory beyond the true state — but "practical variants relax policy-invariance guarantees in favour of learnability" (Section 3.2). The key design principle is that $\alpha$ is kept moderate (0.3) so that the auxiliary signal densifies credit assignment without "overwhelming the ground-truth environmental signal" (Appendix A.6). Table 7 shows that $\alpha = 0.5$ drops success to 28.5% and $\alpha = 0.8$ drops to 25.5%, confirming that the agent over-optimizes for auxiliary progress at the expense of true task completion when shaping dominates.

Auxiliary nature — why subgoals don't corrupt the final objective. The paper is explicit about a critical design constraint: the potential critic provides dense feedback but "does not modify the final optimization target" (Section 5.3, breakout box). Two mechanisms enforce this:

1. "The main critic $V_\phi$ is trained solely on the true task reward (binary success), while $P_\psi$ influences only an additive shaping term." The value function that defines the baseline for advantage computation is trained on terminal outcomes only; the shaping reward is added to the immediate reward signal but does not change what $V_\phi$ eventually learns to predict.

2. "$P_\psi$ is trained entirely from post-hoc subgoal completions (the 'Auto Rater' and 'SubGoal Checker' path in Figure 6) and carries no authority over final success." The potential critic's labels come from an automated checker, not from the environment's ground-truth success signal, so there is no risk of the potential critic accidentally encoding the true terminal reward and creating a self-fulfilling loop.

Additionally, shaping signals are "restricted to those derived from positive traces," meaning the potential critic only learns progress patterns from trajectories that actually succeeded. If a particular subgoal completion pattern does not correlate with final task success in the data, the regression residual for those patterns remains high and their shaping influence naturally diminishes. This statistical correlation constraint is the paper's alternative to the theoretical PBRS guarantee: while the learned potential may not be a perfect potential function, it is grounded in success-correlated behaviors rather than arbitrary heuristics.

---

3.4.5 Goal-Conditioned Value Critic

Alongside the potential critic, the MiRA framework maintains a standard value critic $V_\phi(s, I)$ that estimates the probability of successfully completing task instruction $I$ from state $s$. Unlike the potential critic, the value critic is trained on sparse binary terminal rewards and handles the "what is the expected final outcome" question rather than the "how much progress have we made" question.

Training objective. Because terminal rewards are binary (success = 1, failure = 0), the value function is interpreted as a probability: $V_\phi(s, I) \approx P(\text{success} \mid s, I) \in [0, 1]$. The paper uses a classification-based formulation (binary cross-entropy) rather than regression:

$\mathcal{L}_V(\phi) = -\mathbb{E}_{(s,I) \sim \mathcal{D}}\left[ r(s_T, a_T, I) \log V_\phi(s, I) + (1 - r(s_T, a_T, I)) \log(1 - V_\phi(s, I)) \right]$

where $\mathcal{D}$ is the data distribution (replay buffer + new rollouts), $r(s_T, a_T, I) \in \{0, 1\}$ is the binary terminal success label, and $V_\phi(s, I) \in [0, 1]$ is the critic's predicted success probability from state $s$.

What it computes: The standard binary cross-entropy between the predicted success probability and the true binary label. When $r = 1$ (success), the loss is $-\log V_\phi(s, I)$, which penalizes low predicted probabilities for states that preceded successful outcomes. When $r = 0$ (failure), the loss is $-\log(1 - V_\phi(s, I))$, which penalizes high predicted probabilities for states that preceded failures. The expectation is over state-instruction pairs sampled from the training data.

Why this form: The paper cites recent work (Farebrother et al., 2024) showing that "classification-based formulation for training the value network, as opposed to standard regression, better handles the sparse, binary nature of the terminal rewards" (Section 5.3). The intuition is that binary cross-entropy is the proper scoring rule for probability estimation — it is well-calibrated for Bernoulli targets and naturally constrains the output to $[0, 1]$ through the log-loss shape. MSE regression on binary targets would treat errors near 0.5 the same as errors near 0 or 1, which is suboptimal when the true underlying quantity is a probability. The paper also follows "recent approaches" including DigiRL (Bai et al., 2024) and WebRL (Qi et al., 2024) that adopt this same classification-based value training.

Role in advantage computation. The value critic provides the baseline $V_\phi(s, g)$ used in the doubly-robust advantage estimator (Equation 10), serving as the learned expected return from the current state. This is the critic that remains responsible for modeling final task success — the potential critic's shaping rewards influence the immediate reward term $r'_t$ but the value baseline $V_\phi(s, g)$ is trained only on terminal outcomes, ensuring the advantage signal is ultimately anchored to true task completion.

---

3.4.6 Actor Optimization: Policy Mirror Descent and the MSE Regression Objective

The actor update in MiRA is derived from relative entropy regularized reinforcement learning, specifically Policy Mirror Descent (PMD), which frames the policy optimization as a constrained maximization problem: maximize expected return while minimizing KL-divergence from a reference policy $\pi_{\text{ref}}$. The paper shows that this formulation yields a closed-form optimal policy (Equation 5) and then approximates it via a regression objective that supports off-policy learning (Equation 8).

The KL-regularized optimal policy. The starting point is the constrained optimization objective:

$\max_\pi \mathbb{E}_\pi\left[ r(s, a) - \beta \log \frac{\pi(a \mid s)}{\pi_{\text{ref}}(a \mid s)} \right]$

where $\beta > 0$ is a temperature parameter controlling the strength of the KL penalty (larger $\beta$ means stronger regularization toward $\pi_{\text{ref}}$). Under this objective, the optimal policy takes the form of a Boltzmann distribution re-weighting the reference policy by exponentiated advantage (derived in Appendix A.8):

$\pi^*(a \mid s, I) = \pi_{\text{ref}}(a \mid s, I) \exp\left( \frac{1}{\beta} A^*(s, a, I) \right)$

where $A^*(s, a, I) = Q^*(s, a, I) - V^*(s, I)$ is the optimal advantage function.

What it computes: The optimal policy $\pi^*$ assigns probability to action $a$ in state $s$ for instruction $I$ proportional to the reference policy's probability multiplied by an exponential factor of the optimal advantage. When $A^*(s, a, I) > 0$ (the action is better than average), $\exp(A^*/\beta) > 1$ and the action is upweighted relative to $\pi_{\text{ref}}$. When $A^*(s, a, I) < 0$, the action is downweighted. The temperature $\beta$ controls sensitivity: small $\beta$ makes the policy greedy (sharply favoring high-advantage actions), large $\beta$ pushes it toward $\pi_{\text{ref}}$ regardless of advantage.

Why this form: The Boltzmann distribution is the solution to entropy-regularized RL and has the property of being both tractable (closed-form) and well-behaved (smooth with respect to advantage). The reference policy $\pi_{\text{ref}}$ serves as a prior that prevents the optimal policy from drifting into regions of the action space where the advantage estimates are unreliable (due to limited data). In MiRA, $\pi_{\text{ref}}$ is typically the SFT-trained initial checkpoint, meaning the policy stays close to demonstrated behaviors while being adjusted toward higher-advantage actions.

The MSE regression objective — off-policy learning. Rather than using policy gradient methods (e.g., PPO) that require on-policy data, MiRA approximates $\pi^*$ by minimizing the mean squared error between the current policy's log-probability ratio and the target advantage, over any data distribution $\nu$:

$\arg\min_\theta \mathbb{E}_\nu\left[ \left( \beta \log \frac{\pi_\theta(a \mid s, I)}{\pi_{\text{ref}}(a \mid s, I)} - A^*(s, a, I) \right)^2 \right]$

This is the central policy loss in MiRA (Equation 8 in the paper).

What it computes: For each state-action pair $(s, a)$ in the training data (from any distribution $\nu$ — replay buffer, expert demonstrations, or the current policy), the loss is the squared difference between the scaled log-ratio $\beta \log(\pi_\theta / \pi_{\text{ref}})$ and the target advantage $A^*(s, a, I)$. Minimizing this loss drives $\beta \log(\pi_\theta / \pi_{\text{ref}})$ toward $A^*(s, a, I)$, which means $\pi_\theta$ approaches the Boltzmann form of Equation 5: $\pi_\theta \propto \pi_{\text{ref}} \exp(A^*/\beta)$. The output is a scalar loss averaged over the batch.

Why this form — the key design choice over KL divergence. A natural alternative would be to directly minimize KL divergence between $\pi_\theta$ and $\pi^*$: $\arg\min_\theta \mathbb{E}_s[D_{KL}(\pi^*(\cdot \mid s) \parallel \pi_\theta(\cdot \mid s))]$. The paper explicitly rejects this alternative for two reasons (Sections 5.3 and 6.2.3):

1. Data distribution constraint. Minimizing KL divergence $D_{KL}(\pi^* \parallel \pi_\theta)$ is equivalent to maximizing $\mathbb{E}_{s \sim d(s), a \sim \pi^*(\cdot \mid s)}[\log \pi_\theta(a \mid s)]$ (derived in Appendix A.9). This requires training data to be sampled from $\pi^*$, which in turn is proportional to $\pi_{\text{ref}} \exp(A^*/\beta)$. In practice, this means actions must be drawn from the reference policy $\pi_{\text{ref}}$ (or close to it), "severely limiting the ability to leverage diverse off-policy data from replay buffers" (Section 6.2.3). The MSE objective has no such constraint — it operates on data from any distribution $\nu$.

2. Bidirectional probability adjustment. The KL objective can only increase the probability of sampled actions (since it maximizes $\log \pi_\theta(a \mid s)$ for $a \sim \pi^*$). It cannot explicitly decrease the probability of actions with negative advantage. The paper notes: "When beneficial actions are rare under $\pi_{\text{ref}}$, the KL objective may paradoxically reinforce suboptimal behaviors" (Section 6.2.3). The MSE objective naturally handles both directions: when $A^* < 0$, the regression target $\beta \log(\pi_\theta / \pi_{\text{ref}}) \approx A^* < 0$ drives $\pi_\theta(a \mid s) < \pi_{\text{ref}}(a \mid s)$, downweighting the action.

The ablation in Figure 11b provides empirical support: the KL-based variant (purple curve, "MiRA w. KL") drops below the SFT baseline initially and recovers slowly, achieving only ~33% success by Phase 6 — nearly 10% below the full MSE-based method. This confirms that the MSE objective is not a cosmetic choice but is essential for stable off-policy learning from mixed-quality data.

Gradient interpretation. The paper provides the gradient of the MSE objective (Equation 13) to give mechanistic insight into how updates affect the policy:

$\nabla_\theta \mathcal{L}_\pi = -2\beta \mathbb{E}_{(s,a) \sim \nu}\left[ \left( A^{\text{shaped}}_t - \beta \log \frac{\pi_\theta(a \mid s, I)}{\pi_{\text{ref}}(a \mid s, I)} \right) \nabla_\theta \log \pi_\theta(a \mid s, I) \right]$

This reveals three mechanisms:

• Advantage-guided update: When $A^{\text{shaped}}_t > 0$, the term $A^{\text{shaped}}_t - \beta \log(\pi_\theta / \pi_{\text{ref}})$ is positive (assuming the policy hasn't already fully incorporated the advantage), and the gradient increases $\log \pi_\theta(a \mid s, I)$. When $A^{\text{shaped}}_t < 0$, the term is negative, decreasing the action's log-probability.

• KL-constrained regularization: The term $-\beta \log(\pi_\theta / \pi_{\text{ref}})$ acts as a corrective force. When $\pi_\theta$ already assigns higher probability than $\pi_{\text{ref}}$ ($\log(\pi_\theta / \pi_{\text{ref}}) > 0$), this subtracts from the advantage signal, reducing the update magnitude and preventing overshooting. When $\pi_\theta$ assigns lower probability than $\pi_{\text{ref}}$, it adds to the advantage signal, encouraging recovery toward the reference.

• Magnitude scaling: The $-2\beta$ factor scales the overall gradient, with $\beta$ also appearing inside the parentheses through the KL constraint term. Larger $\beta$ means stronger regularization and smaller effective advantage signals. The paper does not report the specific $\beta$ value used but it is implied to be tuned alongside other hyperparameters (Table 6 shows consistent learning rates and batch sizes across methods but not $\beta$).

Why this is called "regression" rather than "policy gradient": The paper frames this as a "supervised regression target, where the actor predicts the log-ratio and the critic (or return estimator) provides the advantage signal" (Section 5.3). This is distinct from standard policy gradient methods (like PPO or A2C) that estimate the gradient of expected return via likelihood-ratio estimators. The regression formulation is more stable because it "avoids the high-variance gradients associated with probability ratios" and naturally supports off-policy data, allowing the algorithm to "effectively leverage historical high-quality trajectories in the experience pool" without importance sampling corrections. The paper notes that this formulation is "closely related to preference-based objectives such as DPO" (Rafailov et al., 2023), which also regresses log-probability ratios toward externally derived signals.

---

3.4.7 Doubly-Robust Advantage Estimation

The advantage target $A^{\text{shaped}}_t$ used in the regression objective (Equation 12) must be estimated from finite trajectory data, not from the unknown optimal advantage function. MiRA uses a doubly-robust estimator that mixes a low-variance 1-step TD error with a high-variance but unbiased Monte-Carlo return:

$A^{\text{shaped}}_t = \lambda \underbrace{\left( r'_t + \gamma V_\phi(s_{t+1}, g) - V_\phi(s_t, g) \right)}_{\text{1-step TD Error}} + (1 - \lambda) \underbrace{\left( G_t - V_\phi(s_t, g) \right)}_{\text{MC Advantage}}$

where $r'_t = r_t + \alpha(P_\psi(s_{t+1}, g) - P_\psi(s_t, g))$ is the shaped reward, $\gamma$ is the discount factor (set to 0.9 from Table 6), $V_\phi$ is the value critic's predicted expected return, $\lambda \in [0, 1]$ is the mixing coefficient controlling the TD-vs-MC balance, and:

$G_t = \sum_{u=t}^T \gamma^{u-t} r'_u$

is the full Monte-Carlo return — the discounted sum of shaped rewards from step $t$ to the end of the trajectory $T$.

What it computes: The advantage $A^{\text{shaped}}_t$ is a weighted average of two estimates of how much better (or worse) action $a_t$ was than the value critic's baseline:

• The TD-error term $r'_t + \gamma V_\phi(s_{t+1}, g) - V_\phi(s_t, g)$ uses only the immediate reward and the value critic's estimate of the next state's worth. It is low-variance (depends on only one stochastic transition) but biased when the value critic is inaccurate — the benefit of reaching $s_{t+1}$ is estimated from $V_\phi$, which may be wrong.

• The MC advantage term $G_t - V_\phi(s_t, g)$ uses the actual realized returns from step $t$ onward. It is unbiased (uses actual outcomes, not estimated values) but high-variance (depends on the entire stochastic trajectory from $t$ to $T$).

The $\lambda$ parameter blends them: $\lambda = 1$ uses only TD-error (low variance, high bias when critic is poor), $\lambda = 0$ uses only MC returns (unbiased, high variance), intermediate values trade off.

Why this form — the early-phase collapse problem. The paper provides a striking empirical result (Figure 11b, orange curve) demonstrating why the doubly-robust estimator is necessary. The ablation variant "MiRA w/o Doubly Adv." sets $\lambda = 1$ (pure TD-error), and performance collapses to around 25% in early phases before gradually recovering. The explanation: "This occurs because the value critic $V_\phi$ is poorly calibrated at the start of training. Using only the 1-step TD error propagates this bias directly into the advantage estimates, causing the policy to optimize toward incorrect targets" (Section 6.2.3). The policy trusts the critic's wrong estimates and moves in the wrong direction. By Phase 6, as the critic improves through training, performance recovers to over 37% — but the early damage delays convergence significantly.

The doubly-robust estimator mitigates this by including the Monte-Carlo component: "The MC component provides unbiased (though higher variance) gradient signals that anchor learning even when the critic is unreliable, preventing the catastrophic early-phase degradation" (Section 6.2.3). Even if $V_\phi$ wildly underestimates the value of reaching a particular state, the MC return $G_t$ will reflect the actual outcome and pull the advantage estimate toward the correct sign. As training progresses and $V_\phi$ becomes more accurate, the lower-variance TD component naturally dominates (assuming $\lambda$ is not zero), improving estimate precision. This is the "doubly-robust" property: the estimator is robust to both model misspecification (bad $V_\phi$, via the MC term) and high variance (via the TD term). The paper does not report the specific $\lambda$ value used.

Connection to the shaped rewards. Note that the Monte-Carlo return $G_t$ is computed using the shaped rewards $r'_u$, not the sparse environmental rewards $r_u$. This means the potential critic's dense signal propagates through both the TD-error term (via $r'_t$) and the MC return term (via the full sum of shaped rewards). The shaping thus densifies all components of the advantage estimate, not just the immediate TD-error. The value critic $V_\phi$, however, is trained only on the environmental terminal rewards — it learns to predict $\mathbb{E}[\sum_{u=t}^T \gamma^{u-t} r_u]$, the cumulative sparse reward, not the cumulative shaped reward. This separation ensures the advantage signal remains grounded in true task outcomes.

---

3.4.8 Iterative Policy Refinement: The Outer Curriculum Loop

The MiRA training procedure is wrapped in an iterative curriculum that alternates between offline RL updates and online data collection, gradually expanding the task distribution based on failure analysis. This outer loop (Section 5.4, Algorithm 2) addresses a key limitation of static offline RL: "once an agent saturates performance on a fixed task set, it ceases to acquire new skills and fails to generalize to novel task configurations" (Section 5.4).

Phase structure. Each phase $K$ consists of four stages:

1. Environment interaction (rollout collection): The current policy $\pi_K$ (initialized from the SFT checkpoint for Phase 0, from the previous phase's RL training for Phase 1+) is deployed on the task distribution "Task Set for Phase K" to collect trajectories. These are split into successful and failed trajectories based on environment verification (the Outcome Reward Model).

2. Trace processing and filtering:

   • Subgoal annotation: The SubGoal Checker annotates each successful trajectory with binary subgoal completion vectors at each step, which are interpolated into progress labels $p_t^*$.
   • Perplexity filtering: The current actor policy's perplexity is computed on each collected trajectory. Trajectories with perplexity outside a specific range are discarded. The optimal range (determined in Appendix A.6, Table 8) is "moderate-perplexity range $[1/0.9, 1/0.5]$ (i.e., rank score $[0.5, 0.9]$)." These "borderline-difficult" transitions provide the most informative gradients: low-perplexity data is too easy and provides limited learning value (~27.9% success if used exclusively), high-perplexity data is too noisy and destabilizes training (~23.6% success if used exclusively).
   • Replay buffer integration: New successful trajectories are added to the growing replay buffer. The buffer maintains historical high-quality data from all phases, preventing catastrophic forgetting of earlier-learned behaviors.

3. Offline RL training (MiRA-RL inner loop): The aggregated training data (new rollouts + filtered replay buffer) is used to perform one epoch of offline RL updates via Algorithm 1: update value critic, update potential critic, compute shaped rewards and returns, update actor via MSE regression on shaped advantages. This produces the refined policy $\pi_{K+1}$.

4. Curriculum generation (failure-driven task resampling): This is the mechanism that prevents saturation. Instead of training on the same fixed task set every phase, the system generates a new task distribution for Phase $K+1$ based on failures from Phase $K$:

   • The system maintains a pool of 1,573 feasible human-selected tasks (disjoint from the evaluation set).
   • A larger model (Gemini-2.5-flash) identifies semantic similarity between failed task instructions and tasks in the pool, resampling tasks from the pool that exhibit high similarity to failure instances.
   • For "rare or unbalanced task categories where the pool coverage is sparse (e.g., 'Map' navigation)," the system synthesizes new instructions "by perturbing failure samples to ensure balanced distribution" (Appendix A.3).
   • The resampled/synthesized tasks become "Task Set for Phase K+1," and the loop repeats.

Why this curriculum structure: The failure-driven resampling creates a natural progression from easy to hard tasks: early phases expose the agent to a broad distribution, the agent succeeds on easy tasks and fails on harder ones, the next phase's task distribution is weighted toward the harder tasks (those similar to the failures), the agent learns to handle them, and the boundary expands. The paper implicitly grounds this in curriculum learning principles: training on tasks at the frontier of the agent's capability (not too easy, not impossible) is most efficient for skill acquisition.

Implementation details from Table 6 and Appendix A.3:

• Number of phases: At least 6 (based on the Phase 0 through Phase 6 results in Figure 10a). The paper uses "6 rounds of online interaction" for the DigiRL baseline reproduction.
• Rollout temperature: 1 (stochastic sampling during exploration).
• Batch size: 128 for all actor and critic updates.
• Actor and critic learning rates: $1 \times 10^{-6}$ with constant schedule.
• Potential critic learning rate: $2 \times 10^{-5}$ with constant schedule — higher because "the Potential Critic performs a regression task (fitting dense progress scores) rather than standard value estimation, requiring stronger gradient updates to capture the subtle shape of the progress landscape effectively" (Appendix A.4).
• Potential critic training epochs: 3 per phase (vs. 1 epoch for actor and value critic), reflecting the more complex regression target.
• Discount factor $\gamma$: 0.9.
• Replay buffer size: Not explicitly stated for MiRA but DigiRL uses 100,000 (Table 6).
• Maximum trajectory length: Not explicitly stated but trajectories with "total length of less than 15 steps" are retained for potential critic training (Appendix A.7), implying a practical horizon on the order of 15-30 steps for WebArena-Lite tasks.

Why offline batched RL rather than online updates: The paper argues that offline RL offers two practical benefits: "(1) it stabilizes learning by decoupling environment noise from gradient updates, and (2) it enables fine-grained policy improvements at scale by batching large numbers of stored trajectories, including previously failed ones, into a consistent optimization phase" (Appendix A.3). This is particularly important for web navigation where each trajectory involves multiple sequential LLM calls (potentially minutes of wall-clock time) — interleaving gradient updates with environment interaction would be prohibitively slow. The phase-based approach collects data in parallel (multiple trajectories simultaneously), trains in a consolidated batch, and then proceeds.

---

3.4.9 Summary of Design Choices and Their Justifications

Subgoal generation via teacher model + few-shot prompting over learned subgoal generators: provides semantic interpretability and verifiability, avoids the "latent subgoals lack interpretability" limitation of methods like VSC-RL and HIQL. Validated quantitatively through AUROC (0.84) and monotonicity analysis (Kendall's $\tau = 0.46$, $p < 0.001$).

Fixed number of subgoals (typically 4) per task: standardizes temporal granularity for the interpolation scheme. The choice of 4 is empirical (prompts ask for "exactly 4 numbered steps") and reflects a balance between too-coarse (2 subgoals provide insufficient intermediate guidance) and too-fine (8+ subgoals would be harder to verify reliably and would approach the granularity of individual actions).

Dynamic milestoning via self-reflection at inference time over static plan decomposition: prevents the "impaired sub-goal coherence, poor re-planning, and drift" identified in architectures like CUGA. The three reflective questions create an explicit progress loop that static decomposition lacks.

Continuous linear interpolation for progress labels over discrete subgoal bonuses: provides dense gradients at every timestep, not just at subgoal boundaries. Gap anchoring ensures the final administrative steps receive shaping signal.

Potential critic as separate learned network over hand-crafted potential functions: hand-crafted potentials would require domain-specific engineering for each website (GitLab tasks differ from Reddit tasks); a learned critic captures procedural invariants across domains from data.

Restriction of progress labels to positive trajectories only: ensures statistical correlation between the learned progress signal and true success, preventing the shaping landscape from rewarding dead-end behaviors that happen to complete early subgoals.

MSE regression on log-probability ratios over KL divergence minimization: enables off-policy learning from replay buffers, supports bidirectional probability adjustment (both upweighting good actions and downweighting bad ones), and empirically outperforms the KL variant by ~10% absolute success rate (Figure 11b).

Doubly-robust advantage estimation over pure TD-error or pure MC: prevents early-phase collapse when the value critic is poorly calibrated (demonstrated by the "w/o Doubly Adv." ablation dropping to 25% in early phases before recovering), while benefiting from the lower variance of TD estimates as the critic improves.

Moderate shaping factor $\alpha = 0.3$ over larger values: grid search shows $\alpha = 0.5$ or 0.8 causes the agent to overfit to auxiliary rewards at the expense of true task completion, while $\alpha \leq 0.1$ provides insufficient densification.

Failure-driven curriculum resampling over fixed task sets: prevents saturation and progressively expands the agent's competence boundary. Tasks at the frontier of current capability provide the most informative learning signals.

Perplexity filtering to moderate range over using all data: low-perplexity data (~27.9% SR if used alone) provides minimal learning, high-perplexity data (~23.6% SR if used alone) destabilizes training, moderate-perplexity data (~36.4% SR) provides the "borderline-difficult" transitions with most informative gradients.

4. Key Insights and Innovations

Innovation 1: Mid-Task Stagnation as the Dominant and Diagnosable Failure Mode

What's distinctive at the idea level. Prior work on LLM web agents overwhelmingly reports aggregate success rates — "our agent achieves X% on WebArena." This paper makes the intellectual move of asking why agents fail, not just how often, and in doing so identifies mid-task stagnation as the fundamental bottleneck rather than the commonly assumed problems of perception error, wrong-link selection, or insufficient model capability. The automated failure analyzer (Section 4) is not just engineering infrastructure — it is a diagnostic instrument that recharacterizes the problem space. The finding that 42–49% of all failures across model scales and training paradigms are "stuck midway" errors (repetitive loops, navigational dead ends) rather than discrete wrong-termination mistakes changes the framing from "we need better action selection" to "we need better state awareness and progress monitoring."

Comparison to prior work. The dominant evaluation paradigm in web agent research — from WebArena (Zhou et al., 2023b) through WebRL (Qi et al., 2024) to CUGA (Shlomov et al., 2025) — has been outcome-based: report success rate, categorize errors coarsely if at all. When error analysis exists, it tends to be manual and small-scale. The paper's three-function automated analyzer (objective summarization → prioritized rule-based categorization → differential key-decision-step identification) is a systematic diagnostic methodology that uncovers structural patterns impossible to see from aggregate numbers. The insight that a single early decision deviation (Step 3 in Figure 2's example, selecting wrong link on search results) cascades into an entire trajectory's failure — and that this deviation is identifiable by comparison against teacher demonstrations and peer golden paths — is a conceptual advance in how to study agent failures, not just a finding about this particular benchmark.

Significance beyond performance. This diagnostic contribution has implications beyond the paper's own solution. By establishing that stagnation is the dominant failure category across models, the paper creates a new evaluation axis: failure mode distribution, not just success rate. An agent that improves from 30% to 35% success but still loops on 40% of failures is qualitatively different from one that shifts failures from stagnation to wrong-termination — the latter represents a genuine capability shift (the agent can now traverse the full horizon to reach a terminal state), while the former may just reflect better action-level heuristics. The paper explicitly uses this framing to interpret its own results (Section 6.2.5): MiRA's reduction of stagnation from ~33% to ~21% (Figure 13) alongside a rise in wrong-terminations represents progress because the agent has "successfully solved the lower-level planning bottleneck." This is a more nuanced and informative way to evaluate long-horizon agents than aggregate accuracy.

Distinction: fundamental vs. incremental. The failure analysis methodology is a fundamental diagnostic contribution — it provides a lens through which future work can evaluate whether proposed methods actually address the planning problem or merely improve surface-level action quality. The specific findings (42–49% stagnation rates) are benchmark- and model-specific, but the diagnostic approach generalizes. The paper validates the analyzer's soundness through human agreement checks (Table 2, 8–10 out of 10 per category), establishing credibility for this methodology.

Evidence. Figure 3 (failure distribution across Gemini-2.5-pro, Gemma, Gemma-SFT, all showing "Get Stuck Midway" as the dominant category); Table 1 (hardcoded rules defining four mutually exclusive failure modes); Figure 13 (shift in failure distribution post-training, showing MiRA specifically reduces stagnation).

---

Innovation 2: Subgoals as a Unifying Abstraction Across Inference and Training

What's distinctive at the idea level. The paper's central conceptual move is recognizing that the same subgoal structure can serve as both a runtime progress monitor and a dense training signal, unifying two problems that prior work treated separately. Inference-time planning approaches (static decomposition, tree search, self-reflection) address the "what should I do next?" question but don't help the model learn better behaviors from outcomes. Training-time reward shaping approaches (PRMs, HER, intrinsic rewards) densify the learning signal but don't help the deployed agent recognize when it's lost. The paper's thesis — that explicit, verifiable milestones can bridge this gap — means subgoals are not just a decomposition technique but a shared representation between the planning and learning systems.

Comparison to prior work. Process Reward Models (PRMs) like AgentPRM (Xi et al., 2025) and Web-Shepherd (Chae et al., 2025) densify training signals but produce soft, learned scalar scores that are vulnerable to over-optimization and don't provide the agent with interpretable state awareness at deployment time. Hierarchical approaches like VSC-RL (Wu et al., 2025) and HIQL (Park et al., 2023) learn latent subgoals that improve sample efficiency but — as the paper argues — lack the semantic interpretability needed for explicit progress verification. The paper's subgoals achieve a "best-of-both-worlds" property: they are hard, verifiable checkpoints (not soft learned estimates) that provide reliable progress signals for both the training critic and the runtime agent. The paper explicitly positions this against PRMs (Section 2.2): "Unlike PRMs, which rely on noisy, unverifiable scalars to estimate progress, we utilize explicit milestones as rigid semantic checkpoints. This combination allows us to retain the continuous progress tracking while ensuring the verifiable reliability of ground-truth objectives."

What makes this genuinely novel — rather than an obvious combination — is the validation methodology that establishes subgoals as reliable progress estimators despite not being necessary-and-sufficient conditions for success. The paper could have simply asserted "subgoals help." Instead, it quantifies the Exact Equivalence F1 score (0.68 — subgoals are not perfect binary gates), shows the AUROC (0.84 — but they are strong continuous progress indicators), and demonstrates monotonicity (Kendall's τ = 0.46, p < 0.001 — completing more subgoals strictly increases success probability). This careful characterization transforms subgoals from a heuristic decomposition into a calibrated progress signal, which in turn licenses their use as dense reward-shaping targets. The "graded agreement" finding (subgoals work as continuous indicators, not binary constraints) is the keystone insight that distinguishes this work from prior decomposition approaches that assumed strict subgoal-satisfaction implies task completion.

Significance beyond performance. This insight opens a research direction: if subgoals can be validated as calibrated progress estimators (AUROC, monotonicity), then the subgoal generation problem becomes a measurement problem — how do we produce checkpoints whose completion reliably tracks true progress? — rather than a planning problem. This shifts the evaluation of subgoal quality from "do they produce good plans?" to "do they produce well-calibrated progress signals?", which is a more quantifiable and generalizable criterion. The paper's validation methodology (Section 5.1: exact equivalence F1, AUROC, monotonicity, Kendall's τ) provides a template for future subgoal-generation approaches to be evaluated against.

Distinction: fundamental vs. incremental. The subgoal-as-shared-representation insight is a fundamental conceptual reframing. Prior work either used subgoals for inference (static decomposition, dynamic planning) OR for training (reward shaping, curriculum design), but not as a unified abstraction serving both. The validation that subgoals are good progress estimators but not perfect success gates is a key conceptual finding that distinguishes this from brittle "complete all steps → success" assumptions.

Evidence. Figure 4 (left: AUROC = 0.84 for subgoal fraction as success predictor; right: monotonic calibration plot showing strictly increasing P(success | m)); Exact Equivalence F1 = 0.68 (Precision 0.79, Recall 0.60); Figure 12 (subgoal completion dynamics shifting from stagnant vertical band in Phase 0 to diagonal gradient in Phase 6, demonstrating that the agent learns sequential subgoal chaining).

---

Innovation 3: The Dual-Critic Architecture with Auxiliary Shaping That Cannot Distort the Final Objective

What's distinctive at the idea level. The paper introduces an architectural constraint that is more subtle than it first appears: the potential critic $P_\psi$ and the value critic $V_\phi$ are trained on different objective functions from different data sources, and interact only through an additive shaping term whose influence is bounded by design. $V_\phi$ is trained on binary terminal rewards (cross-entropy against success/failure) and serves as the baseline for advantage computation. $P_\psi$ is trained on interpolated progress labels (MSE regression, positive trajectories only) and provides dense shaping rewards. The auxiliary nature of $P_\psi$ — it "carries no authority over final success" (Section 5.3 breakout box) — means that even if the potential landscape is imperfect or learns spurious correlations, it cannot corrupt the optimization target because the value critic $V_\phi$ and the environmental reward $r_t$ remain the ultimate arbiters of what constitutes success.

Comparison to prior work. Standard reward shaping in RL (Ng, 2003; the PBRS framework the paper cites in Section 3.2) provides theoretical policy-invariance guarantees when the potential function depends only on state — but these guarantees break when potentials are learned or approximate. Prior work using learned reward models for shaping (PRMs in web agents, learned intrinsic rewards in exploration) risks reward hacking: the agent discovers behaviors that score highly under the learned model but don't correspond to true progress. The paper's dual-critic design is a pragmatic response to this problem: rather than seeking theoretical guarantees (which are impossible with learned potentials in high-dimensional POMDPs), it builds in architectural constraints that limit the damage an imperfect potential can cause.

The key design decisions that enforce this:

• Separate data sources: $V_\phi$ trains on all trajectories (successes and failures, using terminal labels). $P_\psi$ trains only on positive trajectories (those that ultimately succeed). This ensures $P_\psi$ learns progress patterns correlated with success, not progress-like patterns that lead to dead ends. The paper is explicit: "If a subgoal does not correlate with final task success, its shaping influence naturally diminishes as the regression residual remains high" (Section 5.3).

• Moderate shaping factor $\alpha = 0.3$: The grid search in Appendix A.6 (Table 7) demonstrates that $\alpha > 0.5$ causes the agent to overfit to auxiliary rewards (SR drops to 25.5% at $\alpha = 0.8$), confirming that unbounded shaping would be harmful. The paper doesn't just choose $\alpha = 0.3$ — it validates that the choice matters and shows the U-shaped relationship between shaping strength and final performance.

• The potential critic is not used for advantage computation directly. Shaped rewards influence the immediate reward term $r'_t$ and thereby the Monte-Carlo return $G_t$ and TD-error, but the value baseline $V_\phi(s_t, g)$ is learned from environmental rewards only. This means the advantage estimate $G_t - V_\phi(s_t, g)$ compares shaped returns against a baseline trained on unshaped outcomes — the shaping signal appears as a differential bonus relative to the value estimate, not as a distortion of the value estimate itself.

Significance beyond performance. This design provides a template for safe reward shaping with learned potential functions that generalizes beyond web navigation. The principle — use a learned progress model to densify rewards, but maintain a separate value function trained on true outcomes, and bound the shaping influence through architectural separation and scaling factor tuning — addresses the central tension in reward shaping: how to provide dense guidance without creating perverse incentives. The negative result on $\alpha$ sensitivity (Table 7) and the KL-vs-MSE ablation (Figure 11b, ~10% gap in Phase 6) both serve as empirical validation that the architectural choices are not cosmetic — they prevent specific, documented failure modes.

Distinction: fundamental vs. incremental. The dual-critic architecture is incremental in mechanism (two critics instead of one, additive shaping term) but fundamental in design philosophy — it formalizes the principle that dense guidance must be subordinated to sparse true objectives, not blended with them. This is a design pattern that can be instantiated in other RL-for-agents systems where learned progress models are available.

Evidence. Figure 11b: "MiRA w/o PC" (no potential critic) plateaus at ~35% vs. full MiRA at ~43%, showing the potential critic provides ~8% gain; "MiRA w. KL" (KL divergence alternative) drops to ~33%, confirming the MSE objective's importance for off-policy stability; Table 7: $\alpha = 0.3$ achieves 36.4%, $\alpha = 0.8$ drops to 25.5%, showing the shaping factor must be calibrated.

---

Innovation 4: Test-Time and Training-Time Planning as Complementary Rather Than Alternative Mechanisms

What's distinctive at the idea level. The paper demonstrates — through separate experiments rather than a combined system — that inference-time planning (dynamic milestoning) and training-time RL with dense rewards (MiRA) are complementary mechanisms that address different aspects of the same underlying problem. Gemini-SGO (inference-only, +10% over base Gemini-2.5-pro, from 23.0% to 32.1%) and Gemma+MiRA (training-only, from 6.4% to 43.0%) operate on different axes: the former provides runtime state awareness that prevents the deployed model from losing track, while the latter internalizes subgoal dependencies into the model weights through experience. The paper explicitly frames this complementarity (Section 6.2.1): "the offline RL phase (MiRA) allows the model to internalize subgoal dependencies into its weights, effectively 'compiling' planning into intuition for common web navigations. The inference-time mechanism (SGO), conversely, serves as a runtime guardrail that is independent of the training phase."

Comparison to prior work. Most prior work treats inference and training as a single path: either you have a frozen model with better prompting/planning (prompting-based agents, tree-search methods like Tree of Thoughts), OR you fine-tune the model with better data/objectives (SFT, RL, distillation). The paper's architecture acknowledges that both paths can coexist: training teaches the model what good navigation looks like, inference catches deviations from good navigation in real-time. This is not a theoretical claim but an empirical one: the Gemini-SGO results (Table 3) show gains from inference enhancement alone on an already-capable model, while the Gemma+MiRA results show gains from training alone on a weak base model, and the paper's framing suggests these gains would likely compound (though the combined experiment is left to future work, as noted in Section 7).

The critical insight is that these mechanisms address different failure modes in the same pipeline. The failure analysis (Section 4) shows that agents get stuck because they lack state awareness (don't recognize when they're looping) AND because they haven't learned which intermediate actions matter (credit assignment failure). Dynamic milestoning addresses the first by providing explicit progress checks; MiRA addresses the second by densifying the reward landscape. The fact that both produce substantial gains independently — on different model types and with different mechanisms — is evidence that they are attacking complementary aspects of the long-horizon problem, not redundant solutions to the same sub-problem.

Significance beyond performance. This complementarity insight reframes the "what should we invest in?" question for agent builders. It suggests that the optimal allocation of effort is not "either training or inference enhancement" but "both, targeted at different failure sources." A team building production web agents should invest in: (1) training-time subgoal awareness (MiRA-style dense rewards) to teach the policy good navigation priors, AND (2) inference-time progress monitoring (dynamic milestoning) to catch edge cases and distribution shifts. The paper's failure distribution analysis (Figure 13, Table 4) provides concrete evidence for this: MiRA reduces stagnation (~33% → ~21%) but increases wrong-terminations (~25% → ~31%), while SGO reduces stagnation in proprietary models (48.4% → 39.9%, Table 4). Each mechanism shifts a different part of the error distribution, and the paper's contribution is demonstrating this rather than asserting it.

Distinction: fundamental vs. incremental. This is a fundamental architectural insight rather than a metric gain. The paper does not need to build the combined system to make the point — the separate experiments showing complementary error-mode reduction are sufficient to establish the principle. The explicit acknowledgment that the combined system is left to future work (Section 7) is a strength, not a weakness: it shows the authors understand what they have and haven't demonstrated.

Evidence. Table 3: Gemini-2.5-pro → Gemini-SGO: +9.1% absolute SR (inference enhancement only); Gemma-3-12B base → Gemma+MiRA: +36.6% absolute SR (training enhancement only). Table 4: SGO reduces "Stuck Midway" from 48.4% → 39.9% in proprietary models. Figure 13: MiRA reduces "Stuck Midway" from ~33% (SFT) → ~21% in trained models. The two mechanisms operate on different model types (proprietary vs. open) and produce different error-distribution shifts, establishing complementarity.

---

Innovation 5: MSE Regression on Log-Probability Ratios as a Stable Off-Policy Alternative to KL-Constrained Policy Optimization

What's distinctive at the idea level. The paper makes a methodological argument — supported by both derivation and ablation — that regressing log-probability ratios toward advantage targets (Equation 8) is superior to the more standard KL-divergence minimization for off-policy RL with language models. This is not a new algorithm in the abstract (it builds on Policy Mirror Descent and is related to DPO), but the paper provides a clear head-to-head comparison with a documented failure mode (Figure 11b, "MiRA w. KL" curve) and explains why MSE outperforms KL in this setting. The explanation — KL requires data from $\pi_{\text{ref}}$ and can only upweight actions, while MSE supports arbitrary off-policy distributions and bidirectional probability adjustment — is precise enough to guide future practitioners.

Comparison to prior work. Prior LLM-agent RL systems (DigiRL, WebRL) use variations of Advantage-Weighted Regression (AWR; Peng et al., 2019) or PPO-style updates, both of which have implicit or explicit KL constraints. AWR, in particular, weights regression targets by exponentiated advantage but operates in a different mathematical framework from the policy-mirror-descent formulation used here. The paper's derivation of the MSE objective (Equations 5–8) provides a clean mathematical justification — the optimal KL-regularized policy has a closed form $\pi^* \propto \pi_{\text{ref}} \exp(A^*/\beta)$, and regressing log-ratios toward advantages is a direct supervised approximation — that connects the practical algorithm to the theory. The explicit comparison to KL divergence in Appendix A.9 (showing that KL minimization requires sampling from $\pi^* \propto \pi_{\text{ref}} \exp(A^*/\beta)$, which restricts data distributions) explains why the choice matters for off-policy replay buffers.

Significance beyond performance. The ablation (Figure 11b) shows a nearly 10% absolute success rate gap between MSE and KL formulations by Phase 6 — large enough to be practically meaningful, not just statistically significant. The fact that the KL variant initially drops below the SFT baseline and recovers slowly is the key empirical finding: it validates the theoretical concern that KL-constrained optimization from off-policy data can paradoxically make the policy worse before it gets better. This is a cautionary result for practitioners building RL-for-agents systems: if you're using replay buffers with mixed-quality data, standard KL-divergence policy optimization may actively degrade your model during early phases.

The paper also provides a gradient interpretation (Equation 13) showing the three mechanisms — advantage-guided update, KL-constrained regularization, and magnitude scaling — that makes the MSE objective's behavior interpretable. This is valuable for practitioners who need to debug training dynamics: they can inspect whether the $\beta \log(\pi_\theta / \pi_{\text{ref}})$ term is dominating the advantage signal (indicating the policy has drifted too far from the reference) and adjust $\beta$ accordingly.

Distinction: fundamental vs. incremental. The MSE-on-log-ratios formulation is incremental as an algorithm (it's a special case of Policy Mirror Descent and related to DPO-style regression) but the paper's contribution is fundamental as an empirical demonstration with explanation. The combination of mathematical derivation, ablation comparison to KL divergence, and gradient interpretation provides a complete justification package that future work can cite when choosing optimization objectives for off-policy agent training.

Evidence. Figure 11b: "MiRA w. KL" (KL-divergence alternative) achieves only ~33% at Phase 6 vs. ~43% for full MiRA with MSE, and drops below the SFT baseline (~31%) in early phases. The gradient equation (Equation 13) decomposing the update into advantage-guided, KL-constrained, and magnitude-scaling components.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use WebArena-Lite (Liu et al., 2024), a rigorously curated subset of the WebArena benchmark (Zhou et al., 2023b). The evaluation suite comprises 165 tasks distributed across five distinct real-world application domains: Shopping Admin (35 tasks), Map (26 tasks), Shopping (45 tasks), Reddit (19 tasks), and GitLab (30 tasks). The paper prioritizes this subset over the original 812-task WebArena suite to avoid issues with task feasibility and unstable evaluation — prior analyses showed that many tasks in the full benchmark have underspecified goals or rely on unsupported backend functionality, introducing noise that obscures true agent performance. The reduced task set also lowers evaluation time from over six hours to about 40 minutes, enabling the frequent validation needed for the iterative curriculum design.

• Base model(s). Experiments span two model families. For proprietary models, the paper evaluates GPT-4-Turbo, GPT-4o, Gemini-2.5-flash, and Gemini-2.5-pro, with the Gemini-2.5-pro variant augmented by the SubGoal-Oriented (SGO) inference-time planning framework producing the Gemini-SGO agent. For open-source models, the paper uses Llama-3.1 (8B) and Gemma-3 (12B) as foundational backbones. The Gemma-3-12B base model achieves only 6.4% success rate before any fine-tuning (Table 3, "Gemma3 + SFT (BC)" row yields 30.9%, suggesting the 6.4% figure in the abstract refers to the zero-shot base model), making it a particularly challenging starting point that tests whether the MiRA training framework can teach long-horizon reasoning from a weak initial policy.

• Metrics. The primary metric throughout is task success rate (SR) (%) , measured as Pass@1 — the fraction of the 165 WebArena-Lite tasks for which the agent's single attempt produces the correct final outcome, as verified by the environment's Outcome Reward Model. For Pass@k analysis (Figure 11a), the paper reports the standard unbiased estimator: $\text{pass@}k = 1 - \binom{n-c}{k} / \binom{n}{k}$, where $c$ is the number of successful rollouts out of $n$ total attempts, valid for $k \leq n$. Additionally, the paper reports failure mode distributions (Tables 4, Figure 13) as a secondary evaluation axis, categorizing each failed trajectory into one of four mutually exclusive categories: Stuck Midway, Wrong Termination, Fail Attempt, and Others.

• Baselines. The paper compares against a comprehensive set of baselines spanning multiple paradigms:

  • Proprietary prompting-based agents: GPT-4-Turbo, GPT-4o, Gemini-2.5-flash, Gemini-2.5-pro (all evaluated with standard WebArena prompting, Figure 18).
  • Open-source SFT baselines: AutoWebGLM (6B; Lai et al., 2024), GLM-4-Chat (9B; GLM et al., 2024), GLM-4 + SFT (BC), Llama3.1 + SFT (BC), Gemma3 + SFT (BC). These represent the imitation learning paradigm — models fine-tuned on human or synthetic demonstrations without reinforcement learning.
  • RL-based open-source agents: Llama3.1 + AWR (Advantage-Weighted Regression; Peng et al., 2019), Llama3.1 + DigiRL (Bai et al., 2024), Llama3.1 + WebRL (Qi et al., 2024), Gemma3 + DigiRL, Gemma3 + WebRL. These represent the current state-of-the-art in RL fine-tuning for web agents before MiRA.
  • MiRA ablations: MiRA (w/o PC) — removes the potential critic; MiRA (w. KL) — replaces MSE objective with KL divergence; MiRA (w/o Doubly Adv.) — uses only 1-step TD error for advantage estimation; AWR — the simpler Advantage-Weighted Regression baseline.

• Generation budget / compute accounting. The paper does not measure compute in FLOPs or tokens for the main experiments. Instead, the key resource constraints are training phases (the number of outer curriculum loops, typically up to 6) and rollout counts (the number of environment interactions per phase). For the inference-time analysis (Appendix A.2, Figure 14), the paper measures thinking budget in tokens per reasoning step (256 to 16384) and inference latency in seconds per step. All RL methods use consistent hyperparameters where applicable (Table 6): batch size 128, discount factor $\gamma = 0.9$, actor and critic learning rates $1 \times 10^{-6}$, rollout temperature 1.

• Cross-validation / statistical protocol. For the MiRA training results, all main experiments are averaged over 5 runs ("Results are averaged from 5 runs," Figure 10 caption). For the component ablation (Figure 11b), "each result stands for the average number from 3 trials." For the shaping factor $\alpha$ grid search (Table 7), validation is performed on a held-out set of WebArena-Lite tasks. For the reward model evaluations, training uses a random 10% held-out split (from Appendix A.7, following standard practice). For the failure analyzer validation (Table 2), manual labeling is performed on 40 examples (10 per failure type). The paper does not report confidence intervals or standard deviations for the main success rate numbers in Table 3, which is a notable omission for a benchmark with only 165 test tasks — small absolute differences (2-3%) may not be statistically significant.

---

Main Quantitative Results

Overall Success Rate Comparison (Table 3)

Headline numbers. The paper reports two primary headline results from Table 3:

1. Gemini-SGO achieves 32.1% average success rate, a +9.1 percentage point improvement over the base Gemini-2.5-pro (23.0%), demonstrating the efficacy of inference-time dynamic milestoning on a proprietary model that cannot be fine-tuned.

2. Gemma3 + MiRA (12B) achieves 43.0% average success rate, substantially outperforming all open-source baselines including the prior state-of-the-art WebRL on the same backbone (Gemma3 + WebRL: 35.1%), and surpassing proprietary systems such as GPT-4-Turbo (17.6%) and GPT-4o (13.9%).

Per-domain breakdown. The per-domain results in Table 3 reveal where gains concentrate:

• GitLab (30 tasks): Gemma3 + MiRA achieves 56.7%, compared to Gemma3 + WebRL at 43.3% and Llama3.1 + WebRL at 36.7%. This is the domain with the largest absolute improvement (+13.4% over WebRL), and the paper attributes this to GitLab tasks requiring "strict procedural dependencies that purely sparse-reward methods often fail to capture" (Section 6.2.1).
• Shopping Admin / CMS (35 tasks): Gemma3 + MiRA achieves 54.3%, compared to Gemma3 + WebRL at 40.0% and Llama3.1 + WebRL at 51.4%. This domain and GitLab together represent the strongest gains, consistent with the paper's claim that subgoal-based shaping helps most on tasks with complex multi-step procedural requirements.
• Reddit (19 tasks): Gemma3 + MiRA achieves 73.7%, compared to 68.4% for both Gemma3 + WebRL and Llama3.1 + WebRL. The smaller gain here (+5.3% over WebRL) suggests Reddit tasks have more uniform structure that sparse-reward methods already handle reasonably well.
• Map (26 tasks): Gemma3 + MiRA achieves 30.8%, compared to Gemma3 + WebRL at 30.8% — no improvement. This is a notable negative result: Map navigation tasks appear to benefit minimally from subgoal shaping, possibly because map interactions involve spatial reasoning that the current subgoal granularity does not capture effectively, or because the task pool for Map is sparse (as noted in Appendix A.3: "for rare or unbalanced task categories where the pool coverage is sparse (e.g., 'Map' navigation)").
• Shopping / OSS (45 tasks): Gemma3 + MiRA achieves 28.9%, compared to Llama3.1 + WebRL at 40.0% — MiRA actually underperforms WebRL (Llama3.1) on this domain. This cross-model comparison is confounded by different base models (Llama3.1-8B vs. Gemma3-12B), but the Gemma3 + WebRL baseline (20.0%) shows that MiRA still improves over the same-model WebRL baseline by +8.9% on this domain.

Proprietary model comparison. Gemini-SGO (32.1%) outperforms all other proprietary models tested: Gemini-2.5-pro (23.0%), Gemini-2.5-flash (20.6%), GPT-4-Turbo (17.6%), and GPT-4o (13.9%). The per-domain pattern for Gemini-SGO differs from MiRA: the largest gain over Gemini-2.5-pro appears in GitLab (+20.0%, from 30.0% to 50.0%), while Reddit shows a decrease (-5.3%, from 31.6% to 26.3%). The paper does not analyze why Gemini-SGO underperforms the base model on Reddit specifically, but this may relate to the self-reflection mechanism adding overhead without providing benefits on shorter-horizon tasks where the base model already plans adequately.

Comparison across model scales and training paradigms. The full Table 3 hierarchy shows a clear progression:

• Base proprietary models: GPT-4-Turbo (17.6%) → GPT-4o (13.9%) → Gemini-2.5-flash (20.6%) → Gemini-2.5-pro (23.0%) → Gemini-SGO (32.1%). The SGO enhancement (+9.1%) roughly matches the gap between Gemini-2.5-flash and Gemini-2.5-pro (+2.4%), suggesting inference-time planning is comparable in impact to a full model generation upgrade.
• Base open-source models (SFT only): GLM-4-Chat (6.1%) → AutoWebGLM (18.2%) → Llama3.1 + SFT (20.6%) → Gemma3 + SFT (30.9%). The Gemma3-12B SFT baseline is notably strong, nearly matching Gemini-2.5-pro (23.0%) despite being a much smaller open model.
• RL-trained open-source models: Gemma3 + SFT (30.9%) → Gemma3 + DigiRL (33.3%, +2.4%) → Gemma3 + WebRL (35.1%, +4.2%) → Gemma3 + MiRA (43.0%, +12.1%). The jump from SFT to MiRA represents a +12.1% absolute improvement, while the jump from WebRL to MiRA is +7.9%.

---

Multi-Phase Training Dynamics (Figure 10)

Overall SR across phases (Figure 10a). The paper tracks success rate across 6 training phases (Phase 0 through Phase 6), comparing MiRA against a baseline RL model without subgoal modules (called "standard RL" or "MiRA w/o Subgoal Modules"), WebRL, and the SFT baseline. Key observations:

• MiRA starts at ~31% SR at Phase 0 (the SFT initialization) and steadily improves to ~43% by Phase 6, with near-monotonic improvement across all phases.
• The baseline RL model without subgoal modules starts at the same ~31% but saturates near ~35% by Phase 4-6, demonstrating the "sparse reward problem: without the potential critic's dense shaping signal, learning plateaus" (Section 6.2.3).
• WebRL (on Gemma3, trained from scratch) reaches 35.1% (Table 3), consistent with the baseline RL saturation level.
• The gap between MiRA and baseline RL widens from ~0% at Phase 0 to ~8% by Phase 6, confirming that the subgoal-based shaping provides cumulative benefits that compound across curriculum phases rather than a one-time improvement.

Per-site SR trends (Figure 10b). Domain-wise improvements of MiRA compared to baseline RL show consistent upward trends across Reddit, GitLab, CMS, Map, and Shopping Admin. The strongest absolute gains appear in GitLab and CMS (consistent with Table 3), while Map shows the weakest trend. The paper notes that "MiRA's subgoal-oriented design not only enhances overall learning efficiency but also stabilizes long-horizon behaviors across diverse web environments" (Section 6.2.2).

---

Pass@k Scaling Analysis (Figure 11a)

Beyond Pass@1 (the main metric in Table 3), the paper evaluates Pass@k for $k \in \{1, 2, 4, 8\}$ to assess whether MiRA's improvements reflect genuine policy quality or just lucky sampling. Key findings:

• MiRA consistently outperforms the baseline (MiRA w/o Subgoal Modules) across all sample budgets $k$. At Phase 2, the gap is particularly large: +7.9% at Pass@2.
• The performance gap widens at intermediate phases then narrows slightly. By Phase 6, MiRA maintains a strong lead (+7.5% at Pass@8), but the baseline narrows the gap due to curriculum effects (both models benefit from the failure-driven task resampling).
• Pass@8 for MiRA reaches substantially higher than Pass@1, confirming that allowing multiple attempts (as is common in production deployments) further amplifies the gains from subgoal-based training. The paper does not report exact Pass@8 numbers in absolute terms, but the gap of +7.5% over baseline at Phase 6 suggests MiRA's policy produces a higher density of successful trajectories, not just a lucky first attempt.

---

Failure Mode Distribution Analysis (Table 4, Figure 13)

Proprietary model failure shifts (Table 4). The SGO framework significantly reduces the most problematic failure mode:

• Stuck Midway: Gemini-2.5-pro (48.41%) → Gemini-SGO (39.87%), a reduction of 8.54 percentage points. This directly validates the Dynamic Milestoning mechanism's ability to help the agent recognize and escape navigational dead-ends.
• Wrong Termination: Gemini-2.5-pro (9.52%) → Gemini-SGO (12.03%), a modest increase. The paper interprets this as the agent "now capable of reaching terminal states that were previously inaccessible" (Section 6.2.1) — instead of looping endlessly, it reaches a termination point, even if the termination is premature or on the wrong page.
• Fail Attempt: Remains low and stable (6.35% → 6.96%), confirming that dynamic compute allocation "improves trajectory completion without sacrificing basic instruction adherence."
• Others: Decreases from 11.11% to 8.86%.

Open-source model failure shifts (Figure 13). The comparison across Base (LLaMA3-8b), SFT (Gemma3-12b-SFT), WebRL, and MiRA reveals:

• Stuck Midway: SFT baseline ~33% → WebRL ~25% → MiRA ~21%. MiRA reduces stagnation by approximately 12 percentage points from the SFT baseline and 4 percentage points from WebRL.
• Wrong Termination: Increases from SFT (~20%) → WebRL (~27%) → MiRA (~31%). The paper explicitly frames this as a "clear progression in capability": "Rather than failing to navigate (execution failure), the agent now traverses the full horizon to reach a terminal state, indicating that MiRA has successfully solved the lower-level planning bottleneck." This effectively exposes the higher-level semantic reasoning limitations of the underlying LLM.
• Fail Attempt and Others: Both decrease from the SFT baseline, with MiRA showing the lowest rates in both categories.

The paper's interpretation of the wrong-termination increase is worth scrutinizing: it represents genuine progress if the agent previously couldn't reach the terminal state at all (Stuck Midway → Wrong Termination is a better failure mode), but it also represents a new failure type that the framework does not directly address. The gap between "reaching a terminal state" and "reaching the correct terminal state" is a semantic reasoning challenge that subgoal-based planning, as currently formulated, does not solve.

---

Subgoal Completion Dynamics (Figure 12)

The paper analyzes the temporal evolution of subgoal completion probability across the episode horizon, comparing Phase 0 (initial SFT policy), Phase 1 (early RL), and Phase 6 (fully trained MiRA). This visualization reveals the internal mechanism by which MiRA's training improves behavior:

• Phase 0: "The probability mass is heavily concentrated in the bottom-left region, forming a rigid vertical band over the first two subgoals (indices 0 and 1). This pattern indicates a severe early-stage stagnation: the agent successfully initiates the task but fails to bridge the transition to intermediate objectives, effectively consuming its entire time budget in a local optimum without downstream progress." In plainer terms: the SFT policy can start tasks but gets stuck on the first or second subgoal and never proceeds further.
• Phase 6: "The probability density shifts from this static vertical column to a structured diagonal gradient extending from the top-left to the bottom-right. This strictly monotonic frontier demonstrates that the agent has acquired sequential fluency: it no longer loiters on initial steps but efficiently chains subgoals in lockstep with the episode timeline." The diagonal pattern means that earlier subgoals complete earlier in the episode and later subgoals complete later — the agent has learned to make steady forward progress rather than stalling.

This is arguably the most direct evidence that MiRA teaches the underlying planning capability the paper claims: the policy internalizes the sequential structure of tasks, transitioning from "start and stall" to "chain subgoals efficiently." The emergence of the diagonal gradient — which is a qualitative pattern, not just a metric improvement — provides mechanistic validation of the framework's design principles.

---

Inference Efficiency Trade-offs (Figure 14, Appendix A.2)

For the Gemini-SGO agent, the paper analyzes the relationship between reasoning depth (thinking budget) and both success rate and inference latency. Results from three repeated online experiments across 25 identical tasks at each budget level:

• Static thinking budgets: Success rate peaks at ~32.5% with 8192 tokens per step, at a cost of ~19 seconds per step. At 16384 tokens, success rate drops to ~26% — a clear case of diminishing returns where "additional reasoning may introduce unnecessary deliberation without improving decision quality."
• Minimum static budget (256 tokens): ~24.3% success rate, ~6.5 seconds per step.
• Auto (Dynamic) strategy: Achieves 32.12% success rate — statistically comparable to the optimal static budget — with 16.74 seconds average inference time per step. This represents a ~12% latency reduction compared to the 8192-token static budget while maintaining equivalent accuracy.

The dynamic strategy's advantage is that it "adaptively allocates compute only when milestone verification is ambiguous" rather than spending a fixed budget on every step regardless of difficulty. The paper frames this as confirmation that the framework "succeeds not merely by scaling inference compute, but by intelligently shifting the burden of planning between the amortized cost of offline training and the targeted application of online reasoning."

---

Ablation Studies and Robustness Checks

Potential critic removal (MiRA w/o PC): Removing the potential critic and relying solely on sparse outcome rewards causes performance to plateau at ~35% by Phase 6 (Figure 11b, green curve), compared to ~43% for full MiRA. The gap of ~8 percentage points directly quantifies the contribution of dense subgoal-based shaping. The plateauing behavior confirms that sparse rewards alone cannot overcome the credit assignment problem in long-horizon web tasks, even with the curriculum structure and offline RL optimization.

KL divergence alternative (MiRA w. KL): Replacing the MSE regression objective with KL-divergence minimization causes a severe degradation: performance drops below the SFT baseline (~31%) in early phases and recovers slowly to only ~33% by Phase 6, nearly 10% below full MiRA (Figure 11b, purple curve). The paper explains this through the data distribution constraint: KL minimization requires training data to be sampled from $\pi_{\text{ref}}$, which severely limits the ability to leverage diverse off-policy data from replay buffers. The initial drop below SFT is particularly informative — it demonstrates that the KL formulation can actively degrade the policy when applied to mixed-quality data, a cautionary finding for practitioners.

Doubly-robust advantage estimation removal (MiRA w/o Doubly Adv.): Using only the 1-step TD error ($\lambda = 1$ in Equation 10) causes performance to collapse to ~25% in early phases before gradually recovering to over 37% by Phase 6 (Figure 11b, orange curve). The paper attributes the early collapse to the value critic $V_\phi$ being poorly calibrated at the start of training, with this bias propagating directly into advantage estimates. The eventual recovery demonstrates that the critic improves through training, but the early-phase damage significantly delays convergence. This ablation validates the doubly-robust estimator's role in preventing catastrophic early-phase degradation by mixing in unbiased Monte-Carlo returns.

AWR baseline: The simpler Advantage-Weighted Regression (Peng et al., 2019) achieves only ~29% by Phase 6 (Figure 11b, lowest curve), substantially below all MiRA variants. This confirms that the full MiRA framework — including the MSE objective, potential critic, and doubly-robust estimation — provides benefits beyond a standard off-policy RL baseline.

Reward shaping factor $\alpha$ (Table 7, Appendix A.6): A grid search over $\alpha \in [0.1, 0.8]$ using a held-out validation set reveals:

• $\alpha = 0.0$–$0.1$: 30.9%–31.5% SR — behaves similarly to the sparse-reward baseline, confirming that very weak shaping provides insufficient densification.
• $\alpha = 0.3$: 36.4% SR — optimal, achieving the best trade-off between densifying rewards and preserving the true objective.
• $\alpha = 0.5$: 28.5% SR — performance drops sharply as the auxiliary signal begins to overwhelm the terminal reward.
• $\alpha = 0.8$: 25.5% SR — the worst performance, demonstrating severe over-optimization of the auxiliary progress signal at the expense of true task completion.

This U-shaped relationship validates the paper's claim that the shaping factor must be carefully calibrated and that unbounded shaping would be harmful. The choice of $\alpha = 0.3$ is empirically justified, not arbitrary.

Perplexity filtering bands (Table 8, Appendix A.6): Evaluating different data quality filters for the replay buffer:

• Low-perplexity only ($[1, 1/0.9)$): 27.9% SR — "too easy" trajectories provide minimal learning value.
• High-perplexity only ($(1/0.5, \infty)$): 23.6% SR — noisy, out-of-distribution data destabilizes training.
• Moderate-perplexity ($[1/0.9, 1/0.5]$): 36.4% SR — "borderline-difficult" transitions provide the most informative gradients.
• Full range (no filtering): 29.1% SR — worse than moderate-band filtering, confirming that data quality matters more than data quantity.

This ablation is methodologically important because it shows that the replay buffer filtering is not just a hygiene step but a significant contributor to performance — using all data indiscriminately would reduce success rate by ~7.3 percentage points compared to the optimal band.

Subgoal completion validation (Section 5.1): While not an ablation in the traditional sense, the paper validates subgoal quality through two analyses:

• Exact Equivalence F1: 0.6847 (Precision 0.7917, Recall 0.6032) — subgoals are good precision indicators (completing all subgoals strongly predicts success) but imperfect recall (successful trajectories sometimes bypass subgoals).
• Graded Agreement: AUROC = 0.84, Kendall's $\tau = 0.46$ ($p < 0.001$) — subgoal completion fraction reliably rank-orders trajectories and increases monotonically with success probability.

These results validate the core assumption that subgoals provide useful progress signals, but the moderate Recall (0.60) is a genuine limitation: ~40% of successful trajectories do not complete all generated subgoals, meaning the subgoal decomposition occasionally misses valid solution paths. The paper's design choice to use subgoals as continuous progress indicators rather than binary constraints is a direct response to this finding.

Failure analyzer validation (Table 2): Manual validation on 40 labeled examples (10 per failure type) shows:

• Stuck in Midway: 10/10 agreement — near-perfect detection of procedural stagnation.
• Wrong Termination: 10/10 agreement — reliable identification of incorrect early termination.
• Fail Attempt: 8/10 agreement — "the boundary between a very short unsuccessful attempt and zero attempt can be semantically ambiguous."
• Others: 9/10 agreement.

The high agreement rates (37/40 correct overall) establish the analyzer's reliability as a diagnostic instrument. The slight ambiguity in the "Fail Attempt" category is expected given the semantic nature of distinguishing "tried but failed quickly" from "didn't really try."

---

Critical Assessment

What the Experiments Genuinely Demonstrate

Claim: "MiRA boosts Gemma-3-12B from 6.4% to 43.0% success rate." The experiments in Table 3 genuinely demonstrate a dramatic improvement, but the baseline requires careful interpretation. The 6.4% figure appears to be the zero-shot base Gemma-3-12B (the paper states "GLM-4-Chat (9B) achieves 6.1%" and lists Gemma3 + SFT (BC) at 30.9%). The more relevant comparison is against the SFT baseline (30.9%), since all RL methods (including MiRA) are initialized from SFT checkpoints. Measured from SFT, MiRA provides a +12.1 percentage point gain, which is still substantial but more modest than the +36.6 point headline from the base model. This is an appropriate comparison: SFT represents the best available starting point before RL, and MiRA's gain over SFT (+12.1%) is larger than WebRL's gain over SFT (+4.2%), confirming that subgoal-based shaping provides benefits beyond what curriculum-based sparse-reward RL achieves.

Claim: "Gemini-SGO improves Gemini-2.5-pro by ~10% absolute success rate." Table 3 shows Gemini-2.5-pro at 23.0% and Gemini-SGO at 32.1% — a +9.1 percentage point gain. This is genuinely demonstrated, but with two important caveats. First, the test set is only 165 tasks, so an absolute difference of 9 percentage points represents approximately 15 additional successful tasks — meaningful but not enormous in absolute terms. Second, the per-domain breakdown reveals that the gain is not uniform: GitLab shows +20.0% but Reddit shows -5.3%. The framework helps substantially on procedurally complex tasks but may add overhead that hurts performance on tasks where the base model already plans adequately. The paper does not analyze this domain-specific regression, which is a missed opportunity to characterize when inference-time milestoning is beneficial versus detrimental.

Claim: "MiRA surpasses proprietary systems such as GPT-4-Turbo (17.6%) and GPT-4o (13.9%)." Table 3 confirms this numerically. However, the comparison is between a fine-tuned open model (Gemma-3-12B + MiRA) and zero-shot prompted proprietary models. A fairer comparison would give the proprietary models equivalent fine-tuning or at minimum the same SGO inference enhancement. Gemini-SGO (32.1%) provides a better comparison point for what an enhanced proprietary system achieves, and MiRA (43.0%) still outperforms it, but the gap is 10.9 percentage points, not the 25-29 point gap against GPT-4-Turbo/GPT-4o. The GPT-4 models are also older and weaker than Gemini-2.5-pro, making the comparison partially a reflection of base model capability differences rather than methodology.

Claim: "Mid-task stagnation is the dominant failure mode, and MiRA specifically reduces it." This is the paper's most robustly supported claim. The failure analysis (Figure 3, Section 4.5) establishes that 42-49% of failures across diverse models are "Get Stuck Midway" errors. The post-training analysis (Figure 13) shows MiRA reduces this from ~33% (SFT) to ~21% — a ~12 percentage point reduction. The subgoal completion dynamics (Figure 12) provide mechanistic evidence: the policy transitions from a stagnant vertical band (early subgoals only) to a diagonal gradient (sequential subgoal chaining). The failure analyzer validation (Table 2, 37/40 agreement with human labels) establishes credibility for the diagnostic methodology. This claim holds up well under scrutiny — the paper not only asserts that planning failures dominate, but provides diagnostic infrastructure to measure them and shows that the proposed method specifically targets and reduces them.

Genuine Weaknesses and Limitations

Single benchmark, single evaluation suite. All results are on WebArena-Lite (165 tasks). While the paper justifies this choice (avoiding infeasible tasks, faster evaluation cycles), it limits the generality of the findings. The per-domain results already show substantial variation: MiRA achieves 73.7% on Reddit but only 28.9% on Shopping (OSS) and shows no improvement over WebRL on Map (30.8% for both). This domain-specificity raises questions about whether the approach generalizes to other web benchmarks (Mind2Web, WebShop, VisualWebArena) or to non-web agent tasks. The paper does not evaluate on any held-out benchmark, which is the standard for establishing generalization.

No statistical significance reporting. The main Table 3 reports success rates to one decimal place (e.g., 43.0%, 35.1%, 32.1%) but provides no confidence intervals, standard deviations, or statistical tests. With only 165 test tasks, a difference of 2-3 percentage points (representing 3-5 tasks) could easily arise from sampling variance. The paper mentions "Results are averaged from 5 runs" for Figure 10 and "average number from 3 trials" for Figure 11b, but these appear to apply to the training curves, not the final evaluation numbers in Table 3. The Gemini-SGO vs. Gemini-2.5-pro comparison (32.1% vs. 23.0%, a 15-task difference) is more likely to be significant than Gemma+MiRA vs. Gemma+WebRL (43.0% vs. 35.1%, a 13-task difference), but neither is statistically validated.

The SFT baseline gap is not fully explored. Gemma3 + SFT achieves 30.9% while Llama3.1 + SFT achieves 20.6% — a 10.3 percentage point difference between SFT baselines on different base models. This raises the possibility that Gemma-3-12B is simply a stronger base model for web navigation than Llama-3.1-8B, and that some of MiRA's apparent gains over WebRL (which was evaluated on Llama3.1) are attributable to the base model rather than the methodology. The paper does train Gemma3 + WebRL from scratch (achieving 35.1%) which provides a model-matched comparison, but the large SFT gap suggests base model effects are substantial.

Missing combined experiment. The paper demonstrates that inference-time milestoning (Gemini-SGO) and training-time subgoal shaping (MiRA) are independently effective, but never combines them. The paper acknowledges this explicitly in Section 7 ("we did not experiment with PRM tree-search techniques in combination with revisions"), but the absence of a Gemma+MiRA+SGO experiment leaves open the question of whether the gains are additive, redundant, or even interfering. Given that the paper's central thesis is that subgoals unify inference and training, demonstrating the combined system would be the strongest validation of this thesis.

The wrong-termination increase is under-analyzed. MiRA reduces Stuck Midway failures from ~33% to ~21% (Figure 13) but increases Wrong Terminations from ~20% (SFT) to ~31%. The paper frames this positively: the agent now reaches terminal states rather than looping. But a ~31% wrong-termination rate means nearly one-third of all failures are now cases where the agent confidently terminates with an incorrect answer — a failure mode that may be worse for user trust than looping (which at least signals uncertainty). The paper does not analyze why wrong terminations increase, whether they represent near-misses (correct page, wrong detail) or complete failures (wrong page entirely), or whether the LLM-as-Judge's evaluation criteria might be too strict. This is a significant gap in the failure analysis.

Subgoal generation is never ablated. The paper uses Gemini-2.5-pro with curated few-shot examples (12 per domain) to generate subgoals, but never compares this against simpler alternatives: fixed hand-crafted subgoals per task type, subgoals generated by a weaker model, variable numbers of subgoals, or LLM-generated subgoals without the curated demonstrations. The only validation is the correlation analysis (AUROC = 0.84, monotonicity), which establishes that the generated subgoals are useful but not whether they are optimal or whether simpler alternatives would work as well. Given that subgoal quality is the foundation upon which both inference-time milestoning and training-time shaping depend, this is a notable missing ablation.

The 1,237-task training set for the potential critic is not described in detail. The paper states that rollouts were collected "across 1,237 tasks" using Llama3-8b (WebRL) and a vanilla-RL agent, but does not specify the relationship between these 1,237 tasks and the 165-task test set — are they from the same distribution? A separate WebArena training split? This matters because if the potential critic is trained on tasks highly similar to the test tasks, its progress estimates may benefit from task-specific patterns that wouldn't generalize.

No latency analysis for MiRA-trained models. While the paper provides inference-time latency analysis for Gemini-SGO (Figure 14), it does not report inference latency for the Gemma+MiRA agent. Since MiRA does not add inference-time overhead (the potential critic is only used during training), this is a reasonable omission, but it means the paper cannot directly compare the cost-per-success of Gemini-SGO vs. Gemma+MiRA. A wall-clock or FLOPs-matched comparison between the proprietary SGO approach and the open-source MiRA approach would be informative for practitioners choosing between them.

Experiments That Would Have Strengthened the Paper

1. A combined MiRA + SGO experiment on an open-weight model that supports both fine-tuning and inference-time reasoning. This would directly test whether the two mechanisms are complementary or redundant.

2. Subgoal generation ablation: Compare Gemini-2.5-pro generated subgoals against human-written subgoals, against a weaker model's subgoals, and against no subgoals (pure sparse RL). This would quantify the value of subgoal quality.

3. Cross-benchmark evaluation on at least one additional web navigation benchmark (Mind2Web, WebShop, or VisualWebArena) to establish generalization.

4. Statistical significance tests (e.g., bootstrap confidence intervals on the 165-task test set) to determine whether the reported differences are reliable at this sample size.

5. Per-trajectory cost analysis comparing MiRA training cost (phases × rollouts × LLM calls) against the inference-time cost of Gemini-SGO, to determine the break-even point for deployment scenarios.

6. Analysis of wrong-termination errors — categorizing them into sub-types (near-miss vs. completely wrong page) to understand whether they represent semantic reasoning failures that subgoal-based planning cannot address.

7. Variable subgoal counts — testing whether the fixed-4-subgoal decomposition is optimal or whether task-adaptive subgoal counts would improve the correlation between subgoal completion and success.

Conditional Nature of the Claims

The paper's central claim — that explicit milestone reasoning resolves the planning bottleneck in long-horizon web agents — is supported with specific boundary conditions:

• Holds most strongly on procedurally complex tasks (GitLab: +13.4% over WebRL, CMS: +14.3% over WebRL) where the sparse-reward credit assignment problem is most severe and subgoal structure provides the clearest guidance.
• Holds weakly or not at all on spatially-oriented tasks (Map: 0% improvement over WebRL) where the current subgoal granularity may not capture the relevant reasoning steps.
• Reduces stagnation but increases wrong-terminations, meaning the framework shifts the failure distribution rather than eliminating failures entirely. The agent becomes better at navigating but still struggles with verifying — it can traverse the full task horizon but cannot reliably determine whether it has reached the correct terminal state.
• Requires careful calibration of the shaping factor $\alpha$ (optimal at 0.3, harmful at 0.5+) and the perplexity filtering band (optimal at $[1/0.9, 1/0.5]$), meaning the gains depend on hyperparameter tuning that may not transfer to new domains.
• Depends on subgoal quality (validated at AUROC = 0.84), but subgoals are not perfect (Recall = 0.60 for exact equivalence), so the framework works best when subgoal completion is treated as a continuous progress signal rather than a hard constraint.

These conditions are not weaknesses per se — the paper is generally transparent about them — but they mean the 43.0% headline number should be understood as the result of a carefully tuned system on a specific benchmark, not as a plug-and-play solution that transfers without adaptation.

6. Limitations and Trade-offs

6.1 The Hardest Problems Remain Essentially Unsolved — Test-Time Compute Cannot Substitute for Missing Capability

The assumption or constraint. The paper's entire framework — both dynamic milestoning (SGO) and MiRA training — operates under an implicit assumption that the agent has some non-trivial probability of producing correct behaviors, which the subgoal mechanism can then amplify through better planning and credit assignment. The paper is transparent about the boundary of this assumption in Section 7's discussion of the pretraining-vs-inference tradeoff, where it notes that on the hardest problems, "test-time compute provides essentially zero benefit regardless of budget." But this limitation is structural, not just quantitative: both SGO and MiRA amplify existing capability but do not create it from nothing.

The consequence. On WebArena-Lite tasks where the base model's pass@1 is near zero, neither inference-time planning nor training-time shaping helps. The evidence is visible in the per-domain results (Table 3) but more starkly in the subgoal completion dynamics (Figure 12). At Phase 0, the probability mass concentrates on the first 1–2 subgoals and never progresses to later ones — the agent cannot even begin to execute the intermediate steps that subgoals are designed to encourage. MiRA training shifts this to a diagonal gradient (Phase 6), but this represents the policy learning to chain subgoals on tasks where it already had some capability, not acquiring entirely new competencies. For a practitioner, this means the framework cannot rescue an agent that fundamentally lacks the perception, language grounding, or procedural knowledge needed for a task domain — it can only make an already-competent agent more reliable on longer horizons.

What evidence exists in the paper. The failure analysis (Section 4, Figure 3) shows that base Gemma models exhibit "Fail to Make Reasonable Attempt" errors in approximately 32% of failures, representing cases where the agent never even starts coherent execution. The subgoal completion dynamics (Figure 12, Phase 0) visually demonstrate stagnation at the earliest subgoals. The Map domain results (Table 3) show Gemma3 + MiRA tying Gemma3 + WebRL at 30.8% — the subgoal mechanism provides zero improvement on a domain where the base model may lack fundamental spatial reasoning capabilities. The paper does not provide a quantitative breakdown of per-difficulty-bin performance (as some prior work does, e.g., the MATH benchmark papers that bin by pass@1), which would make this limitation more precisely measurable.

Mitigation status. The paper does not attempt to address this — it acknowledges in Section 7 that "future work must still address the cold start exploration problem in environments where even the first milestone is hard to reach." The current framework has no mechanism for bootstrapping capability on tasks beyond the model's initial competence frontier. For a practitioner, this means that before deploying MiRA, one must verify that the base model (post-SFT) achieves non-trivial success rates on the target task distribution; if pass@1 is near zero, MiRA training will not help.

---

6.2 The Difficulty Estimation Cost (Subgoal Generation and Progress Labeling) Substantially Exceeds the Training Budget and Is Not Amortized

The assumption or constraint. The MiRA framework depends on a pipeline of expensive, large-model inferences that are performed outside the RL training loop and whose cost is never accounted for in any efficiency comparison:

1. Subgoal generation (Section 5.1): Gemini-2.5-pro is prompted with the task instruction, an initial screenshot, and 12 curated few-shot examples per domain to produce subgoals. This is a multimodal reasoning call (image + text input, structured text output) on one of the most compute-intensive proprietary models available.

2. Progress labeling (Section 5.3): For every positive trajectory used to train the potential critic, a SubGoal Checker (Gemini-2.5-pro again) must evaluate subgoal completion at each timestep. The paper uses 1,237 tasks for the initial potential critic training data, each with multiple rollout trajectories. Even conservatively estimating 3–5 successful trajectories per task × ~10–20 steps per trajectory, this represents tens of thousands of LLM-as-Judge calls.

3. Failure analysis and curriculum generation (Section 4, Section 5.4): The automated failure analyzer uses Gemini-2.5-flash to classify failures, identify key decision steps via differential analysis against teacher demonstrations, and resample tasks based on semantic similarity. Each phase of the curriculum loop triggers a new round of failure analysis.

The paper acknowledges this in Section 3.2 when discussing difficulty estimation for the broader test-time compute framework: "estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity." But in MiRA's case, the labeling cost is an order of magnitude larger than simple difficulty estimation because it involves per-timestep annotation across thousands of trajectories.

The consequence. The headline result — "Gemma-3-12B boosted from 6.4% to 43.0%" — compares the amortized cost of training (GPU hours for RL fine-tuning) while treating the subgoal generation and progress labeling as zero-cost oracle signals. In a realistic deployment where a practitioner must generate subgoals and annotate progress labels from scratch for a new task domain, the total cost would include:

• $C_{\text{subgoal}}$: Gemini-2.5-pro calls for subgoal generation (one per task type)
• $C_{\text{label}}$: Gemini-2.5-pro calls for per-timestep progress annotation across all training trajectories
• $C_{\text{failure}}$: Gemini-2.5-flash calls for failure analysis and curriculum generation per phase
• $C_{\text{RL}}$: the actual RL training compute (GPU hours on Gemma-12B)

For a domain with 1,000 tasks, the labeling cost alone could exceed the RL training cost by a large margin. The paper provides no comparison of $C_{\text{label}}$ vs. $C_{\text{RL}}$, making it impossible for a practitioner to determine whether the 43.0% success rate justifies the total pipeline cost. This is particularly acute because the labeling depends on proprietary models (Gemini-2.5-pro) that may not be available or affordable in all settings.

What evidence exists in the paper. The paper provides no accounting of subgoal generation cost, progress labeling cost, or failure analysis cost. Section 5.1 mentions "12 examples for each website category" as few-shot demonstrations but does not estimate token count or inference time. Section 5.3 mentions "1,237 tasks" for potential critic data but does not estimate the number of SubGoal Checker calls. Appendix A.3 mentions using "Gemini-2.5-pro and engineered prompts" for subgoal annotation and "Gemini-2.5-flash" for failure analysis, but without cost estimates. The absence of any cost analysis for the labeling pipeline is a significant gap.

Mitigation status. The paper does not mitigate this — it explicitly excludes these costs and flags them as out of scope. Section 7 mentions "transitioning from heuristic prompts to learnable or hierarchical subgoal generators" as a future direction, which would reduce the per-domain subgoal generation cost but not address the per-trajectory progress labeling cost. For a practitioner, the key question is whether the progress labeling can be done with a cheaper model (e.g., a fine-tuned small classifier rather than Gemini-2.5-pro) or whether the potential critic can be trained with fewer annotated trajectories. Neither question is addressed.

---

6.3 The Approach Is Validated on a Single Small Benchmark Without Generalization Evidence

The assumption or constraint. All experiments — the failure analysis, SGO inference enhancement, MiRA training, all ablations — are conducted exclusively on WebArena-Lite (165 tasks across five domains). The paper justifies this by noting that the full WebArena benchmark contains "underspecified goals or rely on unsupported backend functionality" (Section 6.1) and that the reduced set enables faster evaluation. However, 165 tasks is a very small sample for evaluating an agent system, and the five domains are specific curation choices that may not represent the diversity of real web tasks.

The consequence. Several specific risks arise from single-benchmark evaluation:

• Overfitting to benchmark structure. The subgoal generation prompts are "iteratively optimized until we find the best generation contextual inputs for the teacher model" using a validation set of MiRA agent traces (Section 5.1, footnote 2). This means the subgoal format, granularity, and few-shot examples are tuned specifically to WebArena-Lite task patterns. A practitioner deploying on a different web benchmark (Mind2Web, WebShop, VisualWebArena) or a proprietary web application would need to re-tune the subgoal generation — and the paper provides no evidence that the AUROC = 0.84 validation transfers.

• Domain-specific performance variance is large and unexplained. Table 3 shows MiRA achieving 73.7% on Reddit but only 28.9% on Shopping (OSS) — a 44.8 percentage point gap. This variance is larger than the gap between MiRA and the SFT baseline (+12.1%), meaning domain choice dominates methodology choice in determining absolute performance. A practitioner whose target domain resembles Shopping rather than Reddit would see dramatically different results than the 43.0% average suggests.

• No cross-benchmark correlation established. The paper does not evaluate whether success on WebArena-Lite correlates with success on other web agent benchmarks, or whether the failure mode distribution (42–49% stagnation) generalizes. The automated failure analyzer — one of the paper's key diagnostic contributions — is validated on 40 manually labeled examples from WebArena-Lite (Table 2), but its categories and rules (Table 1) may be specific to WebArena's interaction patterns.

• The Map domain shows no improvement from MiRA (30.8% for both MiRA and WebRL, Table 3). The paper notes this in passing but does not analyze why. If subgoal-based shaping provides zero benefit on spatially-oriented tasks, this is a significant scope limitation that should be characterized, not just reported.

What evidence exists in the paper. The per-domain breakdown in Table 3 directly reveals the variance problem: GitHub (56.7%), CMS (54.3%), Reddit (73.7%), Map (30.8%), OSS (28.9%). The paper briefly notes that GitLab and CMS gains come from "strict procedural dependencies" but does not provide a systematic analysis of which task characteristics predict MiRA's effectiveness. The 40-example validation of the failure analyzer (Table 2) is the only generalization check in the paper, and it is limited to verifying that the analyzer's categories are consistent with human judgment on the same benchmark, not that the categories apply to other benchmarks.

Mitigation status. The paper does not mitigate this — it acknowledges in Section 7 that "several dimensions warrant further exploration to generalize this approach" but does not provide cross-benchmark results, domain-characteristic analysis, or out-of-distribution evaluation. The "future directions" mention "dynamically tailoring the granularity of milestones for knowledge-sparse domains," which implicitly acknowledges that the current fixed-4-subgoal approach may not generalize. For a practitioner, this means the 43.0% headline number should be treated as specific to WebArena-Lite's task distribution and may not predict performance on other web automation tasks.

---

6.4 The Training Regime Requires Proprietary Teacher Models That Constrain Reproducibility and Scaling

The assumption or constraint. MiRA's training pipeline depends on proprietary, API-access-only models (Gemini-2.5-pro and Gemini-2.5-flash) for three critical functions that cannot be replaced by the open-weight model being trained:

1. Subgoal generation (Section 5.1): Gemini-2.5-pro with multimodal prompting (screenshot + text) produces the subgoals. The paper uses 12 hand-curated few-shot examples per domain, which are domain-engineering artifacts that a practitioner would need to recreate.

2. Progress labeling (Section 5.3, Appendix A.3): The SubGoal Checker is "few-shot based Gemini-2.5-pro and engineered prompts." This model judges subgoal completion at every timestep of every positive trajectory.

3. Failure analysis and curriculum generation (Section 4, Section 5.4): "Gemini-2.5-flash" is used to "identify semantic similarity" for task resampling and to perform the three-function failure analysis (objective summarization, categorization, key-decision-step identification).

These are not minor utilities — they are infrastructure that the RL training cannot proceed without. The open-weight model being trained (Gemma-3-12B) never learns to generate its own subgoals or evaluate its own progress at training time; those capabilities remain externalized to the proprietary teacher.

The consequence. This creates several practical barriers:

• Reproducibility: The paper does not release the generated subgoals, the SubGoal Checker prompts, or the labeled progress data, making exact reproduction impossible without access to the specific Gemini model versions used. Even with access, proprietary model APIs are non-deterministic and subject to change, meaning results may not replicate.

• Cost scaling: For each new task domain or website, the entire labeling pipeline must be re-run. If a practitioner wants to deploy MiRA on an internal enterprise web application with 500 task types, they must: (a) curate 12 few-shot subgoal examples per task type (or domain), (b) run Gemini-2.5-pro to generate subgoals for all 500 tasks, (c) collect initial rollouts (thousands of trajectories), (d) run Gemini-2.5-pro to annotate subgoal completion at every timestep, and (e) run Gemini-2.5-flash for failure analysis and curriculum generation each phase. The paper provides no guidance on whether cheaper models (Gemini-2.5-flash for subgoal generation, or a fine-tuned Gemma-based checker) could substitute.

• Capability ceiling: The trained agent can never exceed the subgoal quality of the teacher model. If Gemini-2.5-pro produces subgoals that miss important edge cases (as the Recall = 0.60 in the Exact Equivalence analysis suggests — ~40% of successful trajectories bypass some generated subgoals), the MiRA-trained policy will be shaped toward a suboptimal progress metric. The paper does not investigate whether subgoal quality is the bottleneck for further improvement.

What evidence exists in the paper. Section 5.1 explicitly cites Gemini-2.5-pro as the subgoal generator and validates subgoal quality against MiRA agent traces. Appendix A.3 states: "We adopt few-shot based Gemini-2.5-pro and engineered prompts" for the SubGoal Checker, and "We use larger models such as Gemini-2.5-flash to identify semantic similarity" for curriculum generation. Table 5 shows example subgoals across domains, implicitly demonstrating the teacher model's output quality. The paper does not include an ablation where subgoals are generated by a weaker model or where progress labeling is done with a smaller model, making it impossible to assess how much the teacher model's capability contributes to MiRA's gains.

Mitigation status. The paper does not mitigate this. Section 7 mentions "transitioning from heuristic prompts to learnable or hierarchical subgoal generators" as a future direction, which would reduce dependence on hand-crafted few-shot prompts but would likely still require a capable teacher model for training the generator. The broader question — can the RL-trained agent learn to self-evaluate progress without an external teacher? — is not addressed.

---

6.5 Wrong-Termination Errors Increase as Stagnation Decreases, Exposing a Semantic Reasoning Bottleneck the Framework Does Not Address

The assumption or constraint. The MiRA framework is designed to solve the planning bottleneck — helping agents maintain progress awareness and avoid navigational loops. It does not claim to address the verification bottleneck — determining whether the reached terminal state actually satisfies the task requirements. The potential critic is shaped toward subgoal completion, and the value critic is trained on terminal success signals, but neither provides direct supervision for the semantic reasoning needed to verify "am I on the correct page with the correct information?"

The consequence. The failure distribution shifts (Figure 13, Table 4) reveal a clear trade-off:

• SFT baseline: ~33% Stuck Midway, ~20% Wrong Termination
• WebRL: ~25% Stuck Midway, ~27% Wrong Termination
• MiRA: ~21% Stuck Midway, ~31% Wrong Termination

MiRA reduces stagnation by ~12 percentage points from SFT but increases wrong terminations by ~11 percentage points — the gains in planning are almost exactly offset by losses in verification. The agent now reliably traverses the full task horizon to reach a terminal state, but it terminates on the wrong page or with incorrect information ~31% of the time. For a practitioner, this shift may not represent a net improvement in user experience: a looping agent that never terminates is bad, but an agent that confidently reports wrong information may be worse because it erodes trust without signaling uncertainty.

The paper frames this as "a clear progression in capability" (Section 6.2.5), arguing that solving the lower-level planning bottleneck exposes the higher-level semantic reasoning limitation. This framing is valid as a research narrative but is misleading as a deployment assessment: the total failure rate (Stuck Midway + Wrong Termination) is roughly constant across SFT (~53%), WebRL (~52%), and MiRA (~52%). MiRA redistributes failures across categories but does not substantially reduce the overall failure count.

What evidence exists in the paper. Figure 13 provides the failure mode distribution comparison across Base, SFT, WebRL, and MiRA. Table 4 provides the same for Gemini model variants. The paper explicitly notes: "the rate of wrong termination is compounded by the AutoRater (LLM-as-Judge), which judges success based on reaching the correct terminal page, often failing to verify the semantic details of the final answer" (Section 6.2.5). This acknowledges that the evaluation metric itself may be contributory, but the paper does not decompose wrong terminations into sub-categories (e.g., correct page but wrong extracted value vs. completely wrong page vs. premature termination before task completion) that would clarify the nature of the remaining failures.

Mitigation status. The paper does not mitigate this. The dynamic milestoning framework (SGO) includes a self-checking mechanism that verifies subgoal completion, but this verification is limited to whether the agent reached the expected page, not whether the information on that page satisfies the task intent. The paper does not propose a verification-specific training objective or an inference-time verification mechanism for the final answer. Section 7 briefly mentions "signal annealing strategies" where "subgoals would serve as temporary 'warm-up' scaffolding" — but this addresses the risk of over-optimizing for auxiliary rewards, not the semantic verification gap.

---

6.6 The Potential Critic Depends Entirely on Positive Trajectories, Making the Shaping Signal Unavailable for Recovery from Early Failures

The assumption or constraint. The potential critic $P_\psi$ is trained exclusively on positive trajectories — those that ultimately reach the final goal (Section 5.3 breakout box, Appendix A.7). The paper justifies this with a statistical argument: "Restricting the regression dataset to successful episodes produces a potential function that implicitly captures the common semantic subsequences shared across successful attempts, creating a shaping landscape that rises precisely along paths known to lead to final success." If failed trajectories were included, the progress labels might reward dead-end behaviors that happen to complete early subgoals but never reach the final goal, creating a misleading shaping landscape.

The consequence. This design choice creates an asymmetry: the shaping signal is available only when the agent is already on a path that will (eventually) succeed. When the agent makes an early mistake — clicks the wrong link, misinterprets a search result, navigates to a dead-end page — the potential critic provides no guidance for recovery because it has never seen successful trajectories that pass through that erroneous state. The shaping landscape is defined over the manifold of successful paths, but the agent's exploratory behavior may wander far from that manifold, into regions where $P_\psi$ is untrained, poorly calibrated, or essentially random.

The paper implicitly acknowledges this in Section 7: "If an agent fails to ground the initial subgoals due to extreme exploration difficulty or perception errors, the shaping signal remains silent, effectively reverting the optimization to a sparse-reward regime." For a practitioner, this means that the potential critic is a "rich get richer" mechanism — it provides dense guidance to agents that already have some probability of success, but abandons agents that deviate early, exactly when guidance is most needed.

What evidence exists in the paper. This limitation is not directly measured. The paper does not evaluate whether the potential critic's predictions are calibrated on failed trajectories, whether shaping rewards are informative in early-failure states, or whether the learning dynamics differ between trajectories that stay on the success manifold and those that deviate. The subgoal completion dynamics (Figure 12) show that Phase 0 policies stall at early subgoals — these are trajectories where the potential critic likely provides minimal signal because the agent never enters the regions where $P_\psi$ was trained. The fact that MiRA still improves from Phase 0 to Phase 6 despite this limitation suggests that the sparse terminal reward and the curriculum structure provide sufficient signal for eventual recovery, but the paper does not isolate the potential critic's contribution in early-failure scenarios.

Mitigation status. The paper does not mitigate this. The design choice to restrict to positive trajectories is presented as a necessary constraint for statistical validity ("ensuring that subgoal completion is statistically correlated with task completion"), and the paper does not explore alternatives: training on all trajectories but weighting by eventual success, using hindsight relabeling to treat failed trajectories as partial successes, or training a separate "recovery" potential on trajectories that fail but recover. The future work mention of "cold start exploration" (Section 7) acknowledges the broader problem but does not propose specific solutions for the positive-trajectory restriction.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around LLM-based web agents from a focus on action quality (did the agent click the right button?) toward a focus on state awareness and progress monitoring (does the agent know where it is in the task and whether it is making forward progress?). The central diagnostic contribution — the automated failure analyzer that identifies mid-task stagnation as the dominant failure mode across model scales and training paradigms — changes what it means to evaluate an agent. Success rate alone is no longer sufficient; failure mode distribution becomes a first-class evaluation axis. An agent that improves from 30% to 35% but still loops on 40% of failures is qualitatively different from one that shifts those failures from stagnation to wrong-termination, even if the aggregate number is identical. The paper provides both the diagnostic methodology (the three-function analyzer validated at 37/40 human agreement, Table 2) and the evidence that this distinction matters (Figures 3, 13; Tables 4), establishing a new standard for what a thorough agent evaluation should include.

The conceptual reframing is not that "subgoals help" — that is familiar from hierarchical RL, least-to-most prompting, and PRM literature. The reframing is that subgoals serve as a shared representation between planning and learning, and that their utility can be quantitatively validated as a calibrated progress signal rather than assumed. The paper's careful characterization — Exact Equivalence F1 of 0.68 (subgoals are not necessary-and-sufficient conditions), AUROC of 0.84 (but they are strong continuous progress indicators), monotonic calibration with Kendall's τ = 0.46 (completing more subgoals strictly increases success probability) — transforms subgoals from a heuristic decomposition into a measurement instrument. This shifts the research question from "how do we decompose tasks?" to "how do we produce progress estimates whose completion fraction reliably tracks true task progress?" — a more quantifiable and generalizable framing that licenses the use of subgoals as dense reward-shaping targets.

The paper also resolves a latent tension in the web agent literature between prompting-based approaches (which achieve reasonable zero-shot performance but lack error recovery) and RL-based approaches (which learn from outcomes but suffer from credit assignment sparsity). The dual-critic architecture demonstrates that dense reward shaping and sparse terminal optimization are not alternatives to be chosen between, but complementary mechanisms that can coexist — the potential critic densifies credit assignment while the value critic preserves the true optimization target. The calibration of the shaping factor (α = 0.3 optimal, α = 0.8 catastrophic, Table 7) provides an empirical demonstration of the classical reward-shaping tension between guidance and distortion, and the separation-of-concerns design (potential critic trained on progress labels, value critic trained on terminal rewards, neither having authority over the other's objective) offers a template for safe reward shaping with learned potential functions that generalizes beyond web navigation.

Several research directions become more attractive as a result. Investing in better progress estimators (subgoal generators, learned potential functions) now has a clear evaluation framework: AUROC against final success, monotonicity of P(success | m), and Exact Equivalence F1. Combining inference-time monitoring with training-time shaping becomes an obvious next step — the paper demonstrates each independently (SGO: +9.1% for proprietary models; MiRA: +12.1% over SFT for open models) and the complementarity argument (Section 6.2.1) is compelling, even though the combined experiment is left undone. Failure-mode-aware evaluation — reporting not just success rate but the distribution across stagnation, wrong-termination, and fail-attempt categories — becomes a standard that future web agent papers should meet.

Conversely, some directions become less attractive. Purely sparse-reward RL for web agents is shown to plateau at ~35% (Figure 10a, baseline RL curve), with the potential critic providing an additional ~8 percentage points that sparse methods cannot recover through curriculum learning alone. The ablation (Figure 11b, "MiRA w/o PC") provides direct evidence that dense shaping addresses a bottleneck that curriculum design does not. Latent subgoal methods that lack semantic interpretability (VSC-RL, HIQL) face a steeper burden of proof: the paper argues convincingly that explicit, verifiable milestones are essential for progress monitoring (Section 2.3), and the subgoal validation methodology (AUROC, monotonicity) cannot be applied to latent representations. Learned PRMs that produce soft scalar progress estimates without hard verifiability must now contend with the paper's demonstration that hard subgoal checkpoints, when validated as calibrated progress signals, provide reliable shaping without the over-optimization risks that plague learned reward models. The modest shaping factor (α = 0.3) and the restriction to positive trajectories are explicit design choices that prevent the reward hacking PRMs are vulnerable to, and the paper's ablation on α (Table 7) shows exactly what happens when shaping dominates: performance collapses.

Follow-Up Research This Work Enables

Validation of the failure analyzer on other benchmarks and domains. The automated failure analyzer is validated on 40 manually labeled WebArena-Lite trajectories (Table 2), but its categories and hardcoded rules (Table 1) may be specific to web navigation patterns. A strong follow-up would apply the same three-function analyzer (outcome summarization → rule-based categorization → differential key-step identification) to trajectories from Mind2Web, WebShop, VisualWebArena, or non-web agent benchmarks (OSWorld, AndroidWorld). The key measurement would be inter-annotator agreement with human labels on each new benchmark — does the 37/40 accuracy transfer, or do the rules need domain-specific adaptation? A negative result (e.g., "stuck midway" is not the dominant failure mode in mobile device control) would refine our understanding of whether stagnation is a universal long-horizon agent pathology or specific to web navigation's particular combination of open-ended pages and sparse feedback.

Combined MiRA + SGO experiment on an open-weight model. The paper demonstrates inference-time milestoning (SGO) on proprietary Gemini and training-time shaping (MiRA) on open-weight Gemma, but never combines them. A direct follow-up would fine-tune an open-weight model (Gemma-3-12B or Llama-3.1-8B) with MiRA, then deploy it with the dynamic milestoning self-reflection loop at inference time. The critical measurement would be whether the gains are additive (MiRA baseline ~43% + SGO-like improvement), sub-additive (the MiRA-trained policy already internalizes subgoal awareness, making runtime checking redundant), or super-additive (runtime checking catches distribution-shift errors that training cannot anticipate). The per-domain breakdown in Table 3 suggests the answer may be domain-dependent: Map tasks show no MiRA improvement (30.8% for both MiRA and WebRL), so SGO inference enhancement might be the only path to gains on spatially-oriented domains; conversely, Reddit tasks already reach 73.7% with MiRA alone, suggesting a ceiling effect where inference-time checking adds latency without accuracy.

Teacher model distillation for subgoal generation and progress labeling. The MiRA pipeline depends on Gemini-2.5-pro for subgoal generation (Section 5.1) and per-timestep progress labeling (Section 5.3), creating a reproducibility barrier and a cost bottleneck that the paper does not quantify. A natural follow-up would train a smaller, open-weight model (e.g., Gemma-3-12B itself, or Llama-3.1-8B) to perform these functions by distilling Gemini-2.5-pro's outputs. The training data already exists: the subgoals generated for WebArena-Lite's 165 tasks and the progress labels for 1,237 training tasks. A distilled subgoal generator could be evaluated by comparing its subgoal quality metrics (AUROC, monotonicity, Exact Equivalence F1) against the teacher's, and a distilled progress labeler by comparing potential critic performance when trained on distilled vs. teacher labels. The follow-up would also measure the cost reduction: how many GPU-hours of Gemma inference replace one Gemini-2.5-pro API call? A negative result — the distilled models produce substantially worse subgoals (AUROC drops from 0.84 to 0.70, shaping becomes less effective) — would indicate that subgoal quality is a bottleneck that current open-weight models cannot yet match, clarifying the capability frontier.

Wrong-termination decomposition and verification-specific training. The paper documents a clear trade-off: MiRA reduces stagnation from ~33% to ~21% but increases wrong terminations from ~20% to ~31% (Figure 13). The shift is framed as progress (the agent now reaches terminal states) but leaves unanswered what kind of wrong terminations occur. A strong diagnostic follow-up would manually categorize a representative sample of MiRA wrong-termination trajectories into sub-types: correct page but wrong extracted value, correct page but premature termination before final verification, wrong page that is semantically close to the target, completely unrelated wrong page. The distribution across these sub-types would determine the appropriate intervention: if most wrong terminations are "correct page, wrong value," the bottleneck is information extraction, not navigation, and a verification-specific training objective (e.g., training a separate answer-verification critic, or adding a post-termination self-check step) would be the natural extension. If most are "wrong but semantically close page," the bottleneck is the final decision step, and a confidence-threshold mechanism (only terminate when the value critic's success probability exceeds some threshold, otherwise continue) would be more appropriate. The paper's AutoRater (LLM-as-Judge) is flagged as potentially contributory (Section 6.2.5: "often failing to verify the semantic details of the final answer"), so comparing human evaluation of termination correctness against the AutoRater's judgments would also clarify whether the wrong-termination rate is partially an evaluation artifact.

Generalization of the dual-critic shaping design to non-web agent domains. The MiRA architecture — potential critic trained on interpolated progress labels from positive trajectories, value critic trained on terminal rewards, shaping via PBRS with calibrated α — is domain-agnostic in principle. The web-specific components are the subgoal generator (multimodal Gemini with curated few-shot examples per website) and the state representation (action history + HTML). A generalization experiment would apply the same dual-critic design to a non-web agent domain with a different observation modality: mobile device control (AndroidWorld, using screenshot-based state representations), OS automation (OSWorld, using accessibility trees), or even non-GUI sequential decision tasks with natural subgoal structure (e.g., multi-step retrieval-augmented QA, where subgoals are "identify relevant documents," "extract candidate answers," "synthesize final response"). The key measurement would be whether the dual-critic architecture provides gains over sparse-reward baselines in these new domains, and whether the shaping factor α requires re-calibration or transfers at 0.3. A negative result — the potential critic provides no benefit in a domain where "progress" is harder to define or interpolate — would identify the boundary conditions for when dense progress shaping is useful.

Cold-start exploration mechanisms for tasks where the first milestone is unreachable. The paper explicitly acknowledges (Section 7) that the potential critic provides no signal when the agent "fails to ground the initial subgoals due to extreme exploration difficulty or perception errors," and the subgoal completion dynamics (Figure 12, Phase 0) show early policies trapped at the first 1-2 subgoals. A follow-up could address this by training a separate exploration potential on all trajectories (not just positive ones), using a different shaping objective: reward the agent for reaching states that are novel relative to the training distribution, or for increasing the entropy of the subgoal completion pattern. Alternatively, a hindsight relabeling approach could treat partial subgoal completions in failed trajectories as success signals for a modified goal (e.g., "navigate to the search results page" even if the agent never selects the correct result). The measurement would be whether the Phase 0 → Phase 1 jump in success rate (Figure 10a) can be accelerated, and whether the subgoal completion dynamics (Figure 12) show faster transition from the stagnant vertical band to the diagonal gradient. A negative result — exploration bonuses or hindsight relabeling degrade the potential critic's calibration — would clarify that the positive-trajectory restriction is not just a statistical convenience but a necessary condition for progress signal reliability.

Practical Applications and Downstream Use Cases

Enterprise web automation with open-weight models. The Gemma-3-12B + MiRA result (43.0% on WebArena-Lite, surpassing GPT-4-Turbo at 17.6%) suggests a deployment architecture where an organization fine-tunes a modest open-weight model on their internal web applications (expense reporting, CRM workflows, procurement systems) using the MiRA pipeline, rather than paying per-query API costs to proprietary models. The practical workflow would be: (1) collect a set of task instructions for the target web application, (2) use a capable teacher model to generate subgoals for each task type (one-time cost), (3) collect exploratory rollouts and annotate progress labels, (4) train the potential critic and run the MiRA curriculum loop, (5) deploy the fine-tuned model with no inference-time overhead beyond standard LLM inference. The key economic question — whether the one-time labeling cost amortizes favorably against ongoing API costs — depends on query volume and task diversity, which the paper does not estimate. But the technical feasibility is demonstrated: a 12B-parameter model, after MiRA training, outperforms zero-shot GPT-4-Turbo by 25.4 percentage points on a web navigation benchmark, with an inference cost that is a tiny fraction of GPT-4's per-token pricing.

Self-improving agent pipelines with automated failure-driven curriculum generation. The outer curriculum loop (Section 5.4, Algorithm 2) — where failure analysis drives task resampling for the next training phase — is a self-contained mechanism that could be integrated into existing agent training pipelines independent of the MiRA-specific critic architecture. A team using WebRL, DigiRL, or a custom RL agent could adopt the failure analyzer (Section 4) + semantic similarity resampling (Appendix A.3) as a curriculum generation module, even without implementing the potential critic or the MSE-on-log-ratios objective. The concrete benefit is the demonstrated ability to prevent performance saturation: the baseline RL model without subgoal modules plateaus at ~35% (Figure 10a), while the curriculum structure alone ensures continued improvement across phases. The failure distribution shift analysis (Figure 13) provides a diagnostic dashboard that a production team could monitor during training: if stagnation errors stop decreasing while wrong terminations rise, the curriculum has successfully addressed the planning bottleneck and effort should shift to verification mechanisms.

Targeted remediation of "stuck midway" failures in existing deployed agents. The automated failure analyzer is a lightweight diagnostic tool that does not require any model modification to use. A team with a deployed web agent (proprietary or open-weight, any architecture) could run the analyzer on a sample of production trajectories to determine whether stagnation is their dominant failure mode. If it is (as the paper shows for every tested model, 42–49% of failures), they have a clear menu of interventions ordered by implementation complexity: (1) simplest — add the dynamic milestoning self-reflection loop (Section 5.2) at inference time if using a capable reasoning model (immediate ~9% absolute gain on Gemini-class models, Table 3); (2) moderate — implement the potential critic with progress labeling on their own task distribution using a teacher model (requires training data collection but no architecture changes to the deployed agent); (3) most intensive — full MiRA fine-tuning with curriculum loop (requires RL training infrastructure but provides the largest gains, +12.1% over SFT). The paper provides the diagnostic to determine whether stagnation is the problem, and the interventions to address it at different levels of resource commitment. This is a more actionable deployment roadmap than typical agent papers that present a single monolithic solution.

When to Prefer This Method

The paper does not present MiRA as part of a explicit tradeoff matrix against named alternatives with clearly articulated decision boundaries. It demonstrates that MiRA outperforms WebRL (+7.9%), DigiRL (+9.7%), and SFT (+12.1%) on the WebArena-Lite benchmark with Gemma-3-12B, but it does not characterize when a practitioner should choose MiRA over these alternatives in terms of task characteristics, data availability, or compute budgets. The Map domain result — MiRA ties WebRL at 30.8% — hints that the advantage is domain-dependent, but the paper does not develop this into a decision rule. Similarly, the paper demonstrates that SGO is effective for proprietary models that cannot be fine-tuned (+9.1% for Gemini-2.5-pro) but does not provide guidance on whether SGO or MiRA (or both) should be preferred when both are feasible. The question of when to invest in better reward shaping vs. better inference-time planning vs. both is raised by the paper's architecture but not resolved by its experiments, since the combined system is not evaluated. A practitioner should therefore view the reported gains as evidence that both mechanisms are individually effective, but should not interpret the paper as providing a decision procedure for choosing between them.