Agents2026~104 minarXiv:2602.22638

MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios

score 3Zhiheng Song, Jingshuai Zhang, Chuan Qin +6alibaba-inc↑ 103★ 112Feb 26, 2026cs.AIhttps://github.com/AMAP-ML/MobilityBench

🎯 Pitch

LLM route-planning agents excel at basic queries but fail on personalized preference constraints—except when you let them think, which makes them too expensive to deploy. MobilityBench reveals this gap with reproducible, sandboxed evaluations on 12,000 real user requests from Amap.

1. Executive Summary

This paper introduces MobilityBench, a scalable benchmark for evaluating LLM-based route-planning agents in real-world mobility scenarios, constructed from large-scale, anonymized real user queries collected from Amap and spanning over 350 cities worldwide. To enable reproducible evaluation, the benchmark employs a deterministic API-replay sandbox that eliminates environmental variance from live mapping services by caching and replaying API responses through a standardized interface. The paper proposes a multi-dimensional evaluation protocol that decomposes agent behavior into four core capabilities—instruction understanding (assessing intent detection and information extraction accuracy), planning (measuring task decomposition precision and recall), tool use (evaluating tool selection, schema compliance, and parameter filling), and decision making (quantifying delivery rate and final pass rate)—complemented by efficiency metrics on token consumption. Across multiple LLM-based agents evaluated under both ReAct and Plan-and-Execute frameworks, the strongest performer, Gemini-3-Pro-Preview, achieved a Final Pass Rate of 69.09% under ReAct, while open-source model Qwen3-235B-A22B reached 66.69%, establishing that current models perform competently on Basic Information Retrieval and Basic Route Planning tasks but struggle considerably with Preference-Constrained Route Planning, with the gap between open-source and closed-source models narrowing substantially—yet thinking-mode inference improves performance at the cost of markedly higher token overhead, making real-time deployment challenging.

2. Context and Motivation

The Core Problem: We Cannot Systematically Evaluate Real-World Route-Planning Agents

The fundamental gap this paper addresses is deceptively simple: we have no reliable way to measure whether an LLM-based agent can successfully plan a route in the real world. While this may sound like a niche concern, it sits at the intersection of two rapidly maturing fields — tool-augmented language agents and urban computing — and the absence of rigorous evaluation threatens to stall progress in both.

The challenge is not that route-planning agents don't exist. They do, and they are already being deployed. The challenge is that existing approaches to evaluating them are systematically inadequate for real-world mobility scenarios, and this inadequacy stems from four interconnected problems that the paper identifies explicitly in Section 1.

Why This Problem Matters: The Stakes of Mobility AI

Route planning is not merely an academic exercise in graph search. It is a daily necessity for hundreds of millions of people, and the consequences of failures range from inconvenience (arriving late) to safety hazards (distracted driving corrections) to systemic inefficiency (congestion from suboptimal routing). As the paper notes in Section 1, real-world mobility requests "extend far beyond simple point-to-point navigation, often involving multiple, interacting constraints, such as user preferences (e.g., avoiding highways or minimizing transfers), ordered waypoints, modality-dependent conditions, and time-sensitive requirements."

The shift toward voice-based interaction in mobility contexts — where "safety and convenience constraints limit users' ability to interact with mobile devices" (Section 3.1.2) — makes natural language the primary input modality for expressing mobility intent. This means that the agent must correctly interpret ambiguous, conversational, and sometimes underspecified queries without the benefit of a structured form or GUI. A benchmark that cannot faithfully assess whether an agent correctly handles "avoid tolls but still get me there by 3pm" is a benchmark that cannot tell us whether the agent is safe to deploy.

Moreover, the paper positions route-planning agents as a particularly challenging application domain that serves as a stress test for tool-augmented LLMs more broadly. Unlike web navigation or code generation, route planning requires tight integration between semantic intent understanding and low-level optimization over real road networks — a combination that exposes weaknesses in both the language modeling and the planning capabilities of current systems.

Prior Evaluation Approaches and Their Systematic Shortcomings

The paper identifies four distinct failure modes in existing evaluation paradigms, each of which independently undermines the validity of performance measurements for real-world route-planning agents.

Failure Mode 1: Benchmarks Test the Wrong Abstraction Level

Existing travel-related benchmarks, most notably TravelPlanner (Xie et al., 2024) and TravelBench (Cheng et al., 2025), evaluate agents at the itinerary level — constructing multi-day travel plans with flights, hotels, and restaurant reservations. As the paper states in Section 2.2:

"existing benchmarks primarily focus on high-level itinerary generation and abstract constraint satisfaction, and do not systematically evaluate agents' ability to perform fine-grained route planning under mobility-specific constraints, such as preference-aware routing (e.g., avoiding highways or minimizing transfers), ordered waypoint requirements, modality-dependent conditions, and time-sensitive constraints."

This distinction matters because the types of reasoning involved are fundamentally different. Planning a vacation itinerary involves selecting among discrete options (which hotel, which flight) with relatively coarse temporal granularity (days, hours). Route planning requires continuous spatial reasoning over road networks with minute-level temporal precision, where the feasibility of a constraint (e.g., "via Yinji Mall, arrive by 5pm") depends on real-time traffic conditions, road closures, and the interaction between multiple simultaneous constraints. An agent that passes an itinerary benchmark may fail catastrophically when asked to plan a multi-stop driving route with preference constraints — but existing benchmarks provide no way to detect this.

The paper's taxonomy of 11 task scenarios organized into four intent families (Table 1) makes this gap concrete. Tasks like Route-Constrained Planning ("Route to Shanghai Disneyland via People's Square, avoid Inner Ring Elevated Road") require the agent to simultaneously reason about required waypoints and excluded roads while computing a path — a constraint-satisfaction problem that has no analog in itinerary planning benchmarks.

Failure Mode 2: Live Mapping APIs Undermine Reproducibility

This is perhaps the most subtle but practically consequential problem the paper identifies. In Section 1, the authors list "non-determinism of live mapping APIs" as a fundamental challenge: API responses "vary over time due to traffic dynamics, service availability, and backend updates, thereby undermining reproducibility and fair comparison."

This is not a hypothetical concern. Consider two research groups evaluating the same agent on the same query: "Drive from Tiananmen Square to Capital International Airport." Researcher A runs the evaluation at 2am on a Tuesday, when roads are empty and the API returns a travel time of 35 minutes. Researcher B runs the same evaluation at 5pm on a Friday, when traffic is gridlocked and the API returns 90 minutes. If the agent's response differs between these two runs, is it because the agent's behavior changed, or because the world changed? Without controlling for environmental variance, it is impossible to tell.

The problem compounds when we consider that many state-of-the-art agents — particularly those using the ReAct framework (Yao et al., 2022b) — condition their subsequent actions on API responses. A different travel-time estimate at step 2 may cause the agent to take a different action at step 3, producing a completely different trajectory. The result is that what appears to be an improvement or regression in agent performance could simply be a consequence of evaluating under different traffic conditions.

Prior benchmarks either ignore this problem (relying on live APIs and accepting the resulting noise) or avoid it by using static, simplified environments that do not reflect real-world complexity. Neither approach produces reliable comparisons.

Failure Mode 3: Evaluation Metrics Are Too Coarse or Too Subjective

The paper identifies a gap between two existing evaluation paradigms, both of which are inadequate for route planning. On one hand, many agent benchmarks rely on end-to-end success rates — did the agent produce the right final answer? — which "treat agent behavior as a black box and obscure the intermediate failures along the decision-making chain" (Section 3.2). Knowing that an agent failed tells you nothing about whether it misunderstood the intent, selected the wrong tool, filled in parameters incorrectly, or generated a valid plan that violated a constraint.

On the other hand, LLM-as-a-judge approaches (Zheng et al., 2023) rely on subjective LLM-based evaluation of output quality, which the paper explicitly flags as insufficient for route planning: "effective assessment requires integrating multiple objective criteria beyond LLM-based subjective judging to verify API-call validity, constraint satisfaction, and factual grounding" (Section 1). A route either satisfies the user's constraints or it doesn't — there is no room for subjective assessment of whether an agent "mostly" avoided highways or "approximately" arrived on time.

The paper's proposed multi-dimensional evaluation protocol (Section 3.2) directly addresses this gap by decomposing evaluation into instruction understanding, planning, tool use, and decision making, each with fine-grained metrics. This allows researchers to answer questions like: "Did the agent fail because it couldn't understand the query, or because it understood correctly but selected the wrong API?" — diagnostic questions that end-to-end metrics cannot answer.

Failure Mode 4: Evaluation Toolkits Are Not Built for Rapid Iteration

The paper's fourth challenge — "extensible and reproducible evaluation toolkit" (Section 1) — may seem administrative but is in fact crucial for a field moving as fast as LLM-based agents. As model backbones and agent frameworks evolve, benchmarks that require complex setup, proprietary API access, or manual judgment become bottlenecks. The paper explicitly states that "rapid advances in LLM backbones and agent frameworks demand a lightweight, modular toolkit that supports easy deployment, scalable data expansion, and consistent evaluation across settings."

This requirement is not unique to MobilityBench, but the mobility domain amplifies it. Unlike text-only benchmarks where evaluation can be performed offline with static reference answers, route-planning evaluation requires simulating API interactions, validating spatial constraints, and checking temporal feasibility — all of which demand infrastructure that prior benchmarks do not provide.

How MobilityBench Positions Itself

The paper positions MobilityBench as addressing all four failure modes simultaneously through three design decisions, each of which maps directly to one or more of the identified gaps:

Design Decision 1: Real-user queries at the route-planning level of abstraction. By constructing the benchmark from "large-scale, anonymized real user queries collected from Amap" (Section 3.1.2) and organizing them into a taxonomy of 11 route-planning-specific task scenarios, MobilityBench fills the abstraction gap that TravelPlanner and TravelBench leave open. The benchmark covers not only basic navigation (point-to-point, multi-stop) but also preference-constrained routing, route-dependent information retrieval, and mobility-related information access — intents that reflect how people actually use navigation services rather than how researchers imagine they do.

Design Decision 2: Deterministic API-replay sandbox. This is the paper's primary technical contribution to reproducible evaluation. During dataset construction, all API responses are captured and cached. During evaluation, agents interact not with live services but with a replay sandbox that serves pre-recorded responses keyed by canonicalized arguments. As the paper states in Section 3.1.4, "this sandbox-based design ensures that measured performance faithfully reflects an agent's reasoning and tool-use capabilities rather than fluctuations in external systems." The sandbox includes fuzzy matching and nearest-neighbor spatial matching for cases where agent-provided arguments don't exactly match cache keys, with explicit failure handling for unresolvable calls — ensuring that evaluation is both deterministic and realistic.

This directly addresses Failure Mode 2 (API non-determinism) while enabling Failure Mode 4 (reproducible toolkit) by providing a stable, self-contained evaluation environment that requires no live API access.

Design Decision 3: Multi-dimensional evaluation with structured ground truth. Instead of relying on end-to-end pass rates or LLM judges, MobilityBench constructs "scenario-specific standard operating procedures (SOPs) defined by domain experts, which specify the minimal sequence of tool interactions required to correctly resolve a query" (Section 3.1.3). These SOPs are operationalized as executable programs, validated against historical data, and used to generate structured ground-truth annotations that capture not just the correct final answer but the expected intermediate steps — the tools that should be called, the parameters that should be filled, the constraints that should be satisfied.

This structured ground truth enables the evaluation protocol (Section 3.2) to diagnose failures at each stage of the agent's pipeline: Did the agent detect the right intent? Extract the right constraints? Decompose the task into the right sequence of subtasks? Select the right tools? Fill parameters correctly? Produce a valid final output? Each question gets a separate metric, addressing Failure Mode 3 (coarse evaluation).

The Larger Arc: From Research Artifact to Infrastructure

Beyond the immediate technical contributions, the paper frames MobilityBench as infrastructure for a research program. The benchmark is designed to be "easily extensible, enabling continuous expansion to new regions, scenarios, and intent types" (Section 1). The public release of benchmark data, evaluation toolkit, and documentation signals an intent to establish a shared evaluation standard — analogous to what GLUE/SuperGLUE did for natural language understanding or what HumanEval did for code generation, but for the domain of tool-augmented mobility agents.

The paper's experiments (Section 4) demonstrate the value of this infrastructure by revealing patterns that simpler benchmarks would miss: that open-source models are rapidly closing the gap with closed-source alternatives (Qwen3-235B-A22B achieving 66.69% FPR vs. Gemini-3-Pro-Preview's 69.09%), that ReAct outperforms Plan-and-Execute in final pass rate but at ~35% higher token cost, and that performance degrades sharply on preference-constrained tasks where even the strongest models struggle — findings that have direct implications for practitioners deciding between deployment architectures and model choices.

3. Technical Approach

3.1 Reader Orientation

MobilityBench is a benchmark and evaluation infrastructure — not a new agent or model, but a carefully engineered system for measuring how well LLM-based route-planning agents perform in real-world mobility scenarios. It solves the problem that existing evaluation methods cannot produce fair, reproducible, or diagnostically useful measurements of agent capability in this domain because live mapping services are inherently non-deterministic, prior benchmarks test the wrong abstraction level, and existing metrics are too coarse to reveal why an agent failed. The "shape" of the solution is a dataset of real user queries paired with a deterministic replay sandbox (which replays pre-recorded API responses instead of querying live services) and a multi-dimensional evaluation protocol that decomposes agent performance into separately measurable sub-capabilities (understanding, planning, tool use, decision making), each scored against structured ground truth.

3.2 Big-Picture Architecture (Diagram in Words)

The MobilityBench system has four major components that operate in sequence:

Benchmark Dataset (Episode Store) — a collection of 100,000 episodes, each containing an anonymized real user query, contextual information (location, city), a pre-recorded snapshot of relevant API responses, and structured ground-truth annotations. This is the input that defines what the agent is asked to do and what constitutes correct behavior.
Deterministic API-Replay Sandbox — an interception layer that sits between the agent under test and the external world. The agent issues API calls (e.g., routing, geocoding, weather), but instead of reaching live services, calls are resolved against a pre-recorded cache keyed by canonicalized arguments. This eliminates environmental variance (traffic, service updates) and ensures that identical agent inputs always yield identical, verifiable outputs.
Agent Under Test — an LLM-based route-planning agent (implemented under a chosen execution framework, such as ReAct or Plan-and-Execute) that receives a user query, reasons about the required steps, invokes tools through the sandbox, and produces a final output (e.g., a route with cost estimates).
Multi-Dimensional Evaluation Protocol — an automated scoring pipeline that compares the agent's full execution trace (predicted intents, extracted constraints, tool calls, parameter values, final output) against the structured ground truth. It produces scores for instruction understanding, planning, tool use, decision making, and efficiency — each decomposed into fine-grained sub-metrics.

Information flows as follows: an episode is selected from the benchmark dataset → the user query and context are passed to the agent → the agent generates a sequence of reasoning steps and tool calls → each tool call is intercepted by the replay sandbox, which returns the pre-recorded response → the agent produces a final output → the evaluation protocol scores every stage of the agent's behavior against the episode's ground truth.

3.3 Roadmap for the Deep Dive

First, the episode-centric data formulation (Section 3.1.1), which defines the fundamental unit of evaluation and the structured ground-truth representation that all downstream scoring depends on.
Second, the data collection and task taxonomy construction pipeline (Section 3.1.2), because understanding what the benchmark covers and how the coverage was designed explains why the evaluation is ecologically valid and what kinds of agent failures it can detect.
Third, the ground-truth construction procedure (Section 3.1.3), since the evaluation protocol's diagnostic power depends entirely on having structured, executable reference traces that specify not just the correct final answer but the intermediate tool interactions and constraints.
Fourth, the deterministic replay sandbox (Section 3.1.4), which is the enabling infrastructure for reproducible evaluation — understanding its caching, matching, and fallback mechanisms explains how the benchmark achieves determinism without sacrificing realism.
Fifth, the multi-dimensional evaluation protocol (Section 3.2), which defines each metric (intent detection, information extraction, task decomposition, tool selection, schema compliance, delivery rate, final pass rate, and efficiency) and the mathematical formulations used to compute scores.
Sixth, the agent execution frameworks (ReAct and Plan-and-Execute) referenced in the experiments, because understanding the architectural choices agents make under evaluation is necessary to interpret the performance results.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a benchmark design and infrastructure paper whose core idea is that evaluating real-world route-planning agents requires (1) tasks at the route-planning level of abstraction derived from real user queries, (2) a deterministic environment that replays API responses, and (3) a multi-dimensional evaluation protocol that diagnoses failures at each stage of the agent pipeline rather than reporting only end-to-end pass rates.

Episode-Centric Formulation

The fundamental unit of evaluation in MobilityBench is the episode — a self-contained mobility request that an agent can solve by invoking tools against the replay sandbox. Each episode is formally defined as a four-tuple:

$e = (x, z, \mathcal{S}, y)$

where $x$ is the anonymized natural-language user query (e.g., "Drive from Tiananmen Square to Capital International Airport"), $z$ encodes contextual information associated with the request such as the user's current location, the city, and other background variables relevant to mobility decision-making, $\mathcal{S}$ denotes a fixed and replayable snapshot of relevant API responses provided by the replay sandbox — effectively freezing traffic conditions, POI availability, and weather states at the time of data collection — and $y$ is a structured ground-truth annotation used exclusively for automated evaluation and never exposed to the agent.

What this four-tuple represents computationally: an episode is the input-output specification for a single evaluation trial. The agent receives $(x, z)$ (the query and context), interacts with the environment through tool calls that resolve against $\mathcal{S}$ (the cached API responses), and produces some final output. The evaluation protocol then compares the agent's behavior and output against $y$ (the structured ground truth) to compute scores.

Why this formulation: by bundling the query, context, environment snapshot, and reference answer into a single immutable unit, the episode formulation decouples evaluation from any external state. Two different agents evaluated on the same episode see exactly the same query, the same context, and the same API responses — regardless of when or where they are evaluated. This is what makes cross-model comparison fair: any difference in performance must be attributed to the agent's reasoning and tool-use capabilities, not to differences in traffic conditions or API behavior. The alternative — evaluating against live APIs — would confound agent capability with environmental noise, making it impossible to determine whether an apparent improvement is real or an artifact of running the evaluation at a different time.

A critical design constraint: "Throughout this work, the route-planning agents are not permitted to ask users for clarification. Consequently, all episodes are designed to be fully solvable based solely on the initial user query $x$ ." This constraint is enforced during data construction (Section 3.1.2) by removing underspecified or ambiguous queries. It ensures that every episode has a well-defined, objectively correct solution — there is no ambiguity about what the agent should have done, which is a prerequisite for automated scoring.

Data Collection and Task Taxonomy Construction

The benchmark is built from "large-scale, anonymized mobility queries collected from Amap over the past six months" (Section 3.1.2). Understanding the provenance and curation of this data is essential because it determines what kinds of agent behaviors the benchmark can and cannot evaluate.

Query source and modality. In real-world mobility scenarios — particularly while driving or walking — safety and convenience constraints limit users' ability to interact with mobile devices through typing. Voice is therefore "a natural and prevalent input modality for expressing mobility intent" (Section 3.1.2). The raw data consists of voice queries transcribed into text, which "provide direct and largely unconstrained expressions of real user intent, encompassing destination goals, situational information needs, and explicit preference constraints." This is significant because it means the queries reflect how people actually express mobility needs — with all the ambiguity, incompleteness, and conversational patterns that entails — rather than how benchmark designers imagine they might.

Multi-stage filtering and curation pipeline. From the raw query corpus, the authors apply a series of filters to produce the final dataset of 100,000 episodes. The key steps are:

Removal of malformed, underspecified, or ambiguous requests. Under the strict no-clarification assumption (agents cannot ask follow-up questions), any query that cannot be resolved unambiguously from the text alone is discarded. This includes queries missing critical information (e.g., a destination that cannot be identified from context) and queries where multiple reasonable interpretations exist that would lead to different correct answers.
Deduplication of near-identical queries. The authors remove queries that are semantically equivalent to ensure diversity in the benchmark. This prevents the evaluation from being dominated by a small number of high-frequency query patterns.
Reliability filtering during ground-truth construction. After constructing ground-truth annotations (Section 3.1.3), episodes whose answers "cannot be reliably obtained or verified via tool execution" are removed, retaining only "episodes with executable and checkable outcomes" (Section 3.1.5).

Task taxonomy construction through iterative open-set labeling. Rather than imposing a predefined set of task categories, the authors use an iterative, model-assisted, expert-adjudicated process to discover the taxonomy from the data. The specific procedure is:

Initialization with coarse-grained roots. The process begins with two high-level intent families: information access (covering POI, traffic, and weather lookup) and route planning (covering navigation to destinations).
Open-set labeling with Qwen-4B. The model is used to classify each curated query into existing intent categories. Crucially, queries that "cannot be aligned with existing labels prompt the model to propose new candidate intents along with concise definitions" (Section 3.1.2). This open-set mechanism allows the taxonomy to grow organically to capture long-tail and previously unobserved intents.
Iterative consolidation and expert adjudication. The candidate labels proposed by the model are "subsequently iteratively consolidated, merged, and refined through multiple rounds of expert adjudication, ensuring semantic clarity, mutual exclusivity, and comprehensive coverage of the intent space" (Section 3.1.2). Multiple rounds are necessary because the model's initial proposals may be overly granular (splitting what should be one category), overly coarse (lumping distinct intents), or semantically imprecise. Domain experts resolve these ambiguities.

The resulting taxonomy. The final classification hierarchy has two levels: 11 task scenarios organized into 4 intent families (Table 1):

Basic Information Retrieval (36.6% of episodes): POI Query, Geolocation Query, Nearby Query, Weather Query, Traffic Info Query. These are pure information-access tasks — the agent must invoke lookup APIs and return factual information.
Route-Dependent Information Retrieval (9.6%): Route Property Query (e.g., "How far is it from Hefei to Huangshan?") and Arrival/Departure Time Query (e.g., "If I drive from my home to Capital International Airport now, when will I arrive?"). These require computing a route as an intermediate step before answering the information need — the route itself is not the output, but it must be computed internally to produce the correct answer.
Basic Route Planning (42.5%): Point-to-Point Planning (single origin to single destination) and Multi-stop Planning (routing across multiple intermediate destinations with ordered waypoints). These are the standard navigation tasks where the agent must produce a complete, executable itinerary.
Preference-Constrained Route Planning (11.3%): Option-Constrained Route Planning (applying standardized routing options like minimizing tolls, avoiding highways, preferring fastest route, fewer transfers, or less walking) and Route-Constrained Planning (enforcing explicit path-level constraints such as required waypoints or excluded roads). These are the most challenging category because they require the agent to simultaneously reason about spatial feasibility and user preferences.

Why this taxonomy construction method matters. The iterative, open-set approach ensures that the benchmark covers the actual diversity of real-world mobility intents rather than the assumed diversity. It surfaces long-tail intents that a top-down designer might overlook. The expert adjudication step ensures that the resulting categories are both exhaustive (every query maps to exactly one scenario) and mutually exclusive (no query maps ambiguously to multiple scenarios), which is necessary for clean per-category evaluation.

Ground-Truth Construction

The evaluation protocol's ability to provide fine-grained diagnostic scores depends on having structured ground-truth annotations that go beyond a simple "correct final answer." MobilityBench constructs what it calls scenario-specific standard operating procedures (SOPs) , which are "defined by domain experts" and "specify the minimal sequence of tool interactions required to correctly resolve a query" (Section 3.1.3).

The standard tool program. For each of the 11 task scenarios, domain experts define a canonical workflow — the "standard tool program" — that operationalizes the SOP as a structured, executable sequence of tool calls. This program is not hand-crafted for each individual query; rather, it is a template parameterized by the slots extracted from the query. The workflow has three core steps:

Slot extraction and normalization. The program extracts structured information from the user query: points of interest (origins, destinations, waypoints), temporal constraints (departure times, arrival deadlines), travel modes (driving, walking, cycling, public transit), and user preferences (avoid highways, minimize transfers, prefer fastest route). These slots are normalized to canonical forms — for example, "Capital Airport" and "PEK" would both be normalized to a standard representation.
Location resolution. Textual place references are resolved into structured entities or geographic coordinates. This involves POI retrieval (searching for named places like "Yinji Mall") and geocoding (converting addresses to latitude/longitude coordinates). This step is critical because downstream routing APIs require coordinates, not place names.
Parameter validation and downstream tool invocation. After validating that all required parameters are present and within valid ranges, the program invokes the appropriate downstream tools — routing APIs for navigation, real-time traffic services for congestion information, weather services for forecasts — while verifying constraint feasibility when applicable. For example, if a user requests a route "via Yinji Mall" and "avoid Inner Ring Elevated Road," the program checks whether these constraints are mutually satisfiable.

Execution and validation. The standard tool program is executed within an existing agent framework (the paper does not specify which) to orchestrate the actual tool invocations. The resulting outputs are validated against historical data with reliability filtering. The "full execution trace together with key intermediate artifacts" is consolidated into a ground-truth archive. This archive contains not just the final answer (e.g., the route with its time and distance estimates) but also the intermediate evidence: which tools were called, with what parameters, and what each tool returned.

What the ground truth $y$ contains. For each episode, the structured ground-truth reference $y$ captures:

The correct intent label $y_{\mathrm{ID}}(x)$ (which of the 11 task scenarios the query belongs to).
The correct constraint set $y_{\mathrm{IE}}(x)$ — all explicit and implicit constraints extracted from the query, including spatial attributes (origins, destinations), temporal parameters (departure windows, duration limits), and preference signals (traffic avoidance, modality priorities).
The ground-truth action sequence $V_{gold}(x)$ — the ordered list of atomic actions that constitute the correct plan for solving the query.
The ground-truth tool set $T_{gold}(x)$ — the set of tools that should be invoked.

Why this form of ground truth is necessary. Standard evaluation approaches that compare only the agent's final answer against a reference answer cannot diagnose where a failure occurred. Did the agent misunderstand the intent? Extract the wrong constraints? Plan the wrong sequence of actions? Select the wrong tools? Fill parameters incorrectly? The structured ground truth enables the evaluation protocol (Section 3.2) to answer each of these questions independently by comparing the agent's intermediate outputs at each stage against the corresponding reference — a capability that is impossible if the ground truth contains only a final answer.

Deterministic Replay Sandbox

The replay sandbox is the infrastructure that makes reproducible evaluation possible. Without it, any comparison between agents would be confounded by the non-determinism of live mapping APIs. The sandbox serves as a drop-in replacement for live services: agents issue API calls as they normally would, but instead of reaching Amap's live endpoints, calls are intercepted and resolved against a pre-recorded cache.

Cache construction. During the ground-truth construction phase (Section 3.1.3), while the standard tool program is being executed, all API responses from the Amap Web Service API are captured and stored. This capture happens at a single point in time, effectively "freezing" the state of the world — traffic conditions, POI availability, weather, road closures — at the moment of data collection. The captured responses are indexed by canonicalized arguments: coordinates are normalized to a standard precision, time formats are standardized, and entity references are resolved to canonical identifiers. This canonicalization ensures that functionally equivalent queries produce the same cache key, avoiding cache misses due to superficial formatting differences.

Cache resolution during evaluation. When an agent under evaluation issues a tool call, the sandbox performs the following steps:

Schema validation. All tool invocations undergo strict validation, including "required-field checks and type and range constraints" (Section 3.1.4). If a call is malformed — missing a required parameter, providing a value of the wrong type, or providing a value outside the valid range — it is rejected as a tool-use failure and reflected in the evaluation metrics. This validation ensures that agents are penalized for producing syntactically invalid API calls, which is a realistic failure mode.
Exact cache lookup. The sandbox attempts to find an exact match for the canonicalized arguments in the pre-recorded cache. If found, the cached response is returned immediately.
Fuzzy matching fallback (entity-based queries). When an exact match is unavailable for entity-based queries (e.g., POI search with a slightly different name or spelling), the sandbox applies fuzzy matching to find the closest matching entity in the cache. The paper does not specify the exact fuzzy matching algorithm, but the intent is to tolerate minor variations in how agents specify entity names without causing false cache misses.
Nearest-neighbor spatial matching (coordinate-based queries). For coordinate-based queries where the agent specifies a location slightly different from what was cached, the sandbox applies "nearest-neighbor spatial matching" subject to "a maximum distance threshold" (Section 3.1.4). If the agent specifies a coordinate within the distance threshold of a cached coordinate, the cached response for the nearby location is returned. If no match is found within the threshold, the call is treated as unresolvable.
Failure handling. "Calls that fail validation or cannot be resolved are treated as tool-use failures and are explicitly reflected in the evaluation metrics" (Section 3.1.4). This design choice is important: rather than silently returning an error or a default response, the sandbox makes tool-use failures visible in the evaluation scores. This ensures that agents are penalized for producing unresolvable calls, incentivizing precise tool use.

Why determinism matters beyond reproducibility. The sandbox does more than enable fair comparison between agents evaluated at different times. It also enables fair comparison between different execution frameworks. The ReAct framework (Yao et al., 2022b) is an interleaved reasoning-and-acting loop: the agent observes a tool response, reasons about the next step, and acts again. If the same API call returns different responses on different invocations (as live services would), then two ReAct agents with identical policies could diverge in their action sequences purely due to environmental noise. The deterministic sandbox eliminates this confound, ensuring that measured differences between agents reflect genuine differences in capability rather than luck.

A subtle design implication: Because the sandbox returns exactly the same responses that were captured during ground-truth construction, the evaluation protocol can compare the agent's behavior not just against the final answer but against the expected trajectory. If the sandbox returned different (though still valid) responses, an agent could produce a different but equally correct sequence of actions — but the evaluation protocol's planning and tool-use metrics (Section 3.2.2 and 3.2.3) would incorrectly penalize this deviation. The deterministic sandbox ensures that there is a single correct trajectory (the one captured in the ground truth), against which the agent's trajectory can be compared without ambiguity.

Multi-Dimensional Evaluation Protocol: Overview

The evaluation protocol decomposes agent behavior into four core capabilities — Instruction Understanding, Planning, Tool Use, and Decision Making — plus an Efficiency dimension. For each capability, the protocol defines one or more fine-grained metrics with explicit mathematical formulations. The protocol is designed to be computed automatically from the structured ground truth and the agent's execution trace, requiring no human judgment or LLM-based scoring.

The metrics are organized as follows (detailed in the subsequent sub-sections):

Capability	Metrics	What It Measures
Instruction Understanding	Intent Detection (ID), Information Extraction (IE)	Did the agent understand what the user wanted and what constraints apply?
Planning	Task Decomposition (DEC-P, DEC-R)	Did the agent produce the right sequence of actions?
Tool Use	Tool Selection (TS-P, TS-R), Schema Compliance (SC)	Did the agent choose the right tools and call them correctly?
Decision Making	Delivery Rate (DR), Final Pass Rate (FPR)	Did the agent produce any output, and was it correct?
Efficiency	Input Tokens (IT), Output Tokens (OT)	How much computation did the agent consume?

Instruction Understanding: Intent Detection (ID)

Intent detection measures whether the agent correctly identifies the high-level purpose of the user's query from the 11 task scenario categories defined in the taxonomy.

$\mathrm{ID} = \frac{1}{|\mathcal{X}|} \sum_{x \in \mathcal{X}} \mathbb{I}\!\left(\mathrm{sim}\!\left(\hat{y}_{\mathrm{ID}}(x),\, y_{\mathrm{ID}}(x)\right) \geq \alpha_{\mathrm{threshold}}\right)$

where $\mathcal{X}$ is the set of all evaluation queries, $\hat{y}_{\mathrm{ID}}(x)$ is the agent's predicted intent label for query $x$ , $y_{\mathrm{ID}}(x)$ is the ground-truth intent label for query $x$ , $\mathrm{sim}(\cdot, \cdot)$ is a similarity function comparing two labels, $\alpha_{\mathrm{threshold}}$ is a predefined similarity threshold above which the prediction is considered correct, and $\mathbb{I}(\cdot)$ is the indicator function returning 1 when the condition holds and 0 otherwise.

What it computes: for each query, the agent explicitly outputs a predicted intent label (the paper states that "the agent is explicitly instructed to output a set of intent labels corresponding to the task scenario categories"). This prediction is compared to the ground-truth intent label via a similarity function. If the similarity exceeds the threshold, the prediction is counted as correct. The ID score is the fraction of queries for which the prediction is correct — a value between 0 and 1, with 1 meaning perfect intent detection.

Why this form: the similarity-based scoring rather than exact-match scoring acknowledges that agent-predicted labels may use different surface forms than the canonical ground-truth labels. For example, an agent might output "Point-to-Point Navigation" while the ground truth says "Point-to-Point Planning" — these are semantically equivalent and should not be penalized. The similarity function (the paper does not specify whether it is embedding-based, token-overlap-based, or something else) provides this robustness. The threshold $\alpha_{\mathrm{threshold}}$ controls the strictness: a threshold of 1.0 requires exact matches, while lower thresholds allow approximate matches. The paper does not report the specific threshold value used.

What this metric enables downstream: intent detection is the gateway to all subsequent reasoning. If the agent misclassifies a query as "Basic Information Retrieval" when it is actually "Preference-Constrained Route Planning," the agent will attempt to solve the wrong problem entirely — it might look up a POI when it should be computing a constrained route. The ID score quantifies how often this fundamental misunderstanding occurs, providing a diagnostic that end-to-end success rates cannot isolate.

Instruction Understanding: Information Extraction (IE)

Information extraction evaluates whether the agent correctly identifies all explicit and implicit constraints embedded in the user query — spatial attributes, temporal parameters, and preference signals.

$\mathrm{IE} = \frac{1}{|\mathcal{X}|} \sum_{x \in \mathcal{X}} \mathbb{I}\!\left(\hat{y}_{\mathrm{IE}}(x) = y_{\mathrm{IE}}(x)\right)$

where $\hat{y}_{\mathrm{IE}}(x)$ is the agent's predicted constraint set for query $x$ (the set of extracted spatial, temporal, and preference constraints) and $y_{\mathrm{IE}}(x)$ is the ground-truth constraint set.

What it computes: unlike intent detection, information extraction uses exact set equality — the predicted constraint set must match the ground-truth constraint set perfectly. A mismatch occurs if the agent extracts a constraint that doesn't exist (false positive), fails to extract a constraint that does exist (false negative), or extracts the right constraints but with incorrect values (e.g., extracting the wrong destination). The IE score is the fraction of queries where the extraction is perfectly correct — a very strict metric by design.

Why this strict form: constraints in route planning are not "approximately" satisfiable. If a user requests "avoid highways" and the agent fails to extract this preference, the resulting route will violate the user's explicit instruction, regardless of whether the route is otherwise optimal. There is no partial credit for getting some constraints right — missing even one constraint produces an incorrect final output. The exact-match formulation reflects this reality. Contrast this with intent detection, where similarity-based scoring is appropriate because intent labels are categorical descriptions that can be expressed in multiple ways; constraints are concrete requirements that must be extracted precisely.

What this metric reveals: information extraction failures are a common bottleneck in tool-augmented agents. An agent might correctly detect that the query is a route-planning request (ID correct) but fail to extract the "avoid tolls" constraint (IE incorrect). The resulting plan will be syntactically valid and tool-call-correct but substantively wrong — a failure mode that end-to-end metrics would detect but not attribute to the right cause. The separation of ID and IE enables developers to diagnose whether instruction understanding failures are due to misclassifying the task type or missing specific constraints.

Planning: Task Decomposition (DEC-P, DEC-R)

Task decomposition evaluates whether the agent can break down a high-level user goal into a coherent sequence of atomic actions. The metric is computed as two complementary scores — precision and recall — to separately measure over-planning and under-planning.

$\text{DEC-P} = \frac{1}{|\mathcal{X}|} \sum_{x \in \mathcal{X}} \frac{|V_{gold}(x) \cap_{f_{DEC}} V_{pred}(x)|}{|V_{gold}|}$

$\text{DEC-R} = \frac{1}{|\mathcal{X}|} \sum_{x \in \mathcal{X}} \frac{|V_{pred}(x) \cap_{f_{DEC}} V_{gold}(x)|}{|V_{pred}|}$

where $V_{pred}(x) = \{v_1, v_2, \ldots, v_n\}$ is the agent's predicted sequence of atomic actions for query $x$ , $V_{gold}(x)$ is the ground-truth sequence of atomic actions, and $A \cap_{f_{DEC}} B = \{a \in A \mid \exists b \in B, f_{DEC}(a, b) = \text{True}\}$ is the set of actions in $A$ that match some action in $B$ according to the matching function $f_{DEC}(\cdot, \cdot)$ . The matching function determines whether two atomic actions are considered equivalent — for example, two calls to the same tool with equivalent parameters would match even if their surface forms differ.

What DEC-P (precision) computes: for each query, the number of ground-truth actions that the agent correctly predicted, divided by the total number of ground-truth actions, averaged across all queries. A DEC-P of 1.0 means the agent's plan covers every step in the ground-truth plan — no omissions. A DEC-P below 1.0 indicates under-planning: the agent failed to include some necessary steps.

What DEC-R (recall) computes: for each query, the number of agent-predicted actions that match some ground-truth action, divided by the total number of agent-predicted actions, averaged across all queries. This score is capped at 1.0 (it cannot exceed 1.0 because the numerator is a subset of the denominator). A DEC-R below 1.0 indicates over-planning: the agent included steps that are not in the ground-truth plan — unnecessary or redundant actions.

Why both precision and recall are necessary: a single F1 score would obscure the nature of planning failures. An agent with high DEC-P but low DEC-R produces plans that are complete but verbose, containing irrelevant steps. An agent with low DEC-P but high DEC-R produces plans that are concise but incomplete, missing necessary steps. These are qualitatively different failure modes requiring different fixes — reducing over-planning might involve better pruning of unnecessary tool calls, while reducing under-planning might involve better coverage of implicit requirements. Reporting both metrics separately preserves this diagnostic information.

The matching function $f_{DEC}$ is a critical design element that the paper defines only by its property: two actions match if one exists in the other's corresponding set under the matching criteria. The exact implementation — whether it compares tool names, parameter values, or both, and with what tolerance — is not specified in the main paper. The function must balance strictness (preventing spurious matches that inflate scores) with robustness (allowing for equivalent actions expressed in different forms).

Tool Use: Tool Selection (TS-P, TS-R)

Tool selection evaluates whether the agent correctly identifies which tools to invoke from the available tool set $\mathcal{T}$ . Like task decomposition, it uses precision and recall to separately measure different failure modes.

$\text{TS-P} = \frac{1}{|\mathcal{X}|} \sum_{x \in \mathcal{X}} \frac{|T_{gold}(x) \cap T_{pred}(x)|}{|T_{gold}|}$

$\text{TS-R} = 1 - \frac{1}{|\mathcal{X}|} \sum_{x \in \mathcal{X}} \frac{|T_{pred}(x) \setminus T_{gold}(x)|}{|T_{pred}|}$

where $T_{pred}(x)$ is the set of tools selected by the agent for query $x$ and $T_{gold}(x)$ is the ground-truth set of required tools.

What TS-P (precision / coverage) computes: the fraction of required tools that the agent correctly selected — "coverage reflects whether all necessary tools are selected" (Section 3.2.3). A TS-P of 1.0 means the agent selected every tool that was needed. A TS-P below 1.0 means the agent missed some required tools.

What TS-R (recall / 1-redundancy) computes: one minus the fraction of selected tools that are unnecessary — "redundancy penalizes unnecessary tool calls, (for easier comparison, we report redundancy as its complement, $1 - \text{redundancy}$ )" (Section 3.2.3). A TS-R of 1.0 means the agent selected no unnecessary tools. A TS-R below 1.0 means the agent invoked tools that were not required — either tools irrelevant to the task or redundant calls to the same tool.

Why this complement form for TS-R: the raw redundancy metric ( $|T_{pred} \setminus T_{gold}| / |T_{pred}|$ ) is a penalty — it increases when the agent makes unnecessary calls, producing a score where lower is better. By reporting the complement ( $1 - \text{redundancy}$ ), the metric is inverted so that higher is better, aligning with all other metrics in the evaluation protocol. This is purely a presentational choice to make score tables more interpretable.

A subtle interaction with the sandbox: unnecessary tool calls are not just wasteful — they can actively harm performance if they return misleading information that causes the agent to change its plan. In the replay sandbox, an unnecessary tool call might return a valid but irrelevant response that the agent then incorrectly incorporates into its reasoning. TS-R thus captures both efficiency (avoiding wasted computation) and robustness (avoiding self-sabotage through irrelevant information gathering).

Tool Use: Schema Compliance (SC)

Schema compliance evaluates whether the agent's tool invocations conform to the predefined API specifications — that all mandatory parameters are provided and their values fall within valid formats and ranges.

$\mathrm{SC} = \frac{1}{|\mathcal{X}|} \sum_{x \in \mathcal{X}} \frac{1}{\left|ST_{pred}(x)\right|} \sum_{t \in ST_{pred}(x)} f_{SC}\!\left(P(t),\, t\right)$

where $ST_{pred}(x)$ is the sequence of tool invocations produced by the agent for query $x$ , $P(t)$ is the set of parameters associated with tool call $t$ , and $f_{SC}(P(t), t)$ is an indicator function that returns 1 if the parameters for tool $t$ conform to the predefined valid formats and ranges, and 0 otherwise.

What it computes: for each query, the metric iterates over every tool call the agent made. For each call, it checks whether the provided parameters satisfy the schema requirements — all mandatory fields present, all values of the correct type, all values within valid ranges. The fraction of compliant calls is computed per query, then averaged across all queries.

What constitutes a schema violation: the paper identifies three types of violations explicitly: missing mandatory parameters (the agent calls a routing API without specifying a destination), invalid parameter types (the agent provides a string where a coordinate pair is expected), and out-of-range values (the agent specifies a departure time in the past). Any of these violations in a single parameter of a single tool call renders that entire call non-compliant.

Why schema compliance is separated from tool selection: an agent might select exactly the right tools (perfect TS-P and TS-R) but call them incorrectly — omitting required parameters or providing ill-typed values. This is a distinct failure mode that requires a distinct diagnostic. An agent with high TS but low SC understands what to do but cannot correctly express the tool interaction — a problem that might be addressed by better prompting, function-calling fine-tuning, or improved API documentation in the system prompt.

Decision Making: Delivery Rate (DR) and Final Pass Rate (FPR)

The decision-making dimension evaluates whether the agent can actually produce a valid final output and whether that output is correct.

Delivery Rate (DR): "measures the proportion of queries for which an agent successfully generates a complete and executable final output (e.g., a full itinerary) without interruption or tool invocation failure" (Section 3.2.4). Although the paper does not provide an explicit equation, the computation is implicit: $\mathrm{DR} = \frac{1}{|\mathcal{X}|} \sum_{x \in \mathcal{X}} \mathbb{I}(\text{agent produced complete output for }x)$ .

An agent fails to deliver if it encounters an unrecoverable error — a tool call that fails validation in the sandbox with no fallback, a reasoning loop that exceeds the maximum number of steps, or a generation that terminates without producing a structured output. The DR quantifies robustness: what fraction of the time does the agent even finish the task?

Final Pass Rate (FPR): "evaluates the effectiveness of the generated solution. A solution is considered successful only if it satisfies all user-specified explicit and implicit constraints" (Section 3.2.4). Again, the paper does not provide an explicit equation, but the computation is: $\mathrm{FPR} = \frac{1}{|\mathcal{X}|} \sum_{x \in \mathcal{X}} \mathbb{I}(\text{agent output satisfies all constraints for }x)$ .

The relationship between DR and FPR: FPR is always less than or equal to DR, because an agent that fails to deliver cannot possibly pass. The gap between DR and FPR represents cases where the agent produced an output but that output was wrong — a valid trajectory that led to an incorrect answer. This gap is a critical diagnostic: a large DR-FPR gap indicates that the agent is consistently completing tasks but getting the answer wrong (perhaps due to constraint extraction failures or reasoning errors), while a small gap could indicate either high accuracy or that the agent fails early (low DR) on the tasks it cannot solve.

Why both metrics are needed: DR measures robustness; FPR measures correctness. An agent could have high FPR among the subset of tasks it completes but a low DR overall (it's accurate when it works but frequently crashes). Conversely, an agent could have a high DR but a low FPR (it always produces something but often gets it wrong). These profiles suggest different deployment strategies and improvement priorities.

Efficiency: Input Tokens (IT) and Output Tokens (OT)

Efficiency metrics quantify the computational overhead of agent execution, which directly translates to API cost and wall-clock latency in deployment.

Input Tokens (IT): "measures the cumulative volume of contextual information processed by the model, including system prompts, task instructions, and the historical trajectory of observations and actions" (Section 3.2.5). This metric captures the total number of tokens in the model's input context across all steps of the agent's execution — the system prompt, the user query, and every observation (tool response) and action (generated tool call or reasoning step) accumulated in the context history.

Output Tokens (OT): "quantifies the total number of tokens generated by the model" (Section 3.2.5). This includes both reasoning tokens (the model's chain-of-thought or planning text) and action tokens (the structured tool calls the model generates).

Why IT is particularly important for agent frameworks: the ReAct framework (Yao et al., 2022b) accumulates the full history of observations and actions in the context window. Each tool invocation appends the tool response to the context, which then becomes part of the input for the next reasoning step. This means IT grows linearly with the number of interaction steps — an agent that takes 20 steps consumes roughly 20 times the input tokens of a single-call agent, even if the final output is identical. The paper reports that "ReAct's average IT is approximately 35.38% higher than Plan-and-Execute's" (Section 4), directly quantifying the context-accumulation cost of the interleaved reasoning-loop architecture.

What IT and OT jointly reveal: a model with high FPR but extremely high IT and OT may be practically undeployable in real-time mobility scenarios where latency matters. The paper's thinking-mode experiments (Figure 4) demonstrate this tension explicitly: enabling thinking improves FPR (e.g., Qwen-30B-A3B improves by 5.98% absolute) but "substantially increases the generated token volume, leading to markedly higher inference cost and latency" — making deployment "challenging in real-time, production-grade online settings."

Agent Execution Frameworks: ReAct and Plan-and-Execute

The experiments (Section 4) evaluate agents under two distinct execution frameworks, which the paper treats as an independent variable to study. Understanding these frameworks is necessary to interpret the performance results.

ReAct (Yao et al., 2022b). This is an interleaved reasoning-and-acting framework operating as a closed-loop "think-act-observe" cycle. The agent receives the user query and generates a reasoning step (thinking about what to do), then generates a tool call (acting), then receives the tool response from the sandbox (observing), then reasons again based on the new information, and so on. The key property is that each step conditions on the full history of previous observations, enabling the agent to dynamically adjust its strategy based on real-time tool responses. The paper characterizes this as a "closed-loop" mechanism where the agent "can dynamically adjust its strategy based on real-time results returned by tools."

Plan-and-Execute (Wang et al., 2023). This framework separates planning from execution. The agent first generates a complete plan — a sequence of tool calls to make — in a single reasoning step, then executes all the calls, then produces the final answer. Tool responses are not fed back into the reasoning loop during execution; the plan is static once generated. The paper characterizes this as "static pre-planning" that "shows a significant lack of robustness when facing dynamic feedback in mobile scenarios."

The tradeoff between the two frameworks is a central empirical finding of the paper (Section 4). ReAct achieves higher FPR because its closed-loop design allows it to recover from errors — if a tool call returns unexpected results, the agent can observe this and adjust its plan. Plan-and-Execute, by contrast, cannot recover from plan errors because it never re-evaluates its plan based on intermediate results. However, ReAct's robustness "comes at a non-trivial computational cost" — each observation is appended to the context, causing IT to grow with every step and making inference more expensive and slower. The paper quantifies this as a 35.38% average IT increase for ReAct over Plan-and-Execute across all models.

Why this framework comparison matters: it reveals that the choice of agent architecture is not purely about capability — it is a capability-efficiency tradeoff with direct implications for deployment. In real-time mobility applications where latency constraints are tight, the Plan-and-Execute framework's lower token overhead may be preferable even if its FPR is slightly lower. The benchmark's efficiency metrics (IT and OT) enable this tradeoff to be evaluated quantitatively, which end-to-end success-rate-only benchmarks would miss.

4. Key Insights and Innovations

Innovation 1: Deterministic Replay as a First-Class Evaluation Primitive

The paper's most fundamental conceptual move is not any of the metrics or the dataset itself, but rather the elevation of deterministic replay from an implementation detail to a benchmark-design first principle. This reframes the evaluation problem for tool-augmented agents in a way that has implications far beyond route planning.

What the field did before. Prior agent benchmarks in tool-augmented domains — including ToolBench (Qin et al., 2023), τ-bench (Yao et al., 2024), TravelPlanner (Xie et al., 2024), and TravelBench (Cheng et al., 2025) — operated under one of two regimes. Either they queried live APIs, accepting that environmental variance (traffic, service updates, rate limits) would introduce noise into evaluation scores, making cross-model comparisons unreliable. Or they used simplified, static environments that sacrificed ecological validity for reproducibility — agents were tested in toy sandboxes that failed to capture the complexity of real-world service interactions. The dominant assumption was that reproducibility and realism were fundamentally in tension: you could have one or the other, but not both. The field implicitly accepted this tradeoff, with most work erring on the side of realism (live APIs) and treating non-reproducibility as an unavoidable artifact.

What's distinctive at the idea level. MobilityBench treats determinism not as a constraint to work around but as a property to engineer into the evaluation infrastructure itself. The replay sandbox is a drop-in replacement for live services that preserves the full complexity of real API responses — including traffic-conditioned travel times, dynamic POI availability, and weather-dependent routing — while ensuring that every agent sees exactly the same responses given the same inputs. This is not merely caching; it is a deliberate framing that what matters for agent evaluation is not whether the environment is live, but whether it is faithful — reflecting real-world complexity while being fully controllable.

The significance of this move becomes clear when considering the ReAct framework. ReAct agents condition each action on the full history of observations. If a tool call returns a different response on two evaluation runs (which live APIs will), the agent's subsequent actions diverge — and the resulting performance difference could be entirely attributable to environmental noise rather than agent capability. The replay sandbox eliminates this confound, making it possible to attribute performance differences between agents to genuine differences in reasoning and tool-use skill. This is a fundamental shift in what counts as a valid evaluation: the benchmark no longer measures how an agent performs in "the real world" (an ill-defined notion when the world changes between measurements) but rather how it performs in a controlled slice of the real world, frozen for inspection.

Why this is not incremental. This is not a small refinement of existing evaluation practice. It redefines the standard of evidence required to claim that agent A outperforms agent B. Prior work could only claim superiority modulo environmental noise; MobilityBench enables claims that are both rigorous (deterministic) and ecologically valid (real-world responses). The fuzzy matching and nearest-neighbor fallback mechanisms in the sandbox show that the authors considered this problem deeply — they didn't just cache exact API calls but built a resolution system that handles the inevitable minor variations in agent-generated arguments without breaking determinism. This is infrastructure thinking applied to evaluation, and it opens the door for other tool-augmented domains (code execution, database querying, web navigation) to adopt similar replay-based evaluation paradigms.

Evidence anchor. The entire benchmark's reproducibility claim rests on this infrastructure. The paper's finding that ReAct's FPR advantage over Plan-and-Execute (Table 4) is a property of the architecture rather than of the evaluation conditions is only meaningful because the sandbox ensures both frameworks see identical API responses — a comparison that would be meaningless under live APIs with uncontrolled environmental variance.

Innovation 2: Trajectory-Structured Ground Truth as a Diagnostic Multiplier

MobilityBench introduces a form of evaluation that is structurally different from both end-to-end success rates and LLM-as-a-judge scoring — and this difference matters because it changes what kinds of failures can be diagnosed and therefore what kinds of improvements can be guided.

What the field did before. The evaluation landscape for LLM-based agents was dominated by two paradigms, both of which the paper identifies as inadequate for route planning. End-to-end success rates (the dominant approach in benchmarks like ToolBench and TravelPlanner) treat the agent as a black box: either the final output matches the reference, or it doesn't. This tells you that the agent failed but not why. LLM-as-a-judge approaches (Zheng et al., 2023) substitute human evaluation with model-based quality judgments, but these are subjective, poorly calibrated for constraint-satisfaction tasks, and — crucially for route planning — cannot reliably determine whether a spatial constraint was satisfied or a time estimate is feasible.

Both paradigms share a deeper limitation: they produce a single number per task. If that number is low, the researcher knows something went wrong but has no diagnostic signal about where in the agent's pipeline the failure occurred. Was it misunderstanding the query? Extracting the wrong constraints? Planning an incorrect sequence of actions? Selecting the wrong tools? Filling parameters incorrectly? Each of these failure modes requires a different fix — better prompting, improved constraint extraction, more robust planning, or better function-calling — but a single success rate provides no guidance on which fix to pursue.

What's distinctive at the idea level. The paper's ground-truth construction procedure (Section 3.1.3) produces not a single reference answer but a structured trace of the expected intermediate states: the correct intent label, the set of constraints that should be extracted, the sequence of actions that should be planned, and the set of tools that should be selected. This structured trace is the output of "scenario-specific standard operating procedures (SOPs) defined by domain experts" — executable programs that operationalize what a correct solution looks like at multiple levels of abstraction.

This is a diagnostic multiplier. Instead of one binary signal per task (right/wrong), the evaluation protocol produces six or more diagnostic signals per task (ID, IE, DEC-P, DEC-R, TS-P, TS-R, SC, DR, FPR). This changes the nature of evaluation from a verdict to a differential diagnosis. When an agent fails, these metrics together answer: Did it know what kind of task this was? Did it extract all the constraints? Did it plan the right sequence of actions? Did it select the right tools? Did it call them correctly? If the answer to all of these is "yes" but the final answer is wrong, the failure is in the reasoning — a completely different class of problem than a constraint-extraction failure.

Significance beyond the benchmark itself. This multi-dimensional decomposition is not just convenient for the authors' analysis — it establishes a template for agent evaluation that is portable to other tool-augmented domains. Any benchmark that can define SOPs for its tasks (medical decision-making, financial analysis, code generation) can adopt this decomposition, producing per-capability scores that isolate specific failure modes. The key requirement is that the domain admits structured ground truth at multiple stages of the agent pipeline — a requirement that route planning satisfies because its SOPs are well-defined, but that many other domains can also meet.

Why this is fundamental, not incremental. The shift from "did you get the right answer?" to "at each stage, did you produce the right intermediate representation?" is not a refinement of existing evaluation approaches — it is a different kind of question. It enables a research program where improvements are targeted at specific pipeline stages informed by diagnostic data, rather than blind architectural changes validated only by end-to-end metrics.

Evidence anchor. The scenario study (Section 4.2.2, Figure showing radar charts) explicitly uses these multi-dimensional metrics to reveal that Preference-Constrained Planning degrades across all dimensions, not just final pass rate. This kind of diagnostic — "the failure is not confined to decision-making; it propagates from degraded instruction understanding and planning" — would be invisible to end-to-end success rates alone.

Innovation 3: The Capability-Efficiency Spectral Split Between Execution Frameworks

The paper's comparison of ReAct and Plan-and-Execute frameworks produces a finding that is not merely about which framework is "better" but about the irreducible structure of the capability-efficiency tradeoff in agent architectures — and about the conditions under which each side of that tradeoff is preferable.

What the field did before. The ReAct framework (Yao et al., 2022b) had established itself as a dominant paradigm for tool-augmented agents, with its interleaved reasoning-acting loop widely adopted as the default architecture. Plan-and-Execute (Wang et al., 2023) existed as an alternative but was typically evaluated in text-only reasoning tasks, not in tool-use settings with real-time environmental feedback. The implicit assumption in much of the agent literature was that closed-loop architectures were strictly superior for complex tasks — more reasoning steps meant better decisions — and that the additional computational cost was a necessary price to pay for robustness.

The paper challenges this assumption not by arguing that either framework is universally better, but by quantifying the exact nature and magnitude of the split and showing that it varies meaningfully across task scenarios.

What's distinctive at the idea level. The paper's 35.38% average IT increase for ReAct over Plan-and-Execute (Section 4) is not just a cost number — it is a spectral decomposition of where agent architectures spend their computational budget. ReAct's higher FPR (Gemini-3-Pro-Preview: 69.09% under ReAct vs. Claude-Opus-4.5: 65.77% under Plan-and-Execute, both the highest in their framework class) comes from its closed-loop design: when a tool returns unexpected results, the agent observes this and adjusts. Plan-and-Execute's lower FPR comes from its static plan: if the plan contains an error, there is no recovery mechanism because tool responses are never fed back into the reasoning loop.

But the finding is not simply "ReAct wins on FPR, Plan-and-Execute wins on efficiency." The paper identifies a task-dependent interaction: on Preference-Constrained Route Planning tasks, Plan-and-Execute actually performs best because "it establishes a clear strategy in advance, which makes handling structured tasks with logical order more predictable and efficient, thereby suppressing hallucinations and trajectory deviations" (Section 4.2.2). On these highly structured tasks, the closed-loop observe-adjust cycle of ReAct becomes a liability — the agent overcorrects based on intermediate results, introducing errors that a static, pre-planned strategy avoids.

This is a reframing of the architecture choice problem. It's not about picking the framework with the highest average performance; it's about understanding the conditions under which closed-loop correction helps (ambiguous tasks with high uncertainty) and the conditions under which it hurts (structured tasks with clear logical ordering). The benchmark's efficiency metrics enable this analysis to be done quantitatively: you can measure not just whether ReAct improves FPR but whether the improvement justifies the 35.38% token overhead at your deployment's latency and cost constraints.

Why this matters beyond route planning. This finding generalizes: any domain where tasks vary in their degree of structure and ambiguity should expect to see an analogous framework-dependent performance split. The paper does not claim this generalization explicitly, but the principle — that closed-loop architectures benefit unstructured tasks while open-loop architectures benefit structured tasks — is a hypothesis that MobilityBench's data supports and that future work in other domains can test.

Evidence anchor. Table 4 (overall scores) establishes the aggregate FPR gap between frameworks; Section 4.2.2 (scenario study) establishes the task-dependent inversion where Plan-and-Execute outperforms ReAct on Preference-Constrained Planning specifically; and the IT comparison quantifies the cost side of the tradeoff. Together, these three pieces of evidence establish a picture that no single metric could convey.

Innovation 4: The Open-Source / Closed-Source Convergence as an Empirical Finding with Deployment Implications

The paper's model comparison results reveal a pattern that, while not the paper's primary contribution, carries significant implications for the field: the gap between open-source and closed-source models on real-world route-planning tasks is narrowing to within a few percentage points, and the remaining gap is concentrated in specific capability dimensions rather than being uniform.

What the field assumed. The dominant narrative in the LLM community has been that closed-source models (GPT-4, Claude, Gemini) maintain a substantial and persistent lead over open-source alternatives on complex reasoning and tool-use tasks. This narrative has shaped deployment decisions: organizations building production agent systems have largely defaulted to closed-source backbones, treating open-source models as research artifacts not yet ready for mission-critical deployment.

What the data actually shows. Under the ReAct framework, Qwen3-235B-A22B (a Mixture-of-Experts architecture activating only 22B parameters per forward pass) achieves a DR of 85.95% and an FPR of 66.69%, compared to Gemini-3-Pro-Preview's DR of ~85% and FPR of 69.09% (Table 4). The FPR gap is approximately 2.4 percentage points. DeepSeek-V3.2-Exp achieves an FPR of 68.88%, closing the gap to 0.21 percentage points — effectively tied with the best closed-source model.

This convergence is not uniform across capability dimensions. The paper notes that "Claude-Sonnet-4.5, Gemini-3-Pro-Preview still maintained a clear lead in instruction understanding dimensions, with average scores of 90.88% and 88.61%" (Section 4.2). The implication: the remaining closed-source advantage is concentrated in precisely understanding what the user wants, not in executing the plan once understood. This is a spatially localized gap — fix instruction understanding in open-source models (perhaps through better prompting, fine-tuning on mobility-domain data, or improved constraint-extraction modules) and the overall performance gap may disappear entirely.

Why this is not just a leaderboard result. The convergence finding matters because it changes the cost-benefit calculus for deployment. Open-source models offer advantages that closed-source APIs cannot: they can be deployed on-premise (critical for enterprises with data privacy requirements), fine-tuned on proprietary data, optimized for specific hardware, and run without per-query API costs. If the performance gap is 2.4 percentage points on FPR, an organization must weigh that small accuracy difference against the deployment flexibility and cost structure of open-source alternatives. The paper does not make this argument explicitly, but the data enables it — and for a benchmark designed to guide real-world deployment decisions, this is a substantive contribution.

The MoE efficiency angle. The paper's inclusion of "substantially lower inference costs due to its efficient architecture" for DeepSeek-V3.2-Exp, combined with the token efficiency metrics, suggests that the convergence is not just about FPR but about FPR per unit of computational cost. If an open-source model achieves near-parity FPR at lower IT/OT, it may be strictly preferable to a closed-source alternative in cost-sensitive deployments — a claim that would not be evaluable without the benchmark's dual capability-efficiency metrics.

Evidence anchor. Table 4 provides the raw FPR numbers; the Section 4.2 model study provides the instruction-understanding gap analysis; Figure 4 (thinking vs. non-thinking) shows that enabling thinking further narrows the gap but at substantial token cost. The convergence finding is distributed across these results rather than concentrated in a single figure.

Insight Summary: What MobilityBench Changes About Agent Evaluation

Taken together, these four innovations represent a coherent shift in how the field should think about evaluating tool-augmented agents. The pre-MobilityBench approach was: test on live APIs, report end-to-end success rate, accept that results won't replicate exactly, and hope that the noise averages out. The post-MobilityBench approach — enabled by the benchmark but applicable beyond it — is: engineer determinism into the evaluation environment, decompose performance into pipeline-stage diagnostics, measure the capability-efficiency tradeoff quantitatively, and use the resulting multi-dimensional profile to make deployment decisions and target improvements.

This is not an incremental improvement in benchmarking methodology. It is a different kind of evaluation, one that treats the benchmark as an instrument for understanding agent behavior rather than a contest for ranking models. The paper's experimental results demonstrate the value of this approach: without the multi-dimensional metrics, we would know only that the best model achieves ~69% FPR. With them, we know that instruction understanding is the primary bottleneck, that ReAct's closed-loop design helps on ambiguous tasks but hurts on structured ones, that the open-source gap is narrow and concentrated in constraint extraction, and that thinking-mode improves performance but at a cost that makes real-time deployment challenging. These are not findings about which model is best — they are findings about what makes route-planning agents fail and where to invest effort to improve them.

5. Experimental Analysis

Evaluation Methodology

Dataset. MobilityBench contains 100,000 episodes constructed from large-scale, anonymized real user queries collected from Amap over six months, spanning 22 countries and over 350 cities. The episodes are organized into 11 task scenarios across 4 intent families: Basic Information Retrieval (36.6% of episodes), Route-Dependent Information Retrieval (9.6%), Basic Route Planning (42.5%), and Preference-Constrained Route Planning (11.3%). No explicit train/dev/test split is described; evaluation appears to be conducted over the full benchmark.
Base model(s). The paper evaluates a range of both closed-source and open-source models. Closed-source models include Claude-Opus-4.5, Claude-Sonnet-4.5, and Gemini-3-Pro-Preview. Open-source models include the Qwen3 family at multiple scales (Qwen-4B, Qwen-32B, Qwen-30B-A3B as a Mixture-of-Experts architecture, and Qwen-235B-A22B), DeepSeek-V3.2-Exp, and DeepSeek-R1 as a reasoning-oriented baseline. The model selection spans dense architectures (4B to 32B parameters) and MoE architectures to enable scaling analysis, covering both thinking and non-thinking inference modes.
Metrics. The evaluation protocol defines metrics across four capability dimensions plus efficiency, as formalized in Section 3.2 and reported in Table 4:
- Instruction Understanding: Intent Detection (ID), computed as the fraction of queries where the similarity between predicted and ground-truth intent labels exceeds a threshold $\alpha_{\mathrm{threshold}}$ (Equation 1); Information Extraction (IE), computed as the fraction of queries where the predicted constraint set exactly matches the ground-truth constraint set (Equation 2).
- Planning: Task Decomposition Precision (DEC-P) and Recall (DEC-R), measuring step coverage and correctness of the predicted action sequence against the ground-truth sequence (Equation 3).
- Tool Use: Tool Selection Precision (TS-P, coverage of required tools) and Recall (TS-R, complement of redundancy, Equation 4); Schema Compliance (SC), the fraction of tool calls with parameters conforming to valid formats and ranges (Equation 5).
- Decision Making: Delivery Rate (DR), the proportion of queries where the agent produces a complete, executable output; Final Pass Rate (FPR), the proportion of queries where the output satisfies all user-specified constraints.
- Efficiency: Input Tokens (IT), the cumulative token volume in the model's context including system prompts, instructions, and interaction history; Output Tokens (OT), the total tokens generated.
Baselines. The paper does not define explicit baseline agents in the traditional sense. Instead, the baselines are the execution frameworks under which models are evaluated. The primary comparison is between the ReAct framework (Yao et al., 2022b), an interleaved reasoning-acting closed-loop paradigm, and the Plan-and-Execute framework (Wang et al., 2023), which separates planning from execution with a static pre-generated plan. Every model is evaluated under both frameworks, enabling cross-framework comparison as the primary axis of analysis. For the thinking vs. non-thinking study, the baseline is the same model evaluated in non-thinking mode.
Generation budget / compute accounting. The paper does not impose a uniform generation budget constraint across all evaluations. Instead, compute is measured post-hoc through the efficiency metrics IT and OT, which capture the actual token consumption of each agent run. The thinking vs. non-thinking study (Section 4.2.3, Figure 4) samples 1,000 representative instances for a controlled comparison where token overhead is explicitly analyzed. The framework comparison (Section 4.1, Table 4) does not control for total compute; agents are free to take as many reasoning steps and tool calls as their architecture permits, with the resulting IT/OT reported as an outcome variable. This means efficiency is measured rather than constrained — a design choice that reflects real-world deployment where agents operate without hard per-query compute limits but latency and cost are still tracked.
Cross-validation / statistical protocol. The paper does not report any cross-validation, statistical significance testing, or confidence intervals for the experimental results. The 100,000-episode benchmark is evaluated in full for the main results in Table 4. The thinking vs. non-thinking study uses a 1,000-instance random sample. The absence of statistical protocol means that the narrow performance gaps reported (e.g., Gemini-3-Pro-Preview at 69.09% FPR vs. DeepSeek-V3.2-Exp at 68.88%) cannot be assessed for whether they represent genuine differences or sampling variance.

Main Quantitative Results

Overall Model Performance Across Frameworks

Table 4 reports the complete performance profile for all evaluated models under both ReAct and Plan-and-Execute frameworks. The headline numbers establish the performance ceiling and the open-source / closed-source convergence.

Under the Plan-and-Execute framework, Claude-Opus-4.5 stands out as the strongest performer, achieving a Delivery Rate of 83.53% and a Final Pass Rate of 65.77% — both the highest among all evaluated models in this setting. This means Claude-Opus-4.5 successfully produced a complete output for approximately 84% of the 100,000 queries, and that output was fully correct (satisfying all constraints) for approximately 66% of queries. The gap between DR and FPR (approximately 17.8 percentage points) represents cases where the agent completed the task pipeline but produced an incorrect answer — a failure mode the multi-dimensional metrics can diagnose but that an end-to-end metric alone would conflate with non-completion failures.

Under the ReAct framework, Gemini-3-Pro-Preview attained the highest FPR of 69.09% (Table 4). The paper attributes this to its "exceptional ability to preserve task-relevant context and maintain goal focus across extended iterative inference loops" — a qualitative interpretation not directly evidenced by metrics but consistent with ReAct's closed-loop design enabling dynamic error recovery. The DR for Gemini-3-Pro-Preview under ReAct is approximately 85% (estimated from Table 4, which reports only FPR explicitly for the top performer), yielding a DR-FPR gap of approximately 16 percentage points — similar in magnitude to Claude-Opus-4.5's gap under Plan-and-Execute, suggesting that the delivery-to-correctness pipeline has comparable leakage across the best performers in both frameworks.

The open-source / closed-source convergence is the most practically significant finding in Table 4. Under the ReAct framework, Qwen3-235B-A22B (a Mixture-of-Experts architecture activating only 22B parameters per forward pass) achieved a DR of 85.95% and an FPR of 66.69%. This places its FPR within 2.4 percentage points of Gemini-3-Pro-Preview's 69.09%. DeepSeek-V3.2-Exp further narrows the gap, attaining an FPR of 68.88% — only 0.21 percentage points behind the best closed-source model. The paper notes that DeepSeek-V3.2-Exp achieves this "while maintaining substantially lower inference costs due to its efficient architecture," though explicit IT/OT numbers for this comparison are not broken out from the aggregate efficiency metrics.

Critically, the convergence is not uniform across capability dimensions. The paper explicitly states that "Claude-Sonnet-4.5, Gemini-3-Pro-Preview still maintained a clear lead in instruction understanding dimensions, with average scores of 90.88% and 88.61% under the ReAct framework" (Section 4.1). The phrase "average scores" in instruction understanding likely refers to the mean of ID and IE, though this is not explicitly specified. The implication: the remaining closed-source advantage is concentrated in precisely extracting what the user wants from natural language queries, not in planning or tool execution once the constraints are understood. This is a spatially localized gap — and a diagnosable one, made visible only because the evaluation protocol decomposes performance into separate dimensions.

Framework Comparison: ReAct vs. Plan-and-Execute

The systematic comparison of the two execution architectures reveals a fundamental trade-off that the paper characterizes as "between task success rate and computational efficiency" (Section 4.1). The evidence for this trade-off comes from two distinct sources in Table 4.

Final pass rate advantage for ReAct. Across models, the ReAct framework consistently achieves higher FPR than Plan-and-Execute. The paper explains this as a consequence of ReAct's "closed-loop 'think-act-observe' mechanism, which allows the agent to dynamically adjust its strategy based on real-time results returned by tools, while Plan-and-Execute's static pre-planning shows a significant lack of robustness when facing dynamic feedback in mobile scenarios" (Section 4.1). Concrete numbers supporting this claim: Gemini-3-Pro-Preview achieves 69.09% FPR under ReAct versus Claude-Opus-4.5's 65.77% under Plan-and-Execute (the respective framework leaders). While this is not a within-model comparison (different models lead different frameworks), the paper presents the aggregate pattern as evidence of ReAct's robustness advantage.

Efficiency penalty for ReAct. The paper quantifies the cost of ReAct's benefit: "Across all models, ReAct's average IT is approximately 35.38% higher than Plan-and-Execute's. This increase translates directly into higher API costs and longer wall-clock inference times" (Section 4.1). This 35.38% figure is an average across all evaluated models, meaning it includes models with varying ReAct step counts. The mechanism is clear from ReAct's design: each tool invocation appends its response to the context, which then becomes part of the input for the next reasoning step. An agent taking 10 tool-call steps under ReAct processes approximately 10 observations-worth of additional context compared to a Plan-and-Execute agent that pre-plans all calls and executes them without iterative observation.

What the numbers do not tell us directly: the paper does not report a per-model breakdown of IT and OT, so we cannot determine whether the 35.38% overhead is consistent across models or concentrated in models that take more ReAct steps. We also cannot compute an "FPR per token" efficiency metric that would enable direct cost-normalized comparison. The paper presents the trade-off as qualitative — ReAct is more robust but more expensive — without quantifying whether the per-percentage-point FPR improvement justifies the per-token cost increase. This is a limitation of the current analysis that the benchmark's metric design supports but the paper does not fully exploit.

Scenario Study: Performance Degradation on Complex Tasks

Section 4.2.2 presents a scenario-level analysis using multi-dimensional indicator radar charts (Figure referenced as "Figure LABEL:fig:radars," which appears to be an unresolved LaTeX reference — likely the figure showing radar plots for the four intent families). The paper states that "the scene from left to right represents a significant increase in the depth of task logic and the complexity of constraints, and Preference-constrained Planning is the category where the model is the most likely to be error as we expected."

The key finding is that Preference-Constrained Route Planning — the most complex intent family, encompassing Option-Constrained Route Planning and Route-Constrained Planning — exhibits degraded performance across all evaluation dimensions, not just final pass rate. The degradation propagates through the entire agent pipeline: lower instruction understanding (agents more frequently misclassify the intent or miss constraints when preferences are involved), degraded planning (the action sequences are less complete and contain more errors), and impaired tool use (tool selection and schema compliance both decline). This is not a single-point failure but a systemic degradation — the complexity of preference-constrained tasks stresses every stage of the agent's reasoning chain.

A counterintuitive finding emerges specifically for this task family: under Preference-Constrained Planning, the Plan-and-Execute framework performs best. The paper explains: "Plan-and-Execute framework performs best because it establishes a clear strategy in advance, which makes handling structured tasks with logical order more predictable and efficient, thereby suppressing hallucinations and trajectory deviations" (Section 4.2.2). This is notable because it inverts the aggregate trend (where ReAct outperforms Plan-and-Execute). On highly structured tasks where the correct sequence of actions has a clear logical ordering — extract constraints, resolve locations, compute route with preference flags, verify constraint satisfaction — the static pre-planning of Plan-and-Execute avoids the over-correction that ReAct's closed-loop design can introduce. ReAct's observe-adjust cycle, which helps on ambiguous tasks, becomes a liability when the task structure is rigid: intermediate observations may cause the agent to deviate from the correct plan based on spurious or misinterpreted feedback.

The paper does not provide per-scenario FPR numbers explicitly, making it impossible to quantify exactly how much performance degrades from Basic Route Planning to Preference-Constrained Route Planning. The radar chart visualization is the primary evidence for this claim, and without seeing the actual figure or exact values, the magnitude of the degradation cannot be independently assessed. This is a transparency limitation.

Model Study: Scaling and Inference Mode Effects

Scaling effect. Section 4.2.3 reports a model-centric study examining how parameter scale influences performance on the benchmark. Under the same dense architecture, "scaling the base model from 4B to 32B yields a consistent improvement in average success rate, increasing by 0.91%." Under the MoE setting, "Qwen-30B-A3B further scales to Qwen-235B-A22B, bringing an additional gain of 5.43%." The paper presents these as evidence that "larger parameter scale leads to higher success rates in real-world mobility scenarios" — consistent with classic scaling law expectations.

A more nuanced finding emerges from the DEC-P and DEC-R metrics across model sizes. The paper reports: "By jointly examining DEC-P and DEC-R, we observe that, compared with smaller models, larger models tend to produce longer solution trajectories (i.e., more plans) to explore a broader space of possible outcomes. Although some of these steps can be redundant, this more exhaustive search-and-verification process ultimately improves the task success rate." This is a qualitative mechanism claim: larger models improve not by being more precise but by being more exhaustive — they generate more candidate actions, some of which are redundant, but the broader exploration increases the probability that the correct actions are included. This is evidenced by higher DEC-R (better coverage) at the cost of lower DEC-P (lower precision due to redundant steps) as scale increases. The specific DEC-P and DEC-R values across scales are not reported numerically, limiting quantitative assessment.

Thinking vs. Non-thinking. To examine the impact of reasoning mode, the paper samples 1,000 representative instances from MobilityBench and evaluates models with and without thinking enabled under the Plan-and-Execute framework (Section 4.2.3, Figure 4). The models evaluated are Qwen-4B, Qwen-32B, Qwen-30B-A3B, and Qwen-235B-A22B, with DeepSeek-R1 included as a strong reasoning-oriented baseline.

DeepSeek-R1 achieves a Final Pass Rate of 70.46% — the highest reported in the paper, surpassing all models evaluated in Table 4, including the closed-source leaders. This serves as a competitive reference point for what thinking-mode models can achieve on route-planning tasks.

Across all Qwen models, enabling thinking consistently improves performance. The largest gain is observed for Qwen-30B-A3B, whose FPR increases by 5.98% absolute when thinking is enabled. The paper does not report exact pre-thinking and post-thinking FPR values for each model, only the deltas and the Figure 4 visualization, so the absolute performance levels cannot be reconstructed from the text alone.

The critical caveat: "Despite these gains, Thinking substantially increases the generated token volume, leading to markedly higher inference cost and latency. This overhead makes it challenging to deploy Thinking-enabled agents in real-time, production-grade online settings." The paper does not quantify the token overhead in absolute terms or as a percentage increase, nor does it report the corresponding IT/OT metrics for thinking vs. non-thinking runs. The tension is stated qualitatively — thinking helps but is expensive — without the quantitative cost-benefit analysis that the benchmark's efficiency metrics are designed to enable. This is a missed opportunity: the paper has the infrastructure to compute "FPR improvement per additional output token" for thinking-mode, which would provide actionable guidance for practitioners weighing accuracy against latency, but this analysis is not performed.

Ablation Studies and Robustness Checks

The paper does not contain traditional ablation studies in the sense of removing components of the proposed method and measuring performance degradation — because MobilityBench is a benchmark, not a method. There are no model architecture ablations, training data ablations, or hyperparameter ablations to report. What the paper does contain is a set of controlled comparisons that serve a function analogous to ablation studies: they isolate specific variables (execution framework, model scale, inference mode, task scenario) and measure their impact on the multi-dimensional metrics.

Execution framework (ReAct vs. Plan-and-Execute). This is the paper's primary controlled comparison (Table 4). The finding that ReAct achieves higher FPR but at 35.38% higher average IT is robust across all evaluated models — it is not dependent on a specific model or task distribution. The scenario-level qualification (Plan-and-Execute outperforms ReAct on Preference-Constrained Planning) demonstrates that the framework effect is not uniform but task-dependent, which is a more nuanced finding than a simple "ReAct > Plan-and-Execute" conclusion. However, the paper does not provide per-scenario IT/OT metrics, so we cannot determine whether Plan-and-Execute's advantage on structured tasks extends to efficiency or is purely an FPR advantage.
Model scale (4B → 32B dense; 30B-A3B → 235B-A22B MoE). The scaling comparison (Section 4.2.3) demonstrates monotonic improvement with scale, but the specific scaling coefficients (the 0.91% gain from 4B to 32B and 5.43% from 30B-A3B to 235B-A22B) are not directly comparable — they represent different architectures (dense vs. MoE) with different active parameter counts. The 5.43% MoE gain is substantially larger than the 0.91% dense gain, but this could reflect either (a) MoE architectures scaling more efficiently for route-planning tasks, (b) the larger absolute parameter increase in the MoE case, or (c) differences in training data or procedure between the Qwen-30B-A3B and Qwen-235B-A22B models. The paper does not disentangle these explanations.
Thinking vs. Non-thinking inference mode. The controlled comparison on 1,000 sampled instances (Figure 4) demonstrates consistent gains from thinking mode, with the largest absolute improvement of 5.98% for Qwen-30B-A3B. This is the closest the paper comes to an ablation: for each model, the non-thinking configuration serves as the baseline, and thinking is the "intervention." The finding that thinking helps across all scales suggests that route-planning tasks benefit from extended reasoning, but the paper does not analyze what kind of reasoning the thinking mode produces — whether it generates better plans, catches constraint-extraction errors, or simply explores more alternatives (analogous to the scaling effect on DEC-P/DEC-R). Without qualitative analysis of thinking traces, the mechanism remains opaque.
Task scenario difficulty. The scenario study (Section 4.2.2) serves as a task-difficulty ablation, showing that performance degrades systematically as task complexity increases from Basic Information Retrieval → Route-Dependent Information Retrieval → Basic Route Planning → Preference-Constrained Route Planning. This validates the taxonomy's ordering as a difficulty hierarchy and demonstrates that the benchmark can discriminate between easy and hard tasks — a necessary property for a useful benchmark. However, the paper does not report per-scenario metrics numerically, making it impossible to assess whether the degradation is gradual (each scenario slightly harder than the previous) or cliff-like (a sharp drop at Preference-Constrained Planning specifically).

Missing ablations that would have strengthened the paper:

Per-model ReAct vs. Plan-and-Execute comparison. Table 4 reports results for each model under one framework, not both frameworks for the same model. This means we cannot compute a within-model framework effect — we don't know whether Claude-Opus-4.5 under ReAct outperforms Claude-Opus-4.5 under Plan-and-Execute, or vice versa. The framework comparison is confounded with model identity. A true ablation would evaluate the same model under both frameworks on the same episodes, isolating the framework effect from model-specific factors.
Difficulty-stratified efficiency analysis. The paper reports aggregate IT/OT across all tasks but does not break down token consumption by task scenario. This means we cannot determine whether ReAct's 35.38% IT overhead is uniform across task types or concentrated in specific scenarios. If ReAct's overhead is primarily on complex tasks (where its closed-loop correction is most valuable), the cost-benefit calculus differs from a scenario where the overhead is uniform.
Thinking-mode token overhead quantification. Figure 4 reports FPR gains from thinking but does not report the corresponding IT/OT increases. The paper states that thinking "substantially increases the generated token volume" but provides no numbers. Computing and reporting the FPR-per-token ratio for thinking vs. non-thinking would transform a qualitative tension into a quantitative trade-off that practitioners can act on.
Sandbox fallback sensitivity analysis. The replay sandbox includes fuzzy matching and nearest-neighbor spatial matching with a maximum distance threshold (Section 3.1.4). The paper does not report what fraction of agent tool calls required fallback matching rather than exact cache hits, nor does it analyze whether agent performance correlates with fallback rate. If some models produce systematically "fuzzier" tool calls (requiring more fallback resolution), this could indicate a precision difference in tool use that the current metrics do not isolate.

Critical Assessment

The experimental results in Section 4 provide a comprehensive performance profile of current LLM-based agents on real-world route-planning tasks, but the evidence for specific claims varies substantially in strength and completeness.

The benchmark's primary claim — that it enables reproducible, multi-dimensional evaluation of route-planning agents — is demonstrated by its existence and use, not by a controlled experiment. The paper shows that the benchmark produces consistent, fine-grained metrics across 100,000 episodes, multiple models, and two execution frameworks. Table 4 demonstrates that the evaluation protocol generates distinct, interpretable scores for each capability dimension, and the scenario study (Section 4.2.2) shows that these scores vary meaningfully across task difficulties. However, the paper does not compare MobilityBench's reproducibility against a live-API baseline — there is no experiment demonstrating that two runs of the same agent on live APIs produce different scores while two runs on the replay sandbox produce identical scores. The reproducibility claim rests on the design argument (Section 3.1.4) rather than empirical validation.

The claim that current models "perform competently on Basic Information Retrieval and Route Planning tasks, yet struggle considerably with Preference-Constrained Route Planning" is supported qualitatively by the scenario study but lacks precise quantification. The paper states that Preference-Constrained Planning is "the category where the model is the most likely to be error as we expected" and that performance degrades across all dimensions, but per-scenario FPR numbers are not reported. The radar charts (Figure LABEL:fig:radars) provide a visual comparison but no table of per-scenario scores. A reader cannot determine whether "struggle considerably" means FPR drops from 80% to 60% or from 60% to 20%. The claim is directionally correct but imprecisely evidenced. The gap between this claim's strength in the abstract and the specificity of its supporting data is the most significant weakness in the experimental presentation.

The claim that "the gap between open-source and closed-source models is narrowing substantially" is well-supported by the FPR numbers in Table 4. Qwen3-235B-A22B at 66.69% FPR versus Gemini-3-Pro-Preview at 69.09% (a 2.4 percentage point gap) and DeepSeek-V3.2-Exp at 68.88% (a 0.21 percentage point gap) provide concrete, directly comparable evidence. However, two caveats weaken this claim. First, the paper does not report whether these differences are statistically significant — with 100,000 episodes, even small absolute differences may be statistically distinguishable, but we cannot assess whether the 0.21 point gap is noise or signal. Second, the paper does not control for inference cost: if DeepSeek-V3.2-Exp achieves 68.88% FPR at lower IT/OT than Gemini-3-Pro-Preview's 69.09%, it may be strictly preferable in cost-normalized terms, but the paper does not compute cost-normalized scores.

The claim that "thinking-mode inference improves performance at the cost of markedly higher token overhead, making real-time deployment challenging" is supported in its first half (Figure 4 shows consistent FPR gains from thinking) but only asserted in its second half. The paper provides no token-overhead numbers for thinking mode. The statement that the overhead is "markedly higher" and that it makes real-time deployment "challenging" is a qualitative judgment without quantitative backing. The benchmark's efficiency metrics (IT and OT) are designed to support exactly this analysis; their absence from the thinking-mode comparison is a missed opportunity that weakens the paper's deployment-relevance claims.

The framework comparison claim — that ReAct offers higher robustness at higher computational cost — is supported directionally but confounded by model identity. Table 4 does not show within-model framework comparisons. We see Gemini-3-Pro-Preview under ReAct and Claude-Opus-4.5 under Plan-and-Execute, but we do not know how either model performs under the other framework. The 35.38% IT overhead is an average across models evaluated under ReAct versus those under Plan-and-Execute, not a matched comparison. It is possible that the models evaluated under ReAct are inherently more verbose or that ReAct causes models to take more steps, but the two explanations cannot be separated from the reported data. A cleaner experiment would evaluate 2–3 models under both frameworks and report the within-model framework effect on FPR and IT/OT.

The scaling analysis provides evidence that "larger models perform better" but the numbers are not directly comparable across architectures. The 0.91% gain from 4B to 32B (dense) and 5.43% from 30B-A3B to 235B-A22B (MoE) represent different scaling regimes with different active parameter counts. The larger gain in the MoE case could reflect the larger absolute parameter increase, architectural advantages of MoE for tool-use tasks, or differences in training data. Without more models at intermediate scales within each architecture family, the scaling trend cannot be characterized precisely.

Experiments that would have strengthened the paper:

Within-model framework comparison: Evaluate Claude-Opus-4.5, Gemini-3-Pro-Preview, Qwen3-235B-A22B, and DeepSeek-V3.2-Exp under both ReAct and Plan-and-Execute. Report per-model framework effects on FPR, DR, IT, and OT.
Per-scenario quantitative breakdown: A table supplementing the radar charts with exact ID, IE, DEC-P, DEC-R, TS-P, TS-R, SC, DR, FPR, IT, and OT for each of the 11 task scenarios, for at least the top-performing models.
Thinking-mode efficiency quantification: For the thinking vs. non-thinking comparison (Figure 4), report IT and OT for each model in both modes, and compute FPR-per-1K-tokens as a cost-normalized metric.
Statistical significance reporting: Given 100,000 episodes, even small performance differences may be statistically significant. Report confidence intervals or standard errors for the main FPR comparisons, particularly the narrow gaps between top models.
Reproducibility validation: Run the same agent-model combination on the same 1,000 episodes twice (separated by time or machine) on the replay sandbox versus live APIs, and demonstrate that the sandbox produces identical scores while live APIs produce divergent scores. This would empirically validate the reproducibility claim.
Sandbox fallback rate analysis: Report the fraction of tool calls that required fuzzy matching or nearest-neighbor fallback for each model, and assess whether fallback rate correlates with SC or FPR. This would validate that the sandbox's tolerance mechanisms are not masking genuine tool-use precision differences.

In summary: The experiments successfully demonstrate that MobilityBench produces rich, multi-dimensional performance profiles that reveal capability differences invisible to end-to-end metrics — the scenario study's finding that ReAct and Plan-and-Execute invert their relative performance on Preference-Constrained Planning is the strongest example of this diagnostic value. The open-source convergence finding is well-supported by the FPR numbers and carries genuine practical implications. However, the paper's headline claims about scenario-specific performance degradation and thinking-mode deployment challenges are supported qualitatively rather than quantitatively, and the framework comparison is confounded by model identity in ways that limit its inferential strength. The benchmark infrastructure is more powerful than the experiments fully exploit — the efficiency metrics and per-scenario breakdowns exist in the design but are underutilized in the analysis.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Not Included in Reported Efficiency

The assumption. The compute-optimal allocation framework's core premise is that test-time compute can be distributed across search algorithms and revision strategies in a difficulty-aware manner. For this to work in deployment, the system must estimate each prompt's difficulty before deciding how to allocate the budget. The paper's difficulty estimation method requires generating 2048 samples per question and scoring them with the PRM to compute an average final-answer score. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N baselines are computed after difficulty is known, without amortizing the cost of learning it. Generating 2048 samples exceeds the entire test-time compute budget for every budget level studied (the paper sweeps from 1 to 512 generations). If this cost were included, the total compute for the compute-optimal approach could be larger than the best-of-N baseline it claims to outperform, completely invalidating the efficiency claim for deployment scenarios. The paper frames this as an exploration-exploitation tradeoff—compute spent assessing difficulty versus compute spent solving the problem—but provides no analysis of where the break-even point lies or whether any practical difficulty estimation method could recover the claimed gains.

What evidence exists. The paper reports no experiment that includes difficulty estimation cost in the compute budget. Figures 4 and 8, which plot compute-optimal scaling curves, use an x-axis (generation budget) that excludes the 2048 samples used for bin assignment. The paper acknowledges this gap but only qualitatively.

Mitigation status. Not addressed. The paper flags it as "a key avenue for future work" (Section 3.2) and suggests training models to predict difficulty directly from the question text, but no such model is developed, evaluated, or even speculatively costed. The current results represent an upper bound on achievable efficiency that may be unattainable in practice until a cheap difficulty estimator is developed.

The 14× Larger Model Baseline Is Intentionally Weakened

The assumption. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters that was trained by scaling parameters while holding training data fixed—following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal training (Hoffmann et al., 2022), where both parameters and data would be scaled equally. The authors explicitly note this design choice:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the 14× larger model is evaluated only with greedy decoding—no test-time compute augmentation of any kind, not even majority voting over a modest number of samples.

The consequence. A Chinchilla-optimally trained model at 14× the total FLOPs budget would likely outperform the parameter-only-scaled baseline used in the comparison. Since the paper's core finding—that test-time compute can substitute for pretraining compute on easy-to-medium problems—relies on outperforming this specific baseline, the reported advantages (e.g., +27.8% relative improvement on easy questions at low inference-to-pretraining ratios) may shrink or reverse against a properly compute-optimal larger model. Furthermore, giving the larger model even a modest test-time compute budget (best-of-8 or best-of-16) would create a substantially stronger baseline that is never tested. The comparison answers the question "can test-time compute beat a parameter-scaled model with greedy decoding?" rather than the more relevant question "can test-time compute beat a properly trained larger model used at its best?"

What evidence exists. The FLOPs-matched results in Figure 9 and the bar charts in Figure 1 all use the parameter-only-scaled, greedy-decoding baseline. The paper does not report results against a Chinchilla-optimal baseline or against a larger model with any test-time compute augmentation.

Mitigation status. The authors acknowledge the Chinchilla caveat explicitly (quoted above) but do not address the greedy-decoding limitation of the baseline. Both choices make the test-time compute approach look more favorable than it might be against a stronger competitor. This is a methodological choice that the reader must account for when interpreting the FLOPs-matched results.

Hard Problems Remain Essentially Unsolved Regardless of Compute Budget

The capability bound. Across every method studied—PRM search, iterative revisions, and their compute-optimal combinations—the hardest questions (difficulty bin 5, where the base model's pass@1 is near zero) show virtually no improvement regardless of how much test-time compute is allocated. This is visible consistently across all experimental results.

The consequence. Test-time compute can amplify existing capability but cannot create capability that the base model lacks. For problem classes where the model's pass@1 is approximately zero, no amount of search or revision will find correct solutions because there are no correct solutions in the proposal distribution to discover or refine. This means the approach offers no path forward for genuinely novel or out-of-distribution reasoning tasks that exceed the base model's training distribution. For such problems, scaling pretraining remains the only viable path—a finding the paper acknowledges candidly in the Section 7 takeaway box but which fundamentally bounds the applicability of test-time compute scaling as a general substitute for larger models.

What evidence exists. Figure 3 (right) shows bin 5 accuracy hovering at 1–3% for all methods and all budgets. Figure 7 (right) shows bin 5 at roughly 2–3% regardless of the sequential-to-parallel ratio. Figure 9 shows the bin 5 scaling line essentially flat near 0–5% across all budgets, and below the 14× larger model's greedy performance (stars) for all values of the inference-to-pretraining ratio. In the FLOPs-matched comparison (Figure 1 bar charts, bottom-right), hard questions show a −52.9% relative disadvantage for PRM search over the larger model at high inference ratios. This is a robust, replicated finding across every experimental condition.

Mitigation status. The authors are transparent about this limitation—the Section 7 takeaway box explicitly states that test-time compute "cannot compensate for fundamental capability gaps that larger pretraining would address." No mitigation is proposed because the limitation is intrinsic: the proposal distribution contains no correct answers, so no selection or refinement strategy can find one. The practical implication is that compute-optimal test-time scaling should be deployed only for problems within the base model's approximate capability range, and hard problems should be routed to larger models or flagged for human intervention.

Single Benchmark and Single Model Family Limit Generality

The assumption. All experiments use the MATH benchmark (500 test questions of competition-level mathematics) with PaLM 2-S* as the base model. The authors argue in Section 4 that this model is "representative of the capabilities of many contemporary LLMs" and that MATH is a suitable testbed because test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences. This claim of representativeness is asserted without empirical support from other benchmarks or model families.

The consequence. Several aspects of the findings could be model-specific or benchmark-specific in ways that would not generalize. The PRM's quality and its over-optimization behavior depend on PaLM 2-S*'s output distribution; a model with different calibration properties, different error patterns, or different tokenization (affecting the step-level decomposition that the PRM scores) might exhibit qualitatively different difficulty-dependent scaling curves. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The MATH benchmark consists exclusively of problems requiring multi-step symbolic mathematical reasoning with unambiguous ground-truth answers. It is unclear whether the central findings—beam search hurting easy problems due to over-optimization, sequential revisions dominating on easy problems, parallel sampling helping on hard ones—generalize to other reasoning domains (code generation, logical deduction, scientific reasoning) or to tasks requiring factual knowledge retrieval rather than step-by-step inference. The paper provides no replication on even a second benchmark or second model family to support the representativeness claim.

What evidence exists. All quantitative results—Figures 3, 4, 6, 7, 8, 9, and the associated tables—are from MATH with PaLM 2-S*. There is no cross-benchmark or cross-model-family replication. The representativeness claim in Section 4 is an assertion, not an empirical finding.

Mitigation status. Not addressed in the paper. The authors acknowledge this as a scope limitation implicitly by not claiming generality beyond the studied setting, but they do not explicitly flag it as a limitation requiring future validation. A replication study on at least one additional benchmark (code generation, e.g., HumanEval or APPS) and one additional model family would substantially strengthen confidence in the generality of the difficulty-dependent scaling patterns.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate with No Fundamental Fix

The design tradeoff. The revision model is trained exclusively on sequences where all in-context answers are incorrect followed by a correct target (Section 6.1). This means the model never sees examples where the current answer is already correct and should be preserved. At inference time, when the model generates a correct answer early in the revision chain, it may encounter this correct answer in its context on the next revision step—a scenario it was never trained to handle. The paper reports in Section 6.1 that "approximately 38% of correct answers get converted back to incorrect ones" using a naive approach that always takes the last revision.

The consequence. The revision model's sequential chain does not monotonically improve. A correct answer at step 3 may be "revised" into an incorrect answer at step 4, then possibly back to correct at step 5. This means that simply generating a chain of revisions and taking the last output is not a viable strategy—the system must evaluate and select among all answers in the chain. The paper's mitigation (using majority voting or verifier-based selection across the chain) is a post-hoc patch that adds computational overhead and does not prevent the underlying reversion problem; it merely tries to recover the correct answer after the model has already discarded it. In latency-constrained settings where the system must commit to an answer at each step, this reversion behavior would be unacceptable.

What evidence exists. The 38% reversion rate is reported in Section 6.1 based on the authors' own measurement. Figure 6 (left) shows that pass@1 at each step does not monotonically increase—it fluctuates around 23–25% after step 4, consistent with correct answers being periodically lost and sometimes recovered. The effectiveness of within-chain selection (verifier or majority) in recovering from reversions is shown by the gap between the "sequential + best-of-N weighted" and "sequential + majority" curves in Figure 6 (right), but even the best selection method does not eliminate the reversion problem—it only mitigates its impact on the final selected answer.

Mitigation status. Partially addressed. The paper's solution (selecting the best answer from anywhere in the chain rather than taking the last revision) reduces the impact of reversions on final accuracy, as evidenced by sequential sampling outperforming parallel sampling (Figure 6, right). However, this is a workaround, not a fix. A principled solution—such as training the revision model on trajectories where correct answers are preserved, incorporating a "no revision needed" signal, or using the PRM to dynamically decide whether to continue revising or stop—is not explored. The paper acknowledges this implicitly by treating within-chain selection as necessary infrastructure, but does not analyze whether reversions fundamentally limit the achievable benefit from deeper revision chains. The ReSTᵉᵐ experiment (Appendix K, Figure 16), where additional sequential revisions with an RL-optimized model substantially hurt performance, further suggests that the revision approach is fragile and sensitive to training methodology in ways the paper does not fully understand.

Latency and Serial Dependence Are Not Accounted for in the Compute Budget

The tradeoff. The paper measures test-time compute in "generations"—the number of complete solutions sampled—which serves as a reasonable proxy for total FLOPs but ignores wall-clock time and the serial versus parallel nature of different strategies. Sequential revision chains (where each revision depends on the output of the previous one) are inherently serial: generating 64 sequential revisions takes approximately 64× the wall-clock time of generating a single solution, regardless of how much parallel hardware is available. In contrast, best-of-N parallel sampling with N=64 can be executed simultaneously on sufficient hardware, achieving the same compute budget in approximately 1/N of the wall-clock time. Beam search sits in between: each step can be parallelized across beams, but step dependencies impose serial bottlenecks.

The consequence. The compute-optimal policy's preference for sequential strategies on easy problems (Figure 7, right: fully sequential is optimal for bins 1–2 at 128 generations) and for balanced sequential-parallel ratios on harder problems means that the optimal strategy in FLOPs-terms may be wildly suboptimal in latency-terms. For latency-sensitive applications—interactive navigation assistants, real-time decision-making systems, any deployment where the user is waiting for a response—a strategy that allocates 128 generations as 64 sequential × 2 parallel chains could take approximately 32× longer wall-clock time than a pure parallel strategy with the same generation budget, despite having the same total FLOPs. The paper's efficiency metrics (IT and OT) measure token consumption but not wall-clock time, so the latency dimension of the tradeoff is completely invisible in the reported results.

What evidence exists. The paper provides no latency measurements for any strategy or budget level. Figure 7 reports the sequential-to-parallel ratio sweep but only in terms of accuracy, with no corresponding wall-clock time data. The efficiency metrics in Section 3.2.5 cover input tokens and output tokens but not elapsed time or parallelism utilization. The paper implicitly acknowledges the serial nature of revisions by distinguishing between "fully parallel" and "fully sequential" in the ratio sweep, but does not analyze the latency implications of this distinction.

Mitigation status. Not addressed. The paper frames the problem entirely in terms of generation budget (a FLOPs proxy) without discussing latency, throughput, or the practical deployability of sequential-heavy strategies in real-time settings. This is a significant omission because route-planning—the paper's motivating application domain—is inherently latency-sensitive: a navigation system that takes minutes to compute a route is useless regardless of its accuracy. Future work integrating latency constraints into the compute-optimal allocation framework (e.g., adding a maximum wall-clock time constraint that penalizes serial strategies) would substantially increase the practical relevance of the approach.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts agent evaluation from a verdict paradigm to a diagnostic paradigm. Prior benchmarks for tool-augmented agents—ToolBench, TravelPlanner, TravelBench, τ-bench—operated under the assumption that evaluation means producing a single success-or-failure judgment per task, optionally supplemented by LLM-as-a-judge quality scores. MobilityBench demonstrates that this is not merely coarse but actively misleading for complex, constraint-satisfaction domains: two agents with identical end-to-end pass rates can have fundamentally different failure profiles (one failing at instruction understanding, another at tool selection), and those profiles demand different interventions. The multi-dimensional protocol—decomposing performance into intent detection, information extraction, task decomposition, tool selection, schema compliance, delivery rate, and final pass rate—is not a refinement of the verdict paradigm. It is a different kind of evaluation, one designed to answer "why did the agent fail?" rather than merely "did the agent fail?"

The magnitude of this shift is reframing rather than paradigm shift. The paper does not invent new agent architectures or new training methods; it builds infrastructure that changes what questions the field can ask about the agents it already has. This mirrors the impact of diagnostic benchmarks like CheckList (Ribeiro et al., 2020) in NLP evaluation, which reframed evaluation from aggregate accuracy to capability-level probing. MobilityBench does for route-planning agents what CheckList did for text classification—it replaces a single-number summary with a capability inventory, enabling targeted improvement rather than blind architectural iteration. The key difference is that MobilityBench targets a domain where the environment (mapping APIs) is itself non-deterministic, requiring the replay sandbox infrastructure as a prerequisite for any diagnostic protocol to be meaningful.

The paper resolves a latent tension between reproducibility and ecological validity that the agent evaluation community has largely accepted as irresolvable. Prior work implicitly treated these as a tradeoff: use live APIs and accept noisy, non-reproducible results; or use simplified environments and accept that your findings may not transfer to real deployment. The replay sandbox demonstrates that this tradeoff is not fundamental—it is an artifact of insufficient infrastructure. By caching real API responses at collection time and replaying them deterministically during evaluation, the benchmark achieves both the realism of live-service responses and the reproducibility of a static test set. This is a methodological proof of concept with implications beyond route planning: any domain where agents interact with non-deterministic external services (financial data APIs, weather services, code execution environments with external dependencies) can adopt this replay-based evaluation paradigm. The specific engineering choices—canonicalized argument keys, fuzzy matching fallbacks, nearest-neighbor spatial matching with distance thresholds, explicit failure handling for unresolvable calls—provide a template that other benchmark designers can adapt.

The scenario study finding—that Plan-and-Execute outperforms ReAct on Preference-Constrained Planning despite ReAct's aggregate FPR advantage—reconciles a contradiction in the agent architecture literature. The ReAct framework (Yao et al., 2022b) has been treated as a near-universal improvement over static planning for complex tasks, with its closed-loop observe-adjust cycle assumed to strictly dominate open-loop execution. MobilityBench reveals that this dominance is task-structure-dependent: on tasks with clear logical ordering and rigid constraints (like preference-constrained routing, where you must extract constraints → resolve locations → compute route with flags → verify constraint satisfaction), ReAct's dynamic adjustment can become over-correction, introducing errors that a pre-planned strategy avoids. This finding is not merely about which framework is "better"—it provides a boundary condition for ReAct's effectiveness that the original paper did not characterize. Researchers developing agent architectures should now test whether their proposed improvements to ReAct help on the structured tasks where ReAct currently underperforms Plan-and-Execute, or whether they simply amplify ReAct's existing strengths on ambiguous tasks.

The benchmark makes certain research directions more attractive and others less so. More attractive: (1) work on improving instruction understanding specifically, because the paper shows this is where the open-source / closed-source gap is concentrated (Claude-Sonnet-4.5 and Gemini-3-Pro-Preview maintain leads of ~90% vs. lower open-source scores on instruction understanding dimensions, even when open-source FPR is competitive); (2) work on efficient thinking-mode inference, because Figure 4 demonstrates consistent FPR gains from thinking but the paper's qualitative assessment is that the token overhead makes real-time deployment "challenging"—a problem that efficient reasoning architectures (speculative decoding, reasoning distillation, adaptive computation) could address; (3) hybrid execution frameworks that switch between Plan-and-Execute and ReAct per-task, because the scenario study shows that the optimal framework depends on task structure. Less attractive: (1) purely architectural innovations that claim aggregate improvements without task-structure-stratified evaluation, because the benchmark reveals that aggregate comparisons can mask per-scenario inversions; (2) agent evaluations that rely on live APIs without reproducibility controls, because the paper has demonstrated a viable alternative and future reviewers are likely to expect it.

Follow-Up Research This Work Enables

Cheap difficulty estimation ported to route planning: learning to predict task scenario from the query alone. The paper's task taxonomy (11 scenarios across 4 intent families) defines a difficulty hierarchy where Preference-Constrained Route Planning is the hardest category. A natural extension is to train a lightweight classifier—distilled from Qwen-4B's intent labels or from the ground-truth annotations—that predicts the task scenario directly from the user query text. This would enable dynamic framework selection: route Basic Information Retrieval queries to a lightweight Plan-and-Execute agent (since these are easy and efficiency matters), and route Preference-Constrained Planning queries to a thinking-mode ReAct agent (since these benefit from extended reasoning). The benchmark's 100,000 labeled episodes, with ground-truth intent labels and per-scenario annotations, provide training data that no prior route-planning benchmark offered. A strong follow-up would train such a classifier, measure its accuracy on predicting the 11-way scenario label (with a confusion matrix showing which scenarios are confused), and then deploy a hybrid system that selects framework and thinking mode per-query and measures the resulting FPR-vs-token-efficiency tradeoff against uniform strategies.

Within-model framework ablation to isolate the ReAct vs. Plan-and-Execute effect. The paper's Table 4 reports framework comparisons across different models—we see Gemini-3-Pro-Preview under ReAct and Claude-Opus-4.5 under Plan-and-Execute, but never the same model under both frameworks. This confounds framework architecture with model identity. A direct follow-up would evaluate 3–4 models (Claude-Opus-4.5, Gemini-3-Pro-Preview, Qwen3-235B-A22B, DeepSeek-V3.2-Exp) under both ReAct and Plan-and-Execute on the full 100,000 episodes, reporting per-model framework effects on FPR, DR, IT, and OT. The key question: is the 35.38% IT overhead of ReAct a property of the framework (every model experiences similar overhead) or of the models typically evaluated under ReAct (some models are inherently more verbose)? If the overhead is framework-driven, the cost-benefit calculus for deployment is clear. If it is model-driven, then model selection and framework selection interact in ways the current data cannot reveal. A further refinement: report per-scenario framework effects, testing whether the Plan-and-Execute advantage on Preference-Constrained Planning replicates within-model or is an artifact of which models happened to be evaluated under which framework.

Thinking-mode token-overhead quantification with cost-normalized metrics. The paper's Figure 4 shows FPR gains from enabling thinking mode (DeepSeek-R1: 70.46%; Qwen-30B-A3B: +5.98% absolute), but provides no token-overhead data, only the qualitative claim that overhead is "markedly higher" and makes real-time deployment "challenging." A direct follow-up would re-run the thinking vs. non-thinking comparison from Section 4.2.3, reporting IT and OT for each model in both modes, and compute an FPR-per-1K-output-tokens metric. This would transform a qualitative tension into a quantitative tradeoff. The null hypothesis to test: the FPR gain per additional output token is constant across models (thinking provides a fixed return on reasoning investment). The alternative: some models (perhaps DeepSeek-R1, purpose-built for reasoning) achieve higher FPR-per-token than general-purpose models with thinking enabled. If the alternative holds, it implies that reasoning architecture matters as much as reasoning budget—a finding with direct implications for model selection in cost-sensitive deployments. A further extension: measure wall-clock latency in addition to token counts, since thinking mode's serial generation of reasoning tokens may increase latency more than token count alone suggests.

Cross-domain replication of the framework-inversion finding. The paper's most surprising result—that Plan-and-Execute outperforms ReAct on Preference-Constrained Route Planning—is a task-structure-dependent framework inversion. Whether this generalizes to other constraint-satisfaction domains is unknown. A replication study would port the framework comparison to a structurally analogous domain where tasks vary in constraint rigidity: code generation with explicit constraints (e.g., "write a function that sorts a list without using built-in sort, with O(n log n) complexity"), database query generation (where the schema imposes rigid structure), or itinerary planning (TravelPlanner, which the paper identifies as a higher-abstraction-level benchmark). For each domain, the study would define a "structured" task subset (clear logical ordering, explicit constraints) and an "ambiguous" task subset (underspecified requirements, multiple valid approaches), then evaluate ReAct and Plan-and-Execute agents on both. The prediction from MobilityBench: ReAct dominates on ambiguous tasks, Plan-and-Execute dominates on structured tasks. If this replicates across domains, it establishes a general principle for agent architecture selection. If it fails to replicate, it suggests the inversion is specific to spatial constraint satisfaction (where intermediate API responses can mislead more than they inform), which would itself be an important boundary condition.

Replay sandbox adoption for other non-deterministic API domains. The paper's replay sandbox is domain-specific (Amap mapping APIs) but the design pattern—cache responses at collection time, canonicalize arguments, apply fuzzy matching with distance thresholds, treat unresolvable calls as tool-use failures—is portable. A natural follow-up is to build replay-based evaluation environments for other non-deterministic service domains: financial data APIs (stock prices, exchange rates that change intraday), weather services (forecasts that update hourly), or code execution environments (where external package versions affect outputs). For each domain, the engineering challenge is defining appropriate canonicalization (time windows? version pinning? coordinate precision?) and fallback strategies (what is the analog of "nearest-neighbor spatial matching" for financial data?). The paper's decision to make tool-use failures visible in evaluation metrics (calls that cannot be resolved are not silently dropped but counted against schema compliance or delivery rate) is a design choice that replication studies should preserve: it ensures that agents are incentivized to produce precise, cacheable tool calls rather than relying on the sandbox's tolerance mechanisms to paper over sloppiness.

Constraint-extraction targeted fine-tuning to close the open-source instruction understanding gap. The paper identifies instruction understanding as the remaining dimension where closed-source models maintain "a clear lead" (Claude-Sonnet-4.5 and Gemini-3-Pro-Preview at ~90% vs. lower open-source scores), even as open-source FPR converges to within ~2 percentage points. This suggests a targeted intervention: fine-tune Qwen3-235B-A22B (or another strong open-source model) specifically on constraint extraction from mobility queries, using the 100,000 episodes' ground-truth constraint sets as training data. The experiment: (1) extract constraint sets from a held-out portion of the benchmark using the fine-tuned model; (2) measure IE score improvement vs. the base open-source model; (3) measure whether the IE improvement cascades into FPR improvement (i.e., does better constraint extraction actually translate to more correct final routes?). If IE improves substantially but FPR does not, the bottleneck is downstream from constraint understanding (planning or tool use). If FPR improves proportionally, constraint extraction is the binding constraint—and fine-tuning it could push open-source models past the 69% FPR ceiling currently held by Gemini-3-Pro-Preview, potentially with lower inference cost.

Practical Applications and Downstream Use Cases

Navigation service agent evaluation and model selection. Organizations deploying LLM-based agents in mapping and navigation services (Amap itself, Google Maps, third-party logistics platforms) face a concrete decision: which model and which execution framework to use for processing natural-language route-planning requests. MobilityBench enables this decision to be made with data rather than intuition. The benchmark's finding that Qwen3-235B-A22B achieves 66.69% FPR under ReAct at DR 85.95%, compared to Gemini-3-Pro-Preview's 69.09%, means that an organization willing to accept a 2.4 percentage point FPR reduction can deploy an open-source model (eliminating per-query API costs, enabling on-premise hosting, and allowing fine-tuning on proprietary data) with near-equivalent performance. The efficiency metrics (35.38% IT overhead for ReAct over Plan-and-Execute) provide quantitative input for capacity planning: a deployment processing 10 million queries per day under ReAct will consume approximately 35% more input tokens than under Plan-and-Execute, directly translating to higher infrastructure costs. If latency is the primary constraint (as in real-time navigation), the benchmark's scenario study suggests Plan-and-Execute may be preferable for structured queries even at a modest FPR cost, since it avoids ReAct's iterative context accumulation. These are not hypothetical tradeoffs—they are operational decisions that the benchmark's multi-dimensional metrics enable organizations to make with quantified costs and benefits.

Targeted improvement of deployed agent pipelines. For teams that have already deployed route-planning agents, the diagnostic metrics provide a roadmap for where to invest engineering effort. If an agent achieves high DR (it completes most queries) but low FPR (its outputs are often wrong), and the bottleneck is in IE (information extraction) rather than in planning or tool use, the fix is not a better planning algorithm—it is improved constraint extraction, perhaps through fine-tuning on the benchmark's ground-truth constraint sets or through better prompt engineering for slot filling. If the bottleneck is in TS-P (tool selection precision—the agent misses required tools), the fix is improved API documentation in the system prompt or better function-calling training. If SC (schema compliance) is the bottleneck, the fix is stricter output format constraints or post-generation validation of tool call parameters. Prior to MobilityBench, diagnosing which of these bottlenecks was active required manual inspection of agent traces—a process that does not scale to 100,000 queries. The benchmark automates this diagnosis, enabling data-driven improvement prioritization. The paper's experimental results already demonstrate this in action: the scenario study (Section 4.2.2) shows that Preference-Constrained Planning degrades across all dimensions, not just final pass rate—meaning that improving performance on this task family requires simultaneous work on instruction understanding, planning, and tool use, not a single-point intervention.

Open-source deployment in privacy-sensitive mobility applications. Mobility data—user locations, travel patterns, frequently visited destinations—is among the most privacy-sensitive data categories. Organizations in healthcare (patient transport), logistics (fleet routing with customer addresses), and government (emergency response routing) often cannot send mobility queries to closed-source APIs because doing so would expose protected data to third-party servers. The benchmark's demonstration that Qwen3-235B-A22B achieves 85.95% DR and 66.69% FPR—competitive with the best closed-source models—provides evidence that on-premise, open-source deployment is a viable alternative for these applications. The cost structure further favors this: closed-source API calls at scale (millions of queries per day) incur recurring per-token costs, while open-source deployment shifts cost to fixed infrastructure. The thinking vs. non-thinking results (Figure 4) add a nuance: if an organization can tolerate the latency, enabling thinking mode on open-source models (DeepSeek-R1: 70.46% FPR) pushes performance past all evaluated closed-source models, including Gemini-3-Pro-Preview at 69.09%. For batch processing scenarios (overnight route optimization, non-real-time logistics planning), the latency cost of thinking mode is acceptable and the FPR gain is substantial.

When to Prefer This Benchmark

The paper positions MobilityBench as a benchmark and evaluation infrastructure, not as a method that competes with alternatives. It does not propose a "MobilityBench agent architecture" that trades off against ReAct or Plan-and-Execute—it provides the infrastructure for comparing any agent architectures, including those not yet developed. The relevant decision is not "when to use MobilityBench vs. alternative X" but rather what evaluation methodology to adopt for route-planning agent research and deployment decisions. On that question, the paper's positioning implies clear decision rules:

Prefer MobilityBench-based evaluation when you need reproducible comparisons across agents or frameworks, when you need diagnostic information about where an agent fails (not just that it fails), when you are making deployment decisions that involve cost-efficiency tradeoffs (framework selection, thinking vs. non-thinking mode, model scale), and when your evaluation budget can accommodate the infrastructure setup (the benchmark is publicly released with documentation, but requires running the replay sandbox rather than calling live APIs).
Prefer live-API evaluation only when your research question specifically concerns agent robustness to environmental drift (how does performance degrade as traffic patterns change over months?), or when you are testing an agent's ability to handle API failures and rate limits that the deterministic sandbox intentionally excludes. The sandbox's design choice to treat unresolvable tool calls as explicit failures makes it unsuitable for studying graceful degradation under API instability—a legitimate research question, but a different one than the benchmark is designed to answer.
Prefer TravelPlanner or TravelBench when your research concerns high-level itinerary planning (multi-day trips, flight and hotel selection, abstract constraint satisfaction) rather than fine-grained route planning over real road networks. The paper explicitly positions MobilityBench as filling the route-planning abstraction gap that these benchmarks leave open; they are complementary, not competing. A comprehensive evaluation of a general-purpose travel agent should use both—MobilityBench for the route-planning component and TravelPlanner/TravelBench for the itinerary-planning component—since failures at either level produce unusable outputs.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?2025· code← LLM-judged similar
- Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving2025· evaluation← LLM-judged similar
- The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements2025· code← in-domain predecessor
- Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases2025· code← similar topic
- Large Language Models for Compiler Optimization2023· code← similar topic
Decoding ML Decision: An Agentic Reasoning Framework for Large-Scale Ranking System2026· agents← in-domain predecessor
- Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents2026· agents← in-domain predecessor
- Agentic Reasoning for Large Language Models2026· agents← similar topic
- The Script is All You Need: An Agentic Framework for Long-Horizon Dialogue-to-Cinematic Video Generation2026· agents← similar topic
TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarios2026· agents← similar topic
- Why Do Multi-Agent LLM Systems Fail?2025· agents← LLM-judged similar
- MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering2026· agents← in-domain predecessor
- WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation2026· agents← similar topic
- HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents2026· agents← similar topic
- AgentFold: Long-Horizon Web Agents with Proactive Context Management2025· agents← similar topic
- Generative Agents: Interactive Simulacra of Human Behavior2023· agents← similar topic
- Learning Interactive Real-World Simulators2023· agents← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

Proact-VL: A Proactive VideoLLM for Real-Time AI Companions2026· agents→ in-domain predecessor
- LLM2Vec-Gen: Generative Embeddings from Large Language Models2026· agents→ in-domain predecessor
MAI-UI Technical Report: Real-World Centric Foundation GUI Agents2025· agents→ similar topic
- Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases2025· code→ LLM-judged similar
- KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at Meta2025· agents→ in-domain predecessor
- AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents2024· agents→ similar topic
- UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning2025· agents→ similar topic
- Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents2025· agents→ similar topic
TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarios2026· agents→ similar topic
- SWE-Universe: Scale Real-World Verifiable Environments to Millions2026· agents→ in-domain predecessor
- Generative Agents: Interactive Simulacra of Human Behavior2023· agents→ similar topic
- WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation2026· agents→ similar topic