Evaluation2026~111 minarXiv:2603.24943

FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol

score 3Jie Zhu, Yimin Tian, Boyang Li +8Qwen DianJin↑ 12Mar 26, 2026cs.AI

🎯 Pitch

LLMs paradoxically improve their tool-use accuracy when faced with more complex financial queries. The best models achieve higher tool-calling precision and recall on hard multi-tool tasks than on simple single-tool ones, because easy queries tempt them to call unnecessary tools while harder scenarios force disciplined planning.

1. Executive Summary

This paper introduces FinMCP-Bench, a benchmark for evaluating how large language models invoke real-world financial tools through the Model Context Protocol across single-tool, multi-tool, and multi-turn task categories. Using 613 samples spanning 10 financial scenarios with 65 production MCP tools, the authors systematically assess six LLMs—including Qwen3, DeepSeek-R1, GPT-OSS-20B, and Seed-OSS-36B—on metrics that explicitly measure tool invocation accuracy (Tool Recall, Tool Precision, Tool F1, and Exact Match Rate) rather than final answer quality. The Qwen3 models generally lead, with Qwen3-30B-A3B-Thinking and Qwen3-235B-A22B-Thinking forming the top tier, though the results reveal that stronger models improve rather than degrade on harder samples—their TF1 scores rise from Easy to Hard—establishing that richer multi-tool constraints in complex queries reward balanced tool selection and planning, while easy single-tool queries penalize over-calling with lower precision.

2. Context and Motivation

The Core Problem: We Don't Know How Well LLMs Use Financial Tools in Production

The fundamental problem this paper addresses is deceptively simple: when an LLM agent is deployed in a real financial application and told to use external tools to answer user queries, how well does it actually invoke the right tools in the right order? We simply don't have a good way to measure this. Existing evaluation frameworks either ignore tool use entirely, focus on narrow isolated tasks, or evaluate final answer quality rather than the correctness of the tool invocation process itself. This gap matters enormously because financial applications are increasingly being built around LLM agents that orchestrate multiple API calls to gather information, perform calculations, and synthesize recommendations — and if these agents call the wrong tools, miss critical dependencies, or fail to chain calls properly, the financial consequences can be severe.

The paper frames this problem concretely in its opening paragraph (Section 1):

"LLM agents must understand user intentions, access financial tools to retrieve information such as stock trends, fund holdings, and market analyses, and then apply financial concepts to generate useful responses. This often requires chaining multiple tool calls together, with each step depending on the results of the previous one. Such implicit dependencies make it difficult to evaluate how well LLM agents handle realistic financial tasks."

The italicized phrase "implicit dependencies" is the crux. In a financial advisory conversation, a user might say "I want to invest 100,000 yuan for retirement." The agent needs to (1) look up the user's risk profile, (2) retrieve the user's current portfolio holdings, (3) search for suitable retirement products, (4) compare their historical returns, (5) calculate an allocation recommendation, and (6) present the result — all through tool calls where the output of step 1 determines which tools can be called in step 3. Evaluating the agent's final response (did it recommend a reasonable portfolio?) tells you nothing about whether it actually invoked the right tools in the right order. An agent could produce a plausible-sounding response by hallucinating fund names or using general knowledge, completely bypassing the tool infrastructure. Conversely, an agent could call all the right tools but produce a poor recommendation — yet its tool-use behavior would still be worth studying independently.

Why This Problem Matters: Financial Tool Use Is High-Stakes and Poorly Understood

The importance of this problem rests on three pillars:

Real-world deployment is already happening. The paper's data comes from a production system: the XiaoGu AI assistant in the Qieman APP operated by Yingmi Fund (a CSRC-approved fund management and sales company, as footnoted in Section 2.1). This is not a hypothetical research prototype — the benchmark is built from 10,000 historical interaction records where real users asked real financial questions and a production LLM agent invoked real financial tools to answer them. The paper explicitly states in Section 2.1 that logs were only retained if "the query reflects genuine financial needs" and "the problem is resolved through tool calls." This means any evaluation gap in this domain has immediate practical consequences for deployed systems serving actual customers.

Tool invocation correctness is distinct from answer correctness. The paper makes a pointed design choice in Section 3.2 that signals why this problem is distinct from prior work: they evaluate based on tools invoked, not final answers. The authors justify this with a footnote:

"In financial research and advisory scenarios, queries are inherently open-ended without standard answers; hence, evaluation focuses on tool-use capability."

This is a crucial insight. In math benchmarks, the ground truth is a number; in code benchmarks, it's passing unit tests. In financial advising, what counts as a "good" recommendation depends on market conditions, risk preferences, time horizons, and personal circumstances — there often isn't a single correct final answer. But there is a correct set of tools to invoke and a correct order in which to invoke them: if a user asks about retirement allocation, the agent should retrieve their risk profile before recommending specific funds. Evaluating whether the agent did this correctly is both more measurable and more diagnostically useful than evaluating the recommendation itself.

The Model Context Protocol introduces a new evaluation surface. The benchmark explicitly targets MCP — the Model Context Protocol introduced by Anthropic (2024) — which provides "a standardized schema for tool invocation across diverse servers" (Section 1). MCP is gaining adoption as a standard for connecting LLMs to external tools, but no existing benchmark evaluates whether LLMs can correctly use MCP-compliant financial tools. This creates a chicken-and-egg problem: tool developers need evaluation to improve their MCP implementations, but evaluation requires benchmarks that target MCP specifically. FinMCP-Bench fills this gap.

Prior Approaches and Their Shortcomings

The paper identifies a landscape of existing work that, while valuable, systematically fails to capture the tool-use dimension of financial LLM performance:

Financial benchmarks without tool use. The paper cites a substantial body of financial benchmark work (Section 1): CFBenchmark (Lei et al., 2024), CFLUE (Zhu et al., 2024), CFinBench (Nie et al., 2025), FinanceReasoning (Tang et al., 2025), FinChain (Xie et al., 2025), INVESTORBENCH (Li et al., 2024), and M3FinMeeting (Zhu et al., 2025b). These evaluate LLMs on financial language understanding, numerical reasoning, or decision-making, but critically, none of them involve tool use. An LLM that scores high on CFLUE's multiple-choice questions about financial concepts may still fail catastrophically when asked to orchestrate five API calls to retrieve a user's portfolio, look up market data, and compute a risk-adjusted recommendation. The paper's framing is direct in Section 1:

"While recent work has explored the evaluation of LLMs on general tool use, existing evaluations in the financial domain remain limited to specific tasks and typically do not involve tool use."

This is not a criticism of those benchmarks — they were designed for different purposes — but rather identifies a genuine gap: the financial domain has rich task-specific benchmarks but no tool-use benchmark.

General tool-use benchmarks that don't target finance. The paper does not cite specific general-purpose tool benchmarks by name, but the implication in their introduction is clear: general tool-use evaluations (e.g., benchmarks for function calling, API retrieval, or code execution) exist in the broader literature, but they lack the domain-specific complexity of financial tool chains. Financial tools have particular characteristics that generic benchmarks don't capture: strict dependency chains (you cannot recommend a fund without first checking whether the user is eligible to purchase it), parallelizable calls (you can retrieve stock prices and fund NAVs simultaneously), and domain-specific error modes (confusing fund codes, mismatching date ranges, or retrieving stale data). A model that performs well on generic "find the nearest restaurant" tool-use tasks may not generalize to the sequential, dependency-heavy patterns of financial tool orchestration.

Answer-focused evaluation misses tool correctness. The paper explicitly contrasts its approach with recent tool-learning work (Section 3.2), citing Tool-Star (Dong et al., 2025), ToRL (Li et al., 2025), ToolRL (Qian et al., 2025), and synthetic data generation for tool use (Goldie et al., 2025). These papers focus on reinforcement learning or behavioral cloning to improve tool use, but they evaluate based on whether the model produces the correct final answer. The paper's departure is explicit:

"Unlike previous work that focuses on the accuracy of the final answer... in this paper we evaluate LLM performance based on the tools invoked."

The rationale is that in open-ended financial scenarios, tool invocation correctness is both more measurable and more actionable than final output quality. If an agent calls the right tools in the right order but produces a mediocre summary, you can improve the summarization layer independently. If it calls the wrong tools, the entire pipeline is broken and no amount of post-hoc text polishing can fix it.

No standardized benchmark for MCP in finance. The Model Context Protocol is a relatively recent standardization effort. While it provides a common schema for tool descriptions, input/output formats, and server discovery, there is no existing benchmark — financial or otherwise — that evaluates LLMs specifically on their ability to work within the MCP paradigm. FinMCP-Bench fills this gap by tying the evaluation directly to 65 real financial MCP tools that are actually deployed in production.

How the Paper Positions Itself Relative to Existing Work

The paper's positioning is not as a competitor to existing financial benchmarks or tool-learning methods, but as a complementary evaluation layer that addresses a distinct, previously unmeasured capability. The key strategic choices that define this positioning are:

Tool invocation as the primary evaluation target, not final answers. This is the clearest differentiator. The paper argues that for financial advisory applications, evaluating tool-use correctness is more meaningful than evaluating response quality because (a) responses are open-ended and hard to judge objectively, while (b) tool chains can be compared against reference traces with exact-match-style metrics. The metrics introduced — Tool Recall, Tool Precision, Tool F1, and Exact Match Rate — are specifically designed to capture how well the model's invoked tool set and calling structure match the reference.

Real production tools and interactions, not synthetic scenarios. The paper grounds itself in data from an actual deployed financial assistant with 65 real MCP tools and 10,000 real user interactions. This is a deliberate contrast to benchmarks built from scratch with idealized tool descriptions or synthetic queries. The 65 tools range across 10 financial scenarios (Market Analysis, Investment Planning, Risk Assessment, etc.) and 33 sub-scenarios, covering a genuinely diverse surface of financial operations. The paper's synthesis methods (chain-based multi-tool generation, role-playing-based multi-turn generation) are necessary to augment the real data with harder cases, but they always start from real tool dependencies and real user query patterns.

Difficulty is defined by tool complexity, not content complexity. The paper categorizes samples by structural difficulty — single-tool, multi-tool chain length, and multi-turn conversation length — rather than by financial concept difficulty. A retirement planning query might be conceptually simple (the user knows what they want) but structurally complex (requiring 8 tool calls over 6 conversational turns), while a derivative pricing query might be conceptually hard but structurally simple (one call to a pricing function). The benchmark measures the structural dimension, which is the dimension that matters for evaluating tool orchestration capability.

Emphasis on MCP as a standard. By explicitly targeting MCP, the paper positions itself within an emerging ecosystem. MCP defines a schema that tool providers and LLM agents agree on; evaluating models on MCP-compliant tools means the results are applicable to other MCP-based systems, not just the specific Qieman deployment. This also means that FinMCP-Bench can serve as a regression test for MCP implementations: if an LLM's MCP client library changes, the benchmark can detect whether tool invocation accuracy regresses.

The difficulty-behavior paradox as an insight. One of the paper's most interesting findings — and one that speaks directly to its positioning — is that stronger models often score higher TF1 on Hard samples than on Easy ones (Figure 5). On Easy single-tool queries, models tend to over-call tools (low precision: they generate multiple tool calls when only one is needed), dragging down F1. On Hard queries with complex multi-tool chains, the models' tool selection becomes more accurate because the richer constraints of the task force better planning. This finding would be invisible in an answer-focused evaluation, and it reveals something fundamental about how current LLMs handle tool-use agency — they over-tool on simple queries and only settle into appropriate tool selection when the task demands it. This is a genuine insight that validates the paper's decision to evaluate tool invocation directly.

3. Technical Approach

3.1 Reader Orientation

FinMCP-Bench is not a model or a system — it is a benchmark dataset and evaluation protocol designed to measure how accurately large language models invoke financial tools through the Model Context Protocol. The core problem it solves is that no existing benchmark tells you whether an LLM, when deployed as a financial agent, actually calls the right tools in the right order — prior work either evaluates final answer quality (which is open-ended and subjective in finance) or ignores tool use entirely. The shape of the solution is a carefully constructed set of 613 test samples, each containing a user query paired with a reference tool invocation trace, plus a set of tool-aware metrics (Tool Recall, Tool Precision, Tool F1, Exact Match Rate) that compare a model's predicted tool calls against the reference rather than comparing generated text against ground-truth responses.

3.2 Big-Picture Architecture (Diagram in Words)

The benchmark construction pipeline has four major stages, with the evaluation protocol forming a fifth logical component:

Real Log Collection and Curation — Raw interaction logs from a production financial AI assistant (XiaoGu, deployed in the Qieman APP) are collected, anonymized, and filtered. Only logs where the user query reflects genuine financial needs, is resolved through tool calls, and receives a satisfactory response are retained. This produces 1,484 single-tool samples and 183 multi-tool samples from 10,000 initial records.
Tool Dependency Graph Construction — From the 183 real multi-tool samples, the system extracts pairwise tool dependency relationships by analyzing which tools appear in sequential versus parallel groups within each sample. An LLM (Qwen3-235B-2507) validates each candidate dependency, producing a directed graph with 65 tool nodes and 288 edges.
Synthetic Sample Augmentation — The benchmark augments the real samples with two types of synthetic data:
- Chain-based multi-tool samples: The tool dependency graph is traversed to sample tool chains; for each chain, a user query is generated using in-context examples from real single-tool samples; the query is then executed against a live MCP server to produce a reference trajectory.
- Role-playing multi-turn samples: A planner agent defines user persona and goal; Qwen3-235B-2507 plays both user and assistant in a multi-turn dialogue, invoking tools across turns.
Quality Control — A two-stage pipeline: automated validation checks that all tools execute without errors, then six financial domain experts rate each sample on a 5-point Likert scale across five dimensions (question relevance, tool-chain completeness, tool-chain logical consistency, answer reliability and traceability, and data freshness). Samples are accepted only if both of two randomly assigned reviewers score at least 4 in all dimensions.
Evaluation Protocol — For each test sample, the LLM under evaluation receives the user query and (for multi-turn samples) the gold conversation history up to the current turn. The model generates a reply from which tool calls are extracted. These predicted tool calls are compared against the reference tool calls using the four tool-aware metrics.

Information flows as follows: production logs → real single-tool and multi-tool samples → tool dependency graph → synthetic multi-tool and multi-turn samples → expert quality review → final benchmark. At evaluation time: test sample query → LLM agent → extracted tool calls → metric computation against reference tool calls.

3.3 Roadmap for the Deep Dive

We will work through the benchmark construction and evaluation in the order that traces how a single sample comes into existence and is then used for evaluation:

First, the data source and real sample curation (Section 2.1) — where the raw material comes from, what filtering criteria are applied, and how the 145 single-tool and 183 multi-tool real samples are obtained.
Second, the tool dependency graph construction (Section 2.2, first stage) — how pairwise dependencies are proposed from group-structured tool calls, how an LLM validates them, and what the resulting 65-node, 288-edge graph encodes.
Third, chain-based multi-tool sample synthesis (Section 2.2, stages two and three) — how tool chains are sampled from the graph, how user queries are generated from in-context examples, and how reference trajectories are produced by executing those queries against a live MCP server.
Fourth, role-playing-based multi-turn sample construction (Section 2.3) — how personas and goals are defined, how dialogue is simulated, and how multi-turn samples are validated and filtered.
Fifth, quality control (Section 2.4) — the two-stage expert review process, the five evaluation dimensions, the 5-point Likert scale, and the acceptance threshold.
Sixth, sample categorization and difficulty taxonomy (Section 2.5) — how samples are classified as single-tool, multi-tool, or multi-turn, and how the Easy / Medium / Hard difficulty split is defined.
Seventh, the evaluation protocol and metrics (Sections 3.1–3.2) — how models are prompted and evaluated, the exact definitions of Tool Recall, Tool Precision, Tool F1, and Exact Match Rate, and why these metrics are chosen over answer-based evaluation.

This order follows the construction pipeline from raw data to final benchmark, then covers how the benchmark is used for evaluation, which mirrors the natural dependency chain: the graph depends on the real samples, the synthetic samples depend on the graph, the quality control applies to all samples, and the evaluation protocol uses the final curated benchmark.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark construction and evaluation paper whose core idea is that tool invocation correctness — not final answer quality — should be the primary evaluation target for financial LLM agents, and that a carefully constructed benchmark combining real production data with principled synthetic augmentation can measure this capability across varying levels of task complexity.

Data Source and Real Sample Curation

The raw material for the benchmark comes from 10,000 historical interaction records of the XiaoGu AI assistant, a production financial agent deployed in the Qieman APP operated by Yingmi Fund — a CSRC-approved fund management and sales company (Section 2.1). These records capture real user queries and the assistant's responses, where the assistant follows expert-defined standard operating procedures and invokes financial tools via workflow-style processes integrated through the Model Context Protocol (MCP).

Filtering criteria. From the initial 10,000 records, logs are retained only if they satisfy three conditions, stated verbatim in Section 2.1:

"(i) the query reflects genuine financial needs, (ii) the problem is resolved through tool calls, and (iii) the final response provides a satisfactory solution."

Condition (i) excludes non-financial or chitchat queries that do not exercise financial reasoning. Condition (ii) ensures that the retained samples involve actual tool invocation — queries answered without tools are discarded, because they would not exercise the tool-use capability the benchmark targets. Condition (iii) acts as a quality filter on the reference behavior itself: only interactions where the production assistant actually succeeded are retained, establishing these traces as positive examples of correct tool use.

Output of curation. After filtering, the authors obtain two categories of real samples:

1,484 single-tool samples: queries resolved with exactly one tool call in a single conversational turn.
183 multi-tool samples: queries involving multiple tool calls within a single conversational turn, where the calls may be arranged in sequential or parallel groups.

Split for benchmark and synthesis. The 1,484 single-tool samples are randomly divided into two subsets. The first subset, denoted $\mathcal{S}$ , contains 700 samples; these are reviewed by domain experts, from which 145 high-quality single-tool samples are ultimately retained in FinMCP-Bench. The second subset, denoted $\mathcal{S}_o$ , contains 784 samples; these are reserved as seed material for synthesizing more complex multi-tool samples (Section 2.2) and multi-turn samples (Section 2.3) rather than being included in the benchmark directly.

Why this split? The paper does not explicitly justify the 700/784 split size, but the logic is inferable: the first subset needs to be large enough that expert review can yield a sufficient number of high-quality single-tool benchmark samples (145 after review), while the second subset needs to provide enough seed examples to support chain-based synthesis across all 65 tools and 33 sub-scenarios. The second subset is deliberately larger because the synthesis pipeline needs to sample single-tool examples for each tool in each sub-scenario, which requires broad coverage.

Anonymization. Section 2.1 notes that "all logs are processed through a strict anonymization and disclosure procedure," though the specifics of this procedure are not detailed. Given the financial sensitivity of the data (real user queries about investment, retirement, portfolios), this step is critical for ethical data handling but the paper treats it as a given rather than a technical contribution.

Tool Dependency Graph Construction

The tool dependency graph is the conceptual backbone that enables systematic synthesis of novel multi-tool samples — samples that require tool chains the model has not seen before, but whose dependencies are grounded in real financial workflows. The graph is built entirely from the 183 real multi-tool samples, denoted $\mathcal{M}$ , without relying on manual specification of which tools depend on which others.

Input representation. The paper models the tools invoked in any sample as a set of groups with ordering constraints. For a sample with $M$ groups, the tool invocation is represented as:

$\{(t^1_1, \dots, t^1_{n_1}), \dots, (t^M_1, \dots, t^M_{n_M})\}$

where $M$ is the number of tool groups, and the $i$ -th group contains $n_i$ tools. Tools within the same group are executed in parallel, meaning their order is interchangeable and no dependency exists between them. Tools in different groups are executed sequentially, with group $i$ completed before group $i+1$ begins.

What this representation captures: the grouping encodes the parallelism structure of real financial workflows. If two tools are in the same group — for example, retrieving a stock price and retrieving a fund NAV simultaneously — this indicates they are independent. If tool A appears in group 1 and tool B appears in group 2, B's execution followed A's completion, which suggests (but does not guarantee) a dependency: B may require A's output.

Step 1: Proposing candidate dependencies. For each multi-tool sample $M \in \mathcal{M}$ , the system examines consecutive groups. If a tool $t_j$ appears in a group that immediately follows a group containing tool $t_i$ , the system proposes a candidate dependency $t_i \rightarrow t_j$ . The paper illustrates this with a concrete example (top of Figure 2(a)): suppose three tools are arranged into two groups, $(t_1, t_2)$ and $(t_3)$ . This yields two candidate dependencies: $t_1 \rightarrow t_3$ and $t_2 \rightarrow t_3$ . No dependency is proposed between $t_1$ and $t_2$ because they appear in the same parallel group.

Why group-based rather than flat? If the system treated all tools in a sample as a flat sequence, it would incorrectly propose dependencies between parallel tools (e.g., $t_1 \rightarrow t_2$ ), which would contaminate the graph with spurious edges. The grouping structure, inherited from the production assistant's workflow design, provides a clean signal: only cross-group pairs are candidates for true dependencies.

Step 2: Validation by LLM. For each candidate dependency $t_i \rightarrow t_j$ , the system uses an LLM — specifically Qwen3-235B-2507 — to validate whether it is reasonable for $t_j$ to depend on $t_i$ . The paper states:

"The model is prompted to judge whether it is reasonable for $t_j$ to depend on $t_i$ . If validated, we add a directed edge from $t_i$ to $t_j$ in the dependency graph."

The exact prompt is not provided in the paper, but the validation task is a binary classification: given the names, descriptions, and input/output schemas of two tools (inferred from the paper's description of the tool set), does it make financial sense for the second tool's invocation to require output from the first?

Why use an LLM validator rather than manual rules? The 183 multi-tool samples contain many different tool combinations across 33 sub-scenarios. Manually encoding dependency rules for 65 tools would require domain experts to exhaustively enumerate which tools can feed into which others — an $O(n^2)$ combinatorially explosive task with 65 tools. The LLM validator leverages the model's understanding of tool semantics (what each tool does, what inputs it expects, what outputs it produces) to judge dependency plausibility without manual specification. It is an automated plausibility check, not a ground-truth labeler — it filters out nonsensical pairings (e.g., "stock price retrieval depends on retirement age calculation") while admitting pairs where the dependency could realistically occur.

Output: the dependency graph $\mathcal{G}$ . The final graph contains 65 nodes (one per tool) and 288 edges (validated dependencies). With 65 nodes, the maximum possible number of directed edges is $65 \times 64 = 4,160$ , so 288 edges represents a density of approximately 6.9% — a sparse graph consistent with the idea that most financial tools operate independently and only specific pairs have genuine input-output dependencies.

What the graph enables. Once constructed, any path through the graph from tool $t_i$ to tool $t_j$ defines a valid tool chain: a sequence of tools where each step's output can plausibly serve as input to the next step. This converts the problem of "generate a realistic multi-tool query" from an unbounded creative task into a constrained sampling task: sample a path through the graph, then generate a query that would require following that path.

Chain-Based Multi-Tool Sample Synthesis

With the tool dependency graph constructed, the paper synthesizes additional multi-tool samples that require longer tool chains than those found in the real data. The process has three stages, illustrated in Figure 2 (though the figure's sub-panels are referenced inconsistently — panels a, b, and c are shown but the text refers to top and bottom of (a), top of (c), and bottom of (b), so I will describe the logical sequence).

Stage 1: Sampling tool chains from the graph. The synthesis begins not by writing queries, but by selecting which tools the query should require. For each financial sub-scenario, the system samples tool pairs $(t_i, t_j)$ from the complete tool set $\mathcal{T}$ , with the constraint that both $t_i$ and $t_j$ must appear in real multi-tool samples $\mathcal{M}$ associated with that sub-scenario. This constraint is crucial: it ensures that even the "novel" synthesized chains involve tools that are genuinely used together in real financial workflows, preventing the generation of queries that, while graph-theoretically valid, would never arise in practice (e.g., chaining a retirement calculator to a cryptocurrency price lookup).

If multiple paths exist between $t_i$ and $t_j$ in the dependency graph $\mathcal{G}$ , the system randomly selects one. This introduces diversity: the same tool pair can generate different chains on different sampling iterations, depending on which intermediate tools the random path traverses. The selected path becomes the tool chain:

$\mathcal{C} = \{c_1, \dots, c_n \mid c_i \in \mathcal{T}\}$

where $n$ is the number of tools in the chain and each $c_i$ is a specific tool.

Stage 2: User query generation. Given a tool chain $\mathcal{C}$ with $n$ tools, the system needs a user query that would naturally require invoking those $n$ tools in that order. It generates this query by using in-context learning with real single-tool examples.

For each tool $c_i$ in the chain, the system randomly selects a single-tool sample $s_i$ from the reserved subset $\mathcal{S}_o$ (the 784 real single-tool samples not used in the benchmark) such that $s_i$ involves tool $c_i$ . This produces a set of $n$ in-context examples $\{s_1, \dots, s_n\}$ . Each $s_i$ is a real user query paired with a correct single-tool invocation of $c_i$ .

The system then prompts Qwen3-235B-2507 with these $n$ examples and instructs it to generate a new user query that would require invoking all $n$ tools in the specified chain order. The paper does not provide the exact prompt template, but the implicit task is: given examples of queries that each require one specific tool, compose a query that stitches these requirements together into a coherent financial request where tool $c_1$ must be called first (it provides some output), then tool $c_2$ uses that output, and so on through $c_n$ .

Why in-context examples from real queries? The synthesis goal is to generate queries that sound like real users, not like benchmark designers writing test cases. By providing real user queries as examples for each individual tool, the model sees the vocabulary, phrasing patterns, and level of financial sophistication that actual users exhibit. The model's task is compositional — it must combine $n$ atomic query patterns into a single coherent query — rather than generative from scratch, which would risk producing queries that sound artificial or use unnatural combinations of financial concepts.

Stage 3: Trajectory generation. The generated user query is not directly added to the benchmark. Instead, it is executed against a live MCP server to produce a reference trajectory. Specifically, Qwen3-235B-2507 — the same model that generated the query — is connected to Qieman's MCP server and prompted to resolve the query by invoking the available tools. The paper states:

"Qwen3-235B-2507, connected to Qieman's MCP server, is then used to generate the corresponding trajectory for each user query."

The resulting trajectory is the sequence of tool calls the model actually made, including their parameters, outputs, and ordering. This trajectory serves as the reference for evaluation, replacing the idealized tool chain $\mathcal{C}$ that was used to generate the query.

Acceptance criteria for the trajectory. A query-trajectory pair is retained as a multi-tool sample only if it meets two conditions, stated explicitly in Section 2.2:

"(i) may include additional tools beyond those in $\mathcal{C}$ , and (ii) correctly preserves the dependency relations specified in $\mathcal{C}$ ."

Condition (i) is pragmatic: the executing model might discover that resolving the query requires auxiliary tools not in the original chain (e.g., a query generated for a fund comparison chain might also require a risk-profile lookup). These additions are acceptable because the goal is a realistic reference trajectory, not a minimal one. Condition (ii) is the fidelity check: the core dependency structure of the chain must be preserved. If the executing model calls tool $c_2$ before $c_1$ (violating the dependency $c_1 \rightarrow c_2$ ), the sample is rejected because the reference trajectory would no longer reflect the intended dependency structure.

Why generate trajectories by execution rather than by construction? An alternative approach would be to construct the reference trajectory deterministically from the chain $\mathcal{C}$ , specifying exactly which tools are called with which parameters. The paper's execution-based approach has two advantages. First, it approximates real-world dynamics: the executing model may call tools with slightly different parameters or in slightly different groupings than the chain specifies, and the resulting trajectory captures this variance rather than an idealized template. Second, it verifies that the generated query is actually resolvable — if the query cannot be answered by invoking the available tools (because, for example, the synthesized query combines tools whose outputs are incompatible), the execution will fail, and the sample will be rejected. This acts as an additional quality filter.

Scale and yield. The authors generate 1,000 query-trajectory pairs. From these, 496 are retained as valid multi-tool samples. Combined with the 183 real multi-tool samples, the candidate pool is 679 multi-tool samples. After expert review (described in Section 2.4), 249 high-quality multi-tool samples are ultimately retained in FinMCP-Bench. The yield from synthetic generation is roughly 50% (496 out of 1,000 generated pairs pass the automatic validity check), and the yield from expert review of the combined pool is roughly 37% (249 out of 679 candidates survive expert review), indicating that the quality bar is high and that a substantial fraction of generated content — even when automatically valid — does not meet human expert standards.

Role-Playing-Based Multi-Turn Sample Construction

Multi-turn samples differ from single-tool and multi-tool samples in that tools are invoked over several rounds of dialogue between a user and an assistant. This requires modeling not just tool-chain dependencies, but also conversational dynamics: a user might ask a follow-up question based on the assistant's previous response, refine their request, or introduce a new sub-goal mid-conversation.

Dialogue framework: Planner Agent. The construction is orchestrated by a planner agent that specifies two components for each dialogue, illustrated on the left side of Figure 3:

Component 1: User Persona. The planner samples a user persona from a character profile pool introduced in prior work by Zhu et al. (2025a). The paper describes this pool as providing:

"a comprehensive template for financial customers. The template includes attributes such as age, gender, and income level, all of which are highly relevant in real-world financial contexts."

The persona serves to ground the conversation in a realistic user profile. A 25-year-old with low income and a 60-year-old approaching retirement will ask fundamentally different financial questions, use different vocabulary, and have different goals. The persona template ensures that the synthesized conversations reflect this demographic diversity rather than defaulting to a generic "financially literate" user.

Component 2: User Goal. The planner selects a sub-scenario (from the 33 available in the benchmark's taxonomy) and, together with the chosen persona, prompts Qwen3-235B-2507 to generate a corresponding user goal. The goal is a high-level description of what the user wants to achieve through the conversation — for example, "develop a retirement investment plan with a moderate risk tolerance, starting from an assessment of current savings and expected expenses." The combination of persona and goal creates a coherent character-motivation pairing: a specific type of person wants a specific type of financial outcome, which drives the dialogue forward.

Dialogue simulation: Dual-role LLM. Once the persona and goal are defined, the entire dialogue is simulated by assigning Qwen3-235B-2507 to play both roles — user and assistant — as illustrated on the right side of Figure 3. The user role, conditioned on the persona and goal, generates utterances that progressively advance toward the goal across multiple turns. The assistant role, connected to the Qieman MCP server, generates responses by invoking tools as needed to address each user utterance.

Why dual-role simulation rather than human-written dialogues? Collecting real multi-turn dialogues from production logs is straightforward (they exist in the 10,000 interaction records), but those real dialogues tend to be relatively simple in their tool-use patterns — most real users don't engage in 6-turn conversations requiring 8 different tools. The role-playing approach allows the authors to systematically construct dialogues with higher tool complexity than would naturally occur in logs, while still grounding each dialogue in realistic persona-goal pairs. The dual-role LLM approach (the same model plays both sides) is common in dialogue synthesis because it avoids the coordination problem of having two separate models that may not stay mutually consistent.

Filtering and quality. The authors generate 500 multi-turn dialogues as candidate samples. A first-pass automated filter uses Qwen3-235B-2507 to check validity, "with a focus on whether all user queries in each dialogue are successfully addressed" (Section 2.3). This reduces the pool from 500 to 378 dialogues. These 378 are then manually reviewed by financial experts, from which 219 are retained in FinMCP-Bench. The yield at the human review stage (219 out of 378, or roughly 58%) is higher than for multi-tool samples (37%), suggesting that multi-turn dialogues are easier to synthesize with acceptable quality — possibly because the dialogue format provides more context and constraints that guide both the generation and the evaluation.

Quality Control

The quality control pipeline is a two-stage process applied to all samples — real single-tool, real multi-tool, synthetic multi-tool, and synthetic multi-turn — before inclusion in the final benchmark. The pipeline involves both automated checks and expert human review.

Stage 1: Automated validation. An automated validator checks whether all tools in a sample's reference trajectory execute successfully without errors. A trajectory where a tool call fails (returns an error code, times out, or produces malformed output) is rejected. This catches synthesis failures where the generated query leads to tool invocations that cannot be completed — for example, a query that asks for a fund's 10-year return when the fund has only existed for 3 years, causing the historical-returns tool to fail.

Stage 2: Expert review. Six domain experts and experienced developers in the financial field serve as reviewers. Each sample is independently evaluated by two randomly assigned experts, who score it on a 5-point Likert scale across five dimensions. The paper cites Joshi et al. (2015) for the Likert scale methodology but specifies the five dimensions explicitly in Section 2.4:

Question relevance: Does the user query reflect a genuine, realistic financial need that would arise in practice?
Tool-chain completeness: Does the reference trajectory invoke all tools necessary to fully address the user query, or are there missing steps?
Tool-chain logical consistency: Does the ordering and grouping of tool calls make logical sense? Are dependencies respected?
Answer reliability and traceability: Is the assistant's final response reliable (factually correct given the tool outputs) and traceable (can you follow how each claim in the response is supported by specific tool outputs)?
Data freshness: Is the data retrieved by the tools appropriately current, or does the sample reference stale/outdated information?

Acceptance threshold. A sample is accepted only if both reviewers assign a score of at least 4 (on the 1–5 Likert scale) in all five dimensions. This is a strict conjunctive threshold — a sample that scores 5 on four dimensions but 3 on one dimension is rejected. When two reviewers disagree (one accepts and one rejects), the sample is "resolved through discussion" (Section 2.4), though the paper does not specify the resolution protocol (e.g., whether a third reviewer is brought in, or whether the two reviewers negotiate a consensus score).

Why five dimensions? Each dimension addresses a different failure mode that could make a sample unfit for benchmarking. Question relevance catches queries that are nonsensical or too artificial. Tool-chain completeness catches trajectories that are incomplete — where the assistant answered the query but missed a necessary information-gathering step, making the reference trajectory a poor example of "correct" tool use. Logical consistency catches trajectories that use tools in the wrong order, which would penalize models that get the order right (since they wouldn't match the flawed reference). Answer reliability and traceability catch trajectories where the assistant's response is wrong despite correct tool invocation — the benchmark evaluates tool use, but it still needs the reference to represent a correct interaction. Data freshness catches samples where tool outputs reference time-sensitive financial data that has since changed, which would make the sample unanswerable or misleading if the underlying data is no longer available.

Sample Categorization and Difficulty Taxonomy

Once all samples have passed quality control, the benchmark is organized along two axes: sample type (the structural category) and difficulty (the tool-complexity category).

Sample type categories. Every sample is classified into one of three types based on its conversational and tool-use structure:

Single-tool (145 samples): The user query is resolved with exactly one tool call in one conversational turn. These are the simplest structural category. All 145 single-tool samples come from the real, expert-reviewed subset $\mathcal{S}$ — no synthetic single-tool samples are generated, because the synthesis pipelines target higher complexity.
Multi-tool (249 samples): The user query involves multiple tool calls within a single conversational turn. The calls may be arranged in sequential groups (tool B depends on tool A's output) or parallel groups (tools B and C can be called simultaneously). These samples come from both real logs (183 samples passed through expert review, reduced to some fraction of the final 249) and the chain-based synthesis pipeline (496 automatically valid samples, reduced to some fraction after expert review).
Multi-turn (219 samples): The interaction spans multiple conversational turns, where each turn may involve one or more tool calls. These samples come exclusively from the role-playing synthesis pipeline, filtered from 500 candidates down to 378 by automated validation, then down to 219 by expert review.

Tool invocation representation. Section 2.5 formalizes the tool invocation in any sample as:

$\{(t^1_1, \dots, t^1_{n_1}), \dots, (t^M_1, \dots, t^M_{n_M})\}$

where $M$ is the number of tool groups, and group $i$ contains $n_i$ tools invoked in parallel. Single-tool samples are the special case $M = 1$ and $n_1 = 1$ .

What this formalism implies for evaluation: tools within the same group can be permuted without changing the meaning of the reference — the order $(t^1_1, t^1_2)$ is equivalent to $(t^1_2, t^1_1)$ because they are parallel. Tools across groups cannot be permuted — calling a tool from group 2 before group 1 would violate the dependency structure. The Exact Match Rate metric (defined in Section 3.2) leverages this grouping: it checks whether the predicted tool organization matches the reference, ignoring within-group order but requiring cross-group order to match.

Difficulty taxonomy. The paper defines difficulty based on the total number of tool calls in a sample, not on the financial complexity of the query content (Section 2.5):

Easy: samples with up to 5 tool calls.
Medium: samples with 6 to 10 tool calls.
Hard: samples with more than 10 tool calls.

This is a purely structural difficulty measure: a conceptually trivial query that requires chaining 12 simple tool calls is "Hard," while a conceptually sophisticated query answered with 3 complex tool calls is "Easy." The justification is implicit: the benchmark measures tool orchestration capability, and orchestration complexity scales primarily with the number of tools that must be coordinated, not with the financial sophistication of each individual tool's function.

Dataset statistics (Table 1). Multi-tool samples contain on average 7.32 tool calls across 5.72 steps (where a "step" is a tool group — a set of parallel invocations executed together). Among the 249 multi-tool samples, 73 include parallel calls, meaning the remaining 176 are purely sequential chains. Multi-turn samples span on average 5.95 conversational turns and invoke 5.00 tools on average.

Evaluation Protocol

The evaluation protocol specifies how an LLM is tested on FinMCP-Bench and how its performance is measured. The protocol treats the LLM as an agent that receives a conversation context and must produce a reply that includes tool invocations.

Prompting format. Section 3.1 defines the conversation representation. A multi-turn conversation with $n$ turns is denoted:

$\{(u_1, r_1), \dots, (u_n, r_n)\}$

where $u_i$ is the user utterance at turn $i$ and $r_i$ is the reference (gold) agent reply at turn $i$ , which includes both tool calls and natural language responses.

For each turn $i$ , the model under evaluation is prompted to generate a reply $r_i'$ given two pieces of context:

The current user utterance $u_i$ .
The gold conversation history up to (but not including) turn $i$ : $\{(u_1, r_1), \dots, (u_{i-1}, r_{i-1})\}$ .

For single-tool and multi-tool samples (which are single-turn), $n = 1$ , so there is no conversation history — the model sees only the user query $u_1$ . For multi-turn samples, the model sees the gold history of previous turns (with the correct tool calls and responses that the reference agent produced) and must generate the turn- $i$ reply.

Why gold history rather than generated history? If the model were given its own generated history from previous turns, errors would compound: a wrong tool call in turn 1 would produce incorrect context for turn 2, making evaluation of turn-2 tool-use capability confounded by turn-1 errors. By providing gold history, the protocol isolates the model's tool-use capability at each turn independently — the model is always evaluated on "given that everything up to now was done correctly, can you do the next step correctly?" This is a standard technique in dialogue evaluation to avoid cascading error artifacts.

Tool extraction. From each generated reply $r_i'$ , the system extracts the tools the model invoked. The paper does not specify the extraction mechanism (e.g., whether tool calls are formatted in a parseable schema like XML tags or function-call JSON, or whether they must be heuristically extracted from free text). Given that the benchmark targets MCP — which defines a standardized schema — the extraction likely relies on MCP's structured invocation format, but this is not explicitly stated.

Aggregation across turns. For multi-turn samples, the final set of predicted tool calls is the union (or sequence) of tools extracted across all $n$ generated replies. The metrics compare this accumulated prediction against the accumulated reference across all turns.

Evaluation Metrics: Tool Recall, Tool Precision, Tool F1, Exact Match Rate

The paper introduces four metrics specifically designed to measure tool invocation correctness rather than response quality. Their definitions appear in Section 3.2.

Preliminary: reference and prediction tool sets. For each sample, the system constructs two tool sets:

Reference tool set: the set of all tools that appear in the reference trajectory (across all turns for multi-turn samples). Dependencies and groupings are ignored when constructing this set — it is a flat collection of tool identifiers.
Predicted tool set: the set of all tools extracted from the model's generated replies (across all turns).

Metric 1: Tool Recall (TR). Defined as:

$\text{TR} = \frac{|\text{reference tool set} \cap \text{predicted tool set}|}{|\text{reference tool set}|}$

where $|\cdot|$ denotes set cardinality and $\cap$ denotes set intersection.

What it computes: the fraction of required tools that the model actually called. If the reference requires tools {A, B, C, D} and the model calls {A, B, E}, the recall is $2/4 = 0.5$ — the model found half of the necessary tools.

Why recall matters: a low recall means the model is missing tools that the reference trajectory needed. In a financial context, this corresponds to skipping steps — failing to look up the user's risk profile before recommending funds, or failing to check fund eligibility. High recall is necessary (but not sufficient) for correct tool use.

Metric 2: Tool Precision (TP). Defined as:

$\text{TP} = \frac{|\text{reference tool set} \cap \text{predicted tool set}|}{|\text{predicted tool set}|}$

What it computes: the fraction of the model's called tools that are actually in the reference. If the model calls {A, B, E} and the reference requires {A, B, C, D}, the precision is $2/3 \approx 0.667$ — two-thirds of the model's tool invocations were relevant.

Why precision matters: a low precision means the model is over-calling tools — invoking tools that are not needed for the query. This corresponds to unnecessary API calls (wasting compute and potentially retrieving irrelevant or confusing information) or hallucinating tool invocations. The paper observes exactly this failure mode on single-tool samples (Section 3.3): "Tool Precision (TP) is lower for single-tool samples, since models often over-predict by generating multiple tools even when only one is needed."

Why evaluate both recall and precision separately rather than just F1? They diagnose different failure modes. A model with high recall and low precision is an over-caller — it tries many tools and finds the right ones, but also invokes irrelevant ones. A model with high precision and low recall is a minimalist — it only calls tools it is confident about, but misses necessary steps. These have different implications for deployment: an over-calling model wastes API budget and may confuse users with irrelevant information; a minimalist model produces clean but incomplete responses.

Metric 3: Tool F1 (TF1). Defined as their harmonic mean:

$\text{TF1} = \frac{2 \times \text{TR} \times \text{TP}}{\text{TR} + \text{TP}}$

What it computes: the harmonic mean of Tool Recall and Tool Precision. The harmonic mean is used rather than the arithmetic mean because it penalizes extreme imbalance — a model with TR = 1.0 and TP = 0.1 (calls all the right tools but also 9× as many wrong ones) gets an arithmetic mean of 0.55 but a harmonic mean of only $2 \times 1.0 \times 0.1 / (1.0 + 0.1) \approx 0.18$ .

Why harmonic mean: in tool use, balanced performance matters. A model that achieves high recall by calling every available tool on every query is not useful — the precision penalty should dominate. The harmonic mean enforces that both recall and precision must be reasonably high for the overall score to be high.

Metric 4: Exact Match Rate (EMR). Unlike TR, TP, and TF1, which treat tool invocations as a flat set ignoring ordering and grouping, EMR checks whether the predicted tool organization exactly matches the reference organization. The paper defines EMR as "the proportion of predictions that exactly match the reference organization," with the clarification that "since tools within the same group can be invoked in parallel, their internal order is ignored."

What it computes: for each sample, EMR is binary — 1 if the predicted tool invocation structure $\{(t'^1_1, \dots, t'^1_{n'_1}), \dots, (t'^{M'}_1, \dots, t'^{M'}_{n'_{M'}})\}$ is identical to the reference structure $\{(t^1_1, \dots, t^1_{n_1}), \dots, (t^M_1, \dots, t^M_{n_M})\}$ after ignoring within-group order, and 0 otherwise. The reported EMR is the average over all samples.

What exact match requires: it requires (a) the same number of groups $M' = M$ , (b) each group $i$ contains the same set of tools as the reference group $i$ (regardless of order within the group), and (c) no extra groups or tools appear. This is a strict metric — a model that calls all the right tools but splits them into three groups instead of the reference's two groups gets EMR = 0, even though its set-wise TF1 would be 1.0.

Why include EMR in addition to set-based metrics? EMR captures dependency and parallelism structure that set-based metrics ignore. A model that calls tools A and B in parallel when B actually depends on A's output gets perfect TR, TP, and TF1 — it called the right tools — but the EMR is 0 because the grouping is wrong. In production, calling dependent tools in the wrong order causes failures (B is called without A's required output) that set-level metrics would not detect. EMR provides a stricter, more diagnostic signal about whether the model understands not just which tools to call, but how to organize their invocation.

Why evaluate tool invocation rather than final answer? The paper's rationale (Section 3.2 and its footnote 2) is explicit:

"In financial research and advisory scenarios, queries are inherently open-ended without standard answers; hence, evaluation focuses on tool-use capability."

This is a pragmatic decision grounded in the nature of the domain. Financial advice does not have a single ground-truth answer — two different advisors could produce equally valid recommendations using the same tools and data. What can be objectively evaluated is whether the agent followed the correct process: did it retrieve the necessary information, perform the required calculations, and consider the relevant constraints? The tool invocation trace is a process-level ground truth that exists even when output-level ground truth does not.

Models evaluated. Section 3.1 lists six models:

Three from the Qwen3 family (Yang et al., 2025): Qwen3-4B-Thinking, Qwen3-30B-A3B-Thinking (a 30B-parameter model with 3B active parameters via mixture-of-experts, denoted "A3B"), and Qwen3-235B-A22B-Thinking (a 235B-parameter model with 22B active parameters). All are "Thinking" variants, suggesting they use chain-of-thought reasoning.
Three additional models: DeepSeek-R1 (DeepSeek-AI, 2025), GPT-OSS-20B (OpenAI, 2025), and Seed-OSS-36B (ByteDance Seed Team, 2025).

All models are evaluated with the same protocol: given gold history and current user utterance, generate a reply, extract tools, compute metrics. The paper does not mention any model-specific prompting or fine-tuning — the evaluation is zero-shot tool use based on the models' existing instruction-following and function-calling capabilities.

Inference details. Section 3.1 specifies that single-tool and multi-tool samples are treated as one-turn conversations (the prompt is just the user query, and the model produces one reply with tool calls), while multi-turn samples are evaluated turn-by-turn with gold history.

Synthesis of Design Choices and Their Justifications

The benchmark's design reflects a coherent philosophy about what matters for financial tool-use evaluation and how to construct samples that measure it validly. Here is a synthesis of the key design choices and their rationales:

Tool invocation as the evaluation target rather than answer quality. This is the single most consequential design choice. It reflects the insight that financial advisory tasks have open-ended outputs but structured, objectively evaluable processes. A reference tool trace is a ground truth that can be constructed once (by experts validating a production-quality trace) and used for exact-match-style evaluation, whereas reference answers would require ongoing expert judgment and would be sensitive to market conditions, user preferences, and other subjective factors.

Grounding in real production data while augmenting with synthesis. The benchmark could have been built entirely from real logs or entirely from synthetic generation. The hybrid approach addresses the weaknesses of each extreme: real logs provide authentic user language and tool-use patterns but skew toward simpler interactions (real users typically don't ask questions requiring 12-step tool chains); synthetic generation can produce arbitrarily complex samples but risks creating artificial or nonsensical queries. By using real logs for the single-tool core and as in-context examples for synthesis, and by constraining synthesis to tool chains validated against real dependency patterns, the benchmark balances authenticity with coverage of complex tool-use scenarios.

Tool dependency graph as a synthesis scaffold. The graph is the key mechanism that prevents synthetic multi-tool samples from being arbitrary concatenations of tools. By extracting dependencies from real multi-tool traces and validating them with an LLM, the graph encodes which tool combinations make financial sense. This constrains the synthesis space to plausible chains — the system samples paths through the graph rather than enumerating all $65^n$ possible tool sequences, of which the vast majority would be nonsensical.

Execution-based trajectory generation rather than template-based construction. By having an LLM actually execute the generated query against a live MCP server to produce the reference trajectory, the benchmark verifies two things that template construction would not: (1) the query is actually answerable with the available tools, and (2) the trajectory reflects realistic tool-use dynamics (including the possibility of calling additional helper tools beyond the planned chain). This makes the reference trajectories more faithful to what a competent agent would actually do.

Strict conjunctive quality threshold (≥4 on all five dimensions from both reviewers). This is an unusually high bar for benchmark inclusion. It means samples are rejected for any significant flaw, not just for being "mostly good." The five dimensions cover different aspects of quality, and the conjunctive nature means a sample with excellent question relevance and logical consistency but poor data freshness is rejected. This ensures that all benchmark samples are genuinely high-quality across all relevant dimensions, but it also means the final benchmark size (613 samples) is smaller than it would be under a more permissive threshold — a deliberate trade-off of quantity for quality.

Difficulty defined by tool count rather than content complexity. This choice focuses the benchmark on what it is designed to measure: tool orchestration capability. Financial concept difficulty (understanding derivatives vs. understanding savings accounts) is orthogonal to tool-use difficulty (calling 2 tools vs. calling 12 tools) and is better measured by existing financial benchmarks that target knowledge and reasoning. By defining difficulty structurally, FinMCP-Bench isolates the capability it is uniquely positioned to evaluate.

4. Key Insights and Innovations

Innovation 1: Tool Invocation Correctness as a First-Class Evaluation Target, Decoupled from Answer Quality

The paper's most distinctive conceptual move is its decision to evaluate LLM agents based on whether they call the right tools in the right order, rather than on whether they produce correct final answers. This is not a minor metric tweak — it is a fundamental reframing of what "good performance" means for a financial agent, and it addresses a genuine failure of the dominant evaluation paradigm.

To understand why this is a conceptual innovation rather than a convenience, consider the implicit assumption in virtually all prior tool-use evaluation work. Benchmarks like Tool-Star (Dong et al., 2025), ToRL (Li et al., 2025), ToolRL (Qian et al., 2025), and the synthetic tool-use datasets of Goldie et al. (2025) all evaluate tool-augmented models by checking whether the final answer matches a ground truth — did the model compute the right number, retrieve the correct fact, or select the right option? This paradigm assumes two things: (1) a ground-truth final answer exists and can be objectively verified, and (2) final answer correctness is a valid proxy for tool-use correctness. Both assumptions break down in financial advisory settings, where queries are inherently open-ended. Two different financial advisors, given the same tools and the same client data, can produce different but equally valid investment recommendations. There is no single "correct" final answer to grade against.

The paper recognizes this and makes the counterintuitive leap: even when you cannot judge whether the answer is correct, you can still judge whether the process was correct. The process — the sequence of tool invocations — encodes the agent's understanding of what information is needed, in what order it must be gathered, and how it should be synthesized. A reference tool trace is a ground truth about process that exists independently of any ground truth about outcome. Section 3.2's footnote 2 makes this logic explicit:

"In financial research and advisory scenarios, queries are inherently open-ended without standard answers; hence, evaluation focuses on tool-use capability."

This is significant beyond the financial domain. It suggests an evaluation methodology for any domain where output quality is subjective, multi-dimensional, or context-dependent — legal advice, medical triage, educational tutoring, creative collaboration — that has been largely unexplored. The implicit prior consensus was that tool-use evaluation requires answer-level ground truth, which restricts evaluation to closed-form domains (math, coding, factual QA). FinMCP-Bench demonstrates that process-level evaluation is viable and diagnostically valuable even when output-level evaluation is not.

The evidence that this reframing captures phenomena invisible to answer-based evaluation appears in the difficulty-wise results (Figure 5). The paper observes that stronger models like Qwen3-30B-A3B-Thinking achieve higher TF1 on Hard samples than on Easy ones — a pattern that would be impossible to detect if you were only measuring final answer accuracy, because you would see a monotonic accuracy drop with difficulty and conclude (incorrectly) that model competence uniformly degrades. The tool-invocation lens reveals the real mechanism: on Easy single-tool queries, models over-call tools (dragging down precision), whereas on Hard multi-tool queries, the task's richer constraints force more disciplined tool selection. This is a diagnostic insight about how LLMs handle tool-use agency — they are imprecise when given too much freedom and only settle into appropriate tool selection when the task demands it — that has direct implications for prompt engineering and agent design.

Innovation 2: Difficulty-Behavior Inversion — Why Stronger Models Improve on Harder Samples

The difficulty-wise TF1 results in Figure 5 and discussed in Section 3.4 contain the most counterintuitive empirical finding in the paper: for the strongest models, performance does not monotonically decline with difficulty — it improves on harder samples. This finding challenges the universal expectation in benchmark evaluation that harder tasks always yield lower scores, and it reveals something structural about how current LLMs approach tool-use tasks.

The conventional expectation, grounded in decades of benchmark design across NLP, is that accuracy should decrease as task complexity increases — a model that scores 80% on easy examples should score lower on hard examples. This expectation is so deeply ingrained that it is rarely questioned. The paper's difficulty taxonomy (Section 2.5) defines complexity structurally: Easy = 1–5 tool calls, Medium = 6–10, Hard = 11+. By conventional logic, Hard samples — requiring more tools, longer chains, and more complex dependency structures — should be harder for all models.

The data contradicts this. Qwen3-30B-A3B-Thinking and Qwen3-235B-A22B-Thinking show a clear upward trend in TF1 from Easy to Hard. The paper's diagnosis (Section 3.4) is subtle but important:

"Stronger models... improve from Easy to Hard, suggesting they leverage richer constraints and multi-tool opportunities in harder queries... Easy cases penalize over-calling (lower precision), whereas harder cases reward better recall and planning."

The mechanism is an agency-modulation effect: when the query is simple (one tool needed), the model's default behavior is expansive — it generates multiple tool calls because it has been trained to be helpful and thorough, not because the task requires it. This inflates the denominator of Tool Precision, dragging down TF1 even though the model does successfully call the required tool. When the query is complex (many tools needed in specific order), the model's tool selection tightens because the task's explicit multi-step structure constrains what counts as relevant — the model cannot "over-call" because the query itself demands an extensive tool chain.

This is a genuine empirical discovery, not a hypothesis the paper set out to test. It emerged from the decision to evaluate tool invocation directly rather than final answers — an answer-based evaluation would have shown monotonic difficulty degradation and never revealed the precision-recall dynamics at work. The implication is that tool-use evaluation must account for precision-recall tradeoffs that vary systematically with task complexity, and that aggregate metrics (a single TF1 number averaged over all difficulty levels) can obscure the real capability profile of a model. A model that scores 60 TF1 overall could be an over-caller on simple queries and a competent planner on complex ones, or it could be a balanced-but-mediocre performer across the board — the aggregate number cannot distinguish these cases.

More broadly, this finding suggests that the relationship between task difficulty and model performance may be qualitatively different for agentic tool-use tasks than for static knowledge or reasoning tasks. In knowledge tasks, difficulty correlates with concept complexity, and models degrade because harder concepts are less represented in training data. In tool-use tasks, difficulty correlates with structural complexity, and structural complexity imposes constraints that discipline model behavior — more moving parts means fewer degrees of freedom for arbitrary generation. This is a conceptual distinction with implications for how we design agent benchmarks more generally: harder tasks may sometimes be easier to evaluate because they produce more diagnostic signal.

Notably, GPT-OSS-20B shows a large jump from Easy to Medium/Hard but remains behind the Qwen3 models, suggesting that the ability to leverage structural constraints for improved precision is not universal — it is a capability that stronger models exhibit and weaker models do not, making it a meaningful axis of model differentiation rather than an artifact of the benchmark design.

Innovation 3: Real Production Data as the Foundation for Synthetic Augmentation, Rather than Synthetic-Only or Real-Only Construction

The paper makes a methodological contribution in how it constructs its benchmark, but the innovation is more conceptual than procedural: it establishes a paradigm where real production interaction traces provide the inductive bias for synthetic complexity amplification, rather than treating real data and synthetic data as alternative or competing construction strategies.

The two dominant approaches to constructing tool-use benchmarks are (1) building entirely from real user logs, and (2) generating entirely synthetic scenarios with hand-designed tool descriptions. The real-only approach (which existing financial benchmarks like CFLUE, CFinBench, and INVESTORBENCH effectively follow, albeit without tool use) guarantees authentic user language and tasks but is limited by what users actually ask — real users typically don't produce the complex multi-tool, multi-turn interactions that stress-test tool orchestration capability. The synthetic-only approach (common in general tool-use benchmarks) can generate arbitrarily complex scenarios but risks producing queries that sound artificial, combine tools in implausible ways, or reference unrealistic financial situations that a domain expert would immediately flag.

The paper's hybrid construction avoids the weaknesses of both extremes through a specific dependency structure that is worth articulating because it is not obvious at first glance:

The innovation is that the synthesis pipeline is constrained at every stage by signals extracted from real data, not by human-designed rules. The tool dependency graph — which determines which tool chains can be sampled — is constructed entirely from real multi-tool traces. The 288 edges in the graph are not manually specified by financial experts enumerating plausible dependencies; they are extracted algorithmically from observed co-occurrence patterns in production workflows and then validated by an LLM plausibility check. The in-context examples used to generate user queries for novel tool chains come from real single-tool user queries, preserving authentic user language patterns. And crucially, the reference trajectories are generated by executing the synthesized queries against a live MCP server — not by template construction — which means the reference itself reflects what a competent agent actually does when faced with that query, including potentially calling auxiliary tools not in the planned chain.

This last point — execution-based trajectory generation — is a subtle but important safeguard that distinguishes the paper's approach from standard synthetic data pipelines. In a typical pipeline, you would generate a query, then construct the reference tool chain deterministically from the planned chain. The paper's approach instead generates the query, feeds it to a model that is connected to real financial tools, and records what the model actually calls. If the query is unanswerable, the trajectory will fail, and the sample is rejected. If the query requires additional tools beyond the planned chain, the trajectory will include them, and the sample captures this richer dependency structure. This turns the trajectory generation step into an implicit verification step: samples only survive if they are actually resolvable with real tools.

The quality control results quantify the value of this hybrid approach. The yield from synthetic multi-tool generation is roughly 50% after automatic validation and roughly 37% after expert review (496 out of 1,000 generated pairs survive automated checks; 249 out of 679 candidates survive expert review). This means that a majority of synthetically generated samples fail quality checks, even when generated by a strong model (Qwen3-235B-2507) using a structured pipeline. A purely synthetic benchmark that did not apply this multi-stage filtering would be contaminated with unrealistic or unanswerable samples, making evaluation results uninterpretable — you would not know whether a model's low score reflected genuine tool-use failures or benchmark artifacts. The high rejection rate is itself evidence that the quality bar is meaningful.

This construction paradigm — extracting dependency structure from real traces, using that structure to constrain synthesis, augmenting with execution-based trajectory generation, and applying strict expert review — is generalizable beyond finance. Any domain with production tool-use logs (healthcare, legal, enterprise software) could adopt the same pipeline to build tool-use benchmarks that are simultaneously realistic (grounded in real user behavior), challenging (amplified to include complex chains not observed in logs), and high-quality (filtered by execution-based verifiability and human expert review).

5. Experimental Analysis

Evaluation Methodology

Dataset. FinMCP-Bench, consisting of 613 samples across 10 main scenarios and 33 sub-scenarios, drawn from a combination of real production logs (145 single-tool and some subset of 183 real multi-tool samples after expert review) and synthetically augmented multi-tool (249 total) and multi-turn (219 total) samples, as detailed in Section 2. The benchmark is not split into train/validation/test — all 613 samples serve as the evaluation set, since models are evaluated zero-shot without fine-tuning on the benchmark itself.
Base model(s). Six large language models are evaluated (Section 3.1): Qwen3-4B-Thinking, Qwen3-30B-A3B-Thinking, and Qwen3-235B-A22B-Thinking from the Qwen3 family (Yang et al., 2025); DeepSeek-R1 (DeepSeek-AI, 2025); GPT-OSS-20B (OpenAI, 2025); and Seed-OSS-36B (ByteDance Seed Team, 2025). A key structural note: the Qwen3 "A3B" and "A22B" designations indicate Mixture-of-Experts architectures where the first number is total parameters and the second is active parameters per forward pass — Qwen3-30B-A3B-Thinking has 30B total but only 3B active, and Qwen3-235B-A22B-Thinking has 235B total with 22B active. The "Thinking" variants incorporate chain-of-thought reasoning.
Metrics. Four metrics are defined in Section 3.2: Tool Recall (TR) = |reference ∩ predicted| / |reference|; Tool Precision (TP) = |reference ∩ predicted| / |predicted|; Tool F1 (TF1) = 2 × TR × TP / (TR + TP), the harmonic mean of recall and precision; and Exact Match Rate (EMR) = the proportion of predictions whose tool invocation structure exactly matches the reference tool organization after ignoring within-group order (since tools in the same parallel group are interchangeable). All metrics operate on extracted tool identifiers from the model's generated replies, not on final answer quality.
Baselines. There are no explicit baseline methods (e.g., a random-tool-calling baseline or a no-tool baseline) reported in the paper. The evaluation is structured as a comparative assessment across the six LLMs listed above, with models compared against each other directly on the same benchmark. Any model can be considered a baseline relative to any other — the Qwen3 models collectively form the leading group against which the other three are implicitly compared. There is no reported comparison to a non-MCP tool-use system, a retrieval-augmented generation baseline without structured tool calls, or a heuristic (non-LLM) tool selection baseline.
Generation budget / compute accounting. The paper does not measure or constrain generation budget. Models are evaluated under a fixed per-sample protocol (single-turn for single-tool and multi-tool samples; turn-by-turn with gold history for multi-turn samples) with no variation in compute allocation. There is no sweep over generation budgets, no comparison of performance at different sample counts, and no FLOPs accounting. The evaluation is purely about accuracy at whatever inference cost each model naturally incurs — a model that generates more tokens (e.g., chain-of-thought reasoning) is not penalized or compared at iso-compute with a model that generates fewer.
Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. Each of the 613 samples is evaluated once with each model. The difficulty-wise analysis (Easy / Medium / Hard) in Figure 5 and Section 3.4 reports TF1 averages within each difficulty bin, but no confidence intervals, standard deviations, or significance tests are provided. The scenario-wise radar chart (Figure 4) similarly reports per-scenario TF1 without uncertainty quantification. The quality control process (Section 2.4) uses two-fold independent expert review per sample with a strict conjunctive threshold (both reviewers score ≥4 on all five dimensions), but this is a construction-stage quality mechanism, not an evaluation-stage statistical protocol.

Main Quantitative Results

Aggregate Model Performance Across Sample Types

Table 2 reports the headline results across all six models for single-tool, multi-tool, and multi-turn samples, using Tool Recall (TR), Tool Precision (TP), Tool F1 (TF1), and Exact Match Rate (EMR). The paper does not report a single aggregate number across all sample types, instead presenting performance broken out by structural category.

Overall trends from Table 2. The three Qwen3 models generally outperform DeepSeek-R1, GPT-OSS-20B, and Seed-OSS-36B on both TF1 and EMR across all three sample types. However, the relationship between model size and performance is not monotonic within the Qwen3 family. Section 3.3 notes two specific comparative observations:

"Qwen3-4B-Thinking achieves a higher EMR score than Qwen3-30B-A3B-Thinking, while Qwen3-30B-A3B-Thinking attains a higher TF1 score than Qwen3-4B-Thinking."

This means the 4B model is better at exact structural matching (getting the grouping and ordering precisely right), while the 30B-A3B model is better at the balanced set-level metric that trades off recall and precision. The paper does not report the actual numerical values in the main text — they are visible only in Table 2 — but the qualitative pattern is clear: the optimal model depends on which metric you care about, and larger active parameters (30B-A3B uses ~30B total / 3B active; 235B-A22B uses 235B total / 22B active) do not strictly dominate.

Single-tool vs. multi-tool patterns. Section 3.3 identifies a consistent trend across models when comparing single-tool and multi-tool performance:

"Tool Recall (TR) is higher for single-tool samples, as each contains only one tool. In contrast, Tool Precision (TP) is lower for single-tool samples, since models often over-predict by generating multiple tools even when only one is needed."

This is the over-calling problem: when the reference requires exactly one tool, the true-positive count (the numerator of both TR and TP) is at most 1. TR = 1/1 = 1.0 anytime the model calls the correct tool (even if it also calls 10 irrelevant tools). But TP = 1/(1 + number of extraneous calls), so any extraneous tool invocation directly reduces precision. The paper observes that this pattern is consistent — models are "expansive" tool callers on simple queries, generating unnecessary tool invocations even when the task requires only one.

Multi-turn samples are the hardest category. Across all metrics, multi-turn samples yield the lowest scores. Section 3.3 states:

"Multi-turn samples tend to yield the lowest scores overall, especially in EMR, indicating that handling longer conversations with multiple tool calls remains challenging."

The specific difficulty with EMR on multi-turn samples is worth unpacking: exact structural match across multiple turns requires getting not just the set of tools right, but also precisely which tools are called at which turn, and within each turn, the correct grouping structure. A model that calls the right tools but distributes them differently across turns (e.g., calling a tool in turn 2 that the reference calls in turn 3) gets EMR = 0 for that sample even if its turn-aggregated TF1 is high. This makes multi-turn EMR an especially strict test of conversational tool orchestration.

Scenario-Wise Analysis

Figure 4 presents a radar chart of TF1 performance across the 10 main financial scenarios, with six model profiles overlaid. (Full scenario names are listed in Figure 1; the paper uses abbreviations in Figure 4 but the mapping is recoverable from the earlier taxonomy figure.)

The top tier is a two-model cluster. Section 3.4 describes:

"Qwen3-30B-A3B-Thinking and Qwen3-235B-A22B-Thinking form the leading group with the largest, most rounded profiles, indicating strong and balanced tool use across scenarios."

A "rounded" radar profile means the model's performance does not collapse on any particular scenario — it maintains relatively consistent TF1 across the 10 axes, rather than excelling on 2–3 scenarios and failing on others. This is a qualitative assessment (the paper does not report per-scenario variance or a roundness metric), but it conveys that the leading models' tool-use capability is general rather than scenario-specific.

Performance stratification below the top tier. Qwen3-4B-Thinking is described as a "solid second tier," DeepSeek-R1 and Seed-OSS-36B as "mid-range with noticeable dips," and GPT-OSS-20B as "lagging across all axes." The "noticeable dips" for the mid-range models suggest scenario-specific weaknesses — certain financial scenarios expose tool-use failures that are not present in others — but the paper does not identify which specific scenarios cause these dips, nor does it hypothesize why particular scenarios are harder.

Scenario difficulty gradient. Section 3.4 identifies an interaction between scenario type and model quality:

"Performance gaps widen in scenarios requiring multi-tool planning and cross-source synthesis, but narrow on simpler, single-operation queries."

This makes sense given the structural metrics: scenarios dominated by simple queries (requiring 1–2 tool calls with no complex dependencies) compress the performance distribution because even weaker models can achieve reasonable recall on simple tool sets. Scenarios requiring multi-tool chains and cross-source synthesis (e.g., Market Analysis and Research, which requires pulling data from multiple financial data sources and synthesizing it) create larger score separations between strong and weak models because the structural complexity demands planning capability that weaker models lack.

The paper does not report per-scenario EMR. The radar chart shows only TF1. EMR — the strictest metric, capturing dependency and parallelism structure — may show even larger per-scenario variance (especially for multi-tool-heavy scenarios), but this dimension of analysis is absent.

Difficulty-Wise Analysis

Figure 5 presents TF1 across Easy (≤5 tool calls), Medium (6–10 tool calls), and Hard (11+ tool calls) difficulty bins.

The central counterintuitive finding. Section 3.4 reports:

"TF1 does not monotonically decline with difficulty. Stronger models (Qwen3-30B-A3B-Thinking and Qwen3-235B-A22B-Thinking) improve from Easy to Hard, suggesting they leverage richer constraints and multi-tool opportunities in harder queries."

This is the difficulty-behavior inversion discussed in Section 4 (Key Insights). The mechanism: easy single-tool queries penalize over-calling (extraneous tool invocations reduce precision), while hard queries with extensive tool chains reward balanced tool selection — the richer constraints of the task force more disciplined invocation. On hard queries, the model "settles into" appropriate tool selection because the query structure demands it; on easy queries, the model's default expansiveness works against it because the query structure does not constrain tool selection.

Performance stratification by difficulty. Section 3.4 characterizes each model's trend:

"Qwen3-4B-Thinking shows a mild upward trend, while DeepSeek-R1 and Seed-OSS-36B rise modestly. GPT-OSS-20B exhibits a large jump from Easy to Medium/Hard but remains behind others."

The key differentiator is the magnitude of the improvement: the Qwen3 models benefit substantially from the structural constraints of harder tasks, while DeepSeek-R1 and Seed-OSS-36B show smaller gains. GPT-OSS-20B's "large jump" from Easy to Hard is notable — it suggests the model performs particularly poorly on Easy queries (likely due to severe over-calling) and improves substantially when the task provides more structure — but it still remains behind the Qwen3 models on absolute TF1, meaning its improvement is from a very low Easy baseline.

Implication for aggregate metrics. Section 3.4 concludes:

"Overall, easy cases penalize over-calling (lower precision), whereas harder cases reward better recall and planning, yielding higher TF1 for models with balanced tool selection."

This means that a model's aggregate TF1 (averaged across all difficulty levels) is influenced not just by its intrinsic tool-use capability, but by the difficulty distribution of the benchmark. A benchmark heavily weighted toward Easy samples would penalize over-callers (depressing TF1 for models with expansive tool-selection behavior) and reward minimalists (who call only the required tool). A benchmark heavily weighted toward Hard samples would reward planners and penalize models that cannot handle long tool chains. The paper's difficulty breakdown reveals these dynamics, but the aggregate numbers in Table 2 obscure them — a model with 60 TF1 overall could have 45 TF1 on Easy and 75 TF1 on Hard, which are very different capability profiles.

A missing analysis: per-difficulty EMR and per-difficulty precision/recall separately. Figure 5 reports only TF1. The paper does not show how EMR, TR, and TP individually vary with difficulty, which would illuminate the specific mechanism. For instance, if Easy→Hard TF1 improvement is driven entirely by increasing recall (models finding more of the required tools on harder queries, while precision remains roughly constant), that is a different story than if it is driven by increasing precision (models becoming more selective about which tools they call when the task is complex). The paper's qualitative explanation ("easy cases penalize over-calling, harder cases reward better recall and planning") suggests both mechanisms operate, but the data to disentangle them is not presented.

Summary of Key Quantitative Patterns

Synthesizing across the three analyses (aggregate, scenario-wise, difficulty-wise), the empirical picture is:

Qwen3-30B-A3B-Thinking and Qwen3-235B-A22B-Thinking form the top tier across all metrics and all analyses, with Qwen3-235B-A22B-Thinking showing the strongest and most balanced performance (Table 2, Figure 4, Figure 5).
Model size is not monotonic with performance: Qwen3-4B-Thinking beats Qwen3-30B-A3B-Thinking on EMR, while the 30B model beats the 4B model on TF1 (Table 2). The optimal model depends on whether you prioritize exact structural matching or balanced set-level accuracy.
Multi-turn samples are the hardest category for all models, especially on EMR (Table 2). This indicates that distributing tool calls correctly across conversational turns — not just calling the right tools in aggregate — is a distinct and challenging capability.
Performance gaps between strong and weak models widen on complex scenarios requiring multi-tool planning and cross-source synthesis, and narrow on simple scenarios (Figure 4, Section 3.4).
Stronger models improve rather than degrade on harder samples, driven by richer task constraints disciplining tool selection and reducing over-calling (Figure 5, Section 3.4). This is the paper's most distinctive empirical finding.

Ablation Studies and Robustness Checks

FinMCP-Bench is a benchmark paper, not a methods paper. There are no model ablations (no architectural variants, no training data ablations, no hyperparameter sweeps) because the goal is not to improve tool-use capability but to measure it. The "ablation" equivalent for a benchmark paper is the analysis of how benchmark design choices affect results. The paper provides several analyses that serve this function:

Sample type breakdown (single-tool vs. multi-tool vs. multi-turn). Table 2's disaggregation by sample type serves as a structural ablation: it shows that if the benchmark were composed exclusively of single-tool samples, performance rankings might differ (since models that excel at exact match on simple queries differ from models that excel at balanced tool selection on complex queries). The multi-turn breakdown confirms that excluding multi-turn samples would substantially inflate reported performance, especially on EMR, since multi-turn samples are the hardest category.

Difficulty-level analysis (Easy / Medium / Hard). Figure 5's difficulty breakdown demonstrates that aggregate TF1 obscures difficulty-dependent dynamics. A benchmark that averaged across all difficulty levels without reporting the breakdown would present a misleading picture — model rankings might appear stable, but the underlying capability profile (over-caller on Easy, competent planner on Hard) would be invisible. This is effectively an ablation of the "aggregate-only" reporting convention.

Scenario-wise breakdown. Figure 4's radar chart serves as a coverage ablation: it verifies that no single scenario dominates the benchmark results and that model rankings are not driven by anomalous performance on one or two outlier scenarios. The "rounded" profiles of the top models indicate balanced scenario coverage, while the "dips" in mid-range models indicate scenario-specific weaknesses. Without this breakdown, one might incorrectly attribute a model's low aggregate score to general tool-use weakness when it actually reflects collapse on one specific scenario type.

What the paper does not ablate. Several benchmark design choices are not empirically validated:

Tool Recall / Precision / F1 vs. alternative metrics. The paper proposes these metrics but does not compare them against alternatives (e.g., answer-based evaluation, human judgment of tool-use quality, or softer partial-credit metrics for near-miss tool chains). There is no demonstration that the proposed metrics correlate with downstream task success or user satisfaction.
The 5-point Likert scale threshold. The strict conjunctive threshold (≥4 on all five dimensions from both reviewers) is not compared against more lenient thresholds (e.g., ≥3, or average ≥4). The paper reports that 249 out of 679 multi-tool candidates survived expert review, but does not report how many would have survived under a ≥3 threshold or an average-score threshold, making the impact of this design choice opaque.
The difficulty bin boundaries. Easy = 1–5, Medium = 6–10, Hard = 11+ tool calls. The paper does not test whether alternative boundaries (e.g., 1–3 / 4–8 / 9+, or continuous binning) would change the observed difficulty-behavior inversion pattern.
Gold history vs. generated history for multi-turn evaluation. The protocol provides gold conversation history to the model at each turn. The paper does not compare this against an evaluation where the model receives its own generated history from previous turns, which would quantify the compounding-error effect that gold history is designed to avoid.
The MCP schema itself. All 65 tools follow MCP. There is no comparison against a non-MCP tool interface (e.g., raw function-calling with different schema conventions) to determine whether MCP-specific factors (schema verbosity, input/output format requirements) affect model performance independently of general tool-use capability.

Critical Assessment

Does the benchmark actually measure tool invocation correctness?

The paper's central claim — that FinMCP-Bench evaluates tool invocation capability — rests on the assumption that the reference tool traces represent "correct" tool use. This assumption is auditable in principle (the traces come from a production system with expert-defined SOPs, and samples are expert-reviewed on a 5-point scale across five dimensions including tool-chain completeness and logical consistency), but the paper does not provide inter-annotator agreement statistics for the expert review stage. Section 2.4 states that each sample is independently scored by two randomly assigned experts, and that disagreements are resolved through discussion, but it does not report Cohen's kappa, Krippendorff's alpha, or even raw agreement rate before discussion. Without this, the reader cannot assess whether the "correct" reference traces represent a reliable consensus or whether different experts would have produced substantially different judgments. This is a standard reporting expectation for benchmark construction papers and its absence is a genuine weakness.

A subtler concern: the synthetic multi-tool reference trajectories are generated by Qwen3-235B-2507 executing against a live MCP server (Section 2.2, Stage 3). This means the reference traces for the most complex samples are generated by one specific model (the same model family that later appears as the top-performing evaluated model). If Qwen3-235B-2507 has systematic biases in how it chains tools — for example, a tendency to group certain tools in parallel when a different reasonable agent might call them sequentially — these biases are baked into the reference, and other models that make different but equally valid grouping decisions are penalized (via reduced EMR). The paper's quality control (expert review of trajectories) mitigates this by filtering out trajectories that experts deem logically inconsistent, but it does not eliminate the possibility that multiple valid tool-chain structures exist for the same query, and that the EMR metric unfairly penalizes models that choose a different valid structure.

A related concern: the expert review evaluates whether the reference trajectory is "correct," not whether it is the uniquely correct trajectory. If a query can legitimately be answered via two different tool chains (e.g., retrieving fund information via a search tool vs. a direct lookup tool), a model that chooses the alternative chain will get low recall and EMR despite producing a valid answer. The paper does not discuss whether the financial scenarios admit multiple valid tool-chain solutions, nor does it report whether expert reviewers considered alternative tool chains during quality control.

Does the tool invocation evaluation capture what matters for deployment?

The paper explicitly argues (Section 3.2, footnote 2) that tool-use evaluation is more appropriate than answer-based evaluation for financial advisory scenarios because queries are open-ended. This is a defensible position, but it leaves a gap: the paper never demonstrates that tool invocation accuracy correlates with any downstream measure of deployment success — user satisfaction, answer quality, task completion rate, or expert-judged response appropriateness. It is possible that a model with perfect tool invocation accuracy (TF1 = 1.0, EMR = 1.0) produces financial advice that is technically accurate but poorly communicated, or that a model with moderate tool invocation accuracy produces excellent advice because it compensates for minor tool errors with strong reasoning. The paper claims tool invocation matters, but never empirically validates that it matters for the outcomes that financial applications actually care about.

The paper also does not distinguish between tool invocation errors that are consequential (e.g., failing to look up a user's risk profile before recommending a high-risk fund) and those that are inconsequential (e.g., calling an additional informational tool that provides redundant data). All tool invocation errors are weighted equally in the metrics — an extraneous parallel lookup of a fund's expense ratio (which adds useful context) is penalized identically to a completely irrelevant tool call. This makes the metrics sensitive to a specific definition of "correct" tool use that may not align with what makes a financial agent useful in practice.

Does the benchmark support the claims about model capability?

Claim: Qwen3 models generally outperform others. This is straightforwardly supported by Table 2, Figure 4, and Figure 5. However, the conclusion that "Qwen3 models are better at financial tool use" is confounded by the fact that Qwen3-235B-2507 was used throughout benchmark construction — for dependency validation, query generation, trajectory generation, and multi-turn dialogue simulation. This creates a potential data-generation bias: the benchmark was partially built by the same model family that later excels on it. The paper does not address this circularity. It is plausible that Qwen3 models perform well because they share architectural and training properties with the model that generated the reference traces, not because they have superior financial tool-use capability in an absolute sense. Evaluating a model on traces it (or its larger sibling) generated is a known contamination risk in benchmark construction, and the paper's silence on this point is a significant limitation.

Claim: multi-turn samples are the hardest, especially on EMR. This is supported by Table 2, but the paper does not investigate why multi-turn EMR is so low. Is it because models fail to distribute tools correctly across turns (calling a tool in turn 2 that should have been called in turn 3), or because they fail to maintain coherent multi-turn context and call entirely wrong tools? These are diagnostically different failures, but the aggregate EMR number cannot distinguish them. A per-turn error analysis would substantially strengthen this claim.

Claim: stronger models improve on Hard samples due to richer constraints. This is the paper's most interesting claim, supported by Figure 5. However, the paper does not provide per-model precision and recall breakdowns by difficulty level. The claim that "easy cases penalize over-calling" is qualitative — we can see TF1 is lower on Easy, but we do not see whether this is driven by low precision (as the paper argues) or some other factor. If the paper showed that Qwen3-30B-A3B-Thinking has precision of 0.5 on Easy and 0.9 on Hard (while recall is stable), the over-calling mechanism would be directly validated. Without this breakdown, the claim remains a plausible interpretation rather than a demonstrated fact.

Claim: performance gaps widen on complex scenarios and narrow on simple ones. This is a qualitative observation from Figure 4, but the paper does not quantify it — there is no reported variance across scenarios per model, no statistical test of whether the gap between top and bottom models is larger on complex vs. simple scenarios, and no definition of which scenarios count as "complex" (beyond the general observation about "multi-tool planning and cross-source synthesis"). The claim is visually plausible from the radar chart but not rigorously supported.

What experiments would have strengthened the paper?

Inter-annotator agreement for expert review. Reporting Cohen's kappa or raw agreement rate for the two-expert review process would allow readers to assess the reliability of the reference traces. Without this, the quality of the ground truth is asserted rather than demonstrated.

Alternative valid tool-chain analysis. For a subset of samples, having domain experts enumerate all valid tool chains (not just the reference chain) would reveal whether the EMR metric penalizes models for choosing valid-but-different tool invocation structures. If multiple valid chains exist for 20% of samples, then EMR has a hard ceiling of 80% even for a perfect model — the paper would need to report this ceiling and potentially adjust metrics accordingly.

Ablation of MCP vs. alternative tool interfaces. Evaluating models on the same queries but with tools described in a non-MCP format (e.g., simple function signatures without MCP schema overhead) would isolate whether MCP-specific factors affect performance. If models perform identically under both interfaces, then FinMCP-Bench is a general financial tool-use benchmark; if they perform differently, then its results are MCP-specific and may not generalize to other tool-use protocols.

Answer-based evaluation as a complementary metric. Even if tool invocation is the primary evaluation target, reporting whether models that score high on TF1/EMR also produce answers that experts prefer would validate the implicit assumption that tool-use correctness correlates with output quality. A small-scale human evaluation (e.g., 50 samples rated by 3 experts on response quality) comparing top-TF1 and low-TF1 models would provide this validation.

Per-turn error analysis for multi-turn samples. Breaking down multi-turn EMR by turn position (turn 1, turn 2, ..., turn n) would reveal whether errors concentrate early (context grounding failures) or late (context maintenance failures), which would have direct implications for agent design.

Per-difficulty precision and recall breakdown. Showing TR, TP, TF1, and EMR separately for Easy/Medium/Hard would validate the over-calling mechanism and reveal whether EMR follows the same difficulty-inversion pattern as TF1 or shows a different trend.

Baselines beyond LLMs. Including a non-LLM baseline (e.g., a retrieval-based system that maps queries to tool chains using embedding similarity to training examples) would establish a lower bound on task difficulty and reveal whether the benchmark is solvable without language understanding. If a simple retrieval baseline achieves non-trivial TF1, the benchmark may be measuring tool co-occurrence patterns rather than genuine tool-use reasoning.

Summary of experimental strengths and weaknesses

Strengths. The paper provides a comprehensive multi-dimensional evaluation (3 sample types × 4 metrics × 10 scenarios × 3 difficulty levels), with clear disaggregation that reveals patterns invisible in aggregate metrics — the difficulty-behavior inversion and the scenario-dependent performance gap widening are genuinely informative findings. The evaluation covers six models spanning three orders of magnitude in parameter count, allowing the paper to speak to scaling trends (or their absence — the non-monotonic relationship between size and performance is an important negative result). The use of gold history for multi-turn evaluation isolates per-turn tool-use capability from cascading error effects, which is methodologically sound.

Weaknesses. The data-generation circularity (Qwen3-235B-2507 used for benchmark construction; Qwen3 models top the leaderboard) is the most significant concern and is unaddressed. The absence of inter-annotator agreement statistics undermines confidence in the reference traces. The metrics, while well-defined, are not validated against any external criterion of tool-use quality. The difficulty-behavior inversion claim — the paper's most interesting finding — is supported only by aggregate TF1 data without the precision-recall breakdown that would confirm the proposed over-calling mechanism. And the paper reports no statistical uncertainties, making it impossible to assess whether observed differences (especially the small gaps between Qwen3-30B-A3B-Thinking and Qwen3-235B-A22B-Thinking) are reliable or within sampling noise given the 613-sample test set.

6. Limitations and Trade-offs

Data-Generation Circularity: The Benchmark Was Partially Built by the Model Family That Tops the Leaderboard

The assumption or constraint. The paper uses Qwen3-235B-2507 — the largest model in the Qwen3 family — for three critical stages of benchmark construction: (1) validating candidate dependencies during tool dependency graph construction (Section 2.2, "The model is prompted to judge whether it is reasonable for $t_j$ to depend on $t_i$ "), (2) generating user queries for chain-based multi-tool samples using in-context examples (Section 2.2, "we prompt Qwen3-235B-2507 to generate a proper user query"), and (3) generating reference trajectories by executing those queries against a live MCP server (Section 2.2, "Qwen3-235B-2507, connected to Qieman's MCP server, is then used to generate the corresponding trajectory"). The same model family — Qwen3-235B-2507's architectural relatives Qwen3-4B-Thinking, Qwen3-30B-A3B-Thinking, and Qwen3-235B-A22B-Thinking — subsequently appears as the top-performing evaluated models on the benchmark (Table 2, Figure 4, Figure 5).

The consequence. There is a non-trivial risk that the benchmark's reference traces encode systematic preferences of the Qwen3 family — how it groups tools into parallel vs. sequential calls, which specific tools it chooses when multiple tools could serve the same purpose, how it phrases synthetic user queries — and that Qwen3 models score higher not because they have superior financial tool-use capability in an absolute sense, but because they share training objectives, architectural biases, and tokenization patterns with the model that generated the ground truth. This is a known contamination risk in benchmark construction: evaluating a model on traces produced by itself or its siblings can inflate performance estimates relative to what the model would achieve on independently constructed references.

The consequence is most acute for the Exact Match Rate (EMR) metric, which checks whether the predicted tool invocation structure exactly matches the reference organization. If Qwen3-235B-2507 has a consistent tendency to group certain tools in parallel when other models might group them sequentially, the reference trajectories will reflect this grouping preference. Non-Qwen3 models that make different but equally valid grouping decisions are penalized (EMR = 0 for that sample), while Qwen3 models' similar grouping preferences yield higher EMR. The set-based metrics (TR, TP, TF1) are less sensitive to this bias because they ignore grouping structure, but they are not immune — if Qwen3-235B-2507 systematically prefers one tool over another for a given subtask (e.g., choosing get_fund_details_v2 over search_funds), the reference will encode that preference, and Qwen3 models that share the preference will show higher recall and precision.

What evidence exists in the paper. There is no analysis addressing this concern. The paper does not:

Report whether the evaluated Qwen3 models were trained on data that overlaps with Qwen3-235B-2507's training distribution in ways that could confer an advantage on this specific benchmark.
Compare Qwen3 model performance on real samples (where reference traces come from the production system, not from Qwen3-235B-2507) vs. synthetic samples (where reference traces were generated by Qwen3-235B-2507) to check whether the Qwen3 advantage is larger on synthetic samples — which would be direct evidence of circularity.
Evaluate whether the reference traces generated by Qwen3-235B-2507 differ systematically in tool grouping, tool selection, or chain structure from the real production traces produced by the XiaoGu assistant (which presumably uses a different underlying model or rule-based workflow system).
Include a model from a different family (or an earlier Qwen generation) in the benchmark construction pipeline to produce an alternative set of reference traces, and then compare leaderboard rankings across the two reference sets.

Mitigation status. The paper does not acknowledge this limitation, does not attempt to mitigate it, and does not suggest it as a direction for future work. The expert review process (Section 2.4) provides some mitigation — if Qwen3-235B-2507 produces reference trajectories with grouping structures that are genuinely incorrect (violating logical consistency), expert reviewers would reject those samples. But expert review cannot detect valid-but-idiosyncratic preferences: if Qwen3-235B-2507 groups two tools in parallel and an expert agrees that parallel grouping is valid, the sample passes review, but a non-Qwen3 model that groups them sequentially (also valid) is penalized. Expert review filters out incorrect references; it does not ensure that the reference represents the uniquely correct structure, nor that it is neutral with respect to model family.

Tool Invocation Evaluation Is Not Validated Against Any Downstream Measure of Agent Quality

The assumption or constraint. The paper's core design choice — evaluating tool invocation correctness rather than final answer quality — rests on an implicit assumption that correct tool invocation correlates with downstream deployment success: that an agent which calls the right tools in the right order will produce better financial advice, higher user satisfaction, or more trustworthy recommendations than an agent that makes tool invocation errors. The paper asserts this in Section 3.2, footnote 2:

"In financial research and advisory scenarios, queries are inherently open-ended without standard answers; hence, evaluation focuses on tool-use capability."

But the paper never empirically tests whether higher TF1 or EMR translates into any outcome that a financial application deployer would actually care about.

The consequence. Several failure modes are plausible, and the paper provides no evidence to rule them out:

High tool-invocation accuracy, poor response quality. A model might call exactly the right tools in exactly the right order but then generate a poorly written, confusing, or unactionable summary for the user. An agent with TF1 = 0.95 that produces incomprehensible financial advice is worse in practice than an agent with TF1 = 0.80 that communicates clearly, but the benchmark would rate the former as superior.
Tool invocation errors vary in consequence. The metrics weight all tool-calling errors equally. Calling an extraneous tool that retrieves redundant but harmless information (e.g., looking up a fund's expense ratio when it was already retrieved) incurs the same precision penalty as calling a completely irrelevant tool or skipping a critical compliance check. In deployment, these errors have vastly different consequences: the first wastes a small amount of API budget and adds noise to the response; the second could produce a regulatory violation or financial loss. The benchmark's uniform weighting does not distinguish them.
Tool invocation is a means, not an end. The goal of a financial assistant is to help users make financial decisions, not to call tools. If a model achieves perfect tool invocation but users find the interaction frustrating (too many follow-up questions, overly technical explanations, poor conversational flow), the deployment fails. The benchmark provides no signal about these user-facing qualities.

What evidence exists in the paper. None. The paper does not:

Conduct a human evaluation comparing responses from high-TF1 and low-TF1 models on dimensions of helpfulness, clarity, or actionability.
Report whether tool invocation metrics correlate with expert judgments of response quality on a subset of samples.
Analyze whether specific types of tool invocation errors (missing tools vs. extraneous tools vs. ordering errors) have systematically different impacts on response quality.
Survey users of the production XiaoGu assistant to determine whether tool invocation correctness (as opposed to answer quality, conversational ability, or trustworthiness) is what drives their satisfaction.

Mitigation status. The paper does not attempt to validate the tool-invocation-to-outcome link, nor does it frame this as a limitation requiring future work. The choice to evaluate tool use rather than answers is presented as a feature (it enables objective evaluation of open-ended tasks) without acknowledging the tradeoff — that the evaluation target is one step removed from what actually matters. A footnote or appendix validating the assumption on even a small sample (e.g., "expert raters preferred responses from models with higher TF1 in 80% of blind comparisons") would substantially strengthen the paper's central claim that tool invocation evaluation is the right choice for this domain.

Difficulty Estimation and Strategy Selection Cost Are Unaccounted For in the Headline Results

The assumption or constraint. This limitation is the analog of the difficulty estimation cost issue in the reference paper, adapted to the benchmark context. The paper defines difficulty structurally — Easy = 1–5 tool calls, Medium = 6–10, Hard = 11+ (Section 2.5) — and reports per-difficulty results (Figure 5). However, this difficulty taxonomy relies on knowing the reference tool chain length, which is a form of ground-truth information not available at deployment time. A real financial agent receiving a user query does not know a priori how many tool calls the query "should" require — the difficulty is latent and must be inferred from the query text and available tools.

The consequence. The difficulty-wise analysis in Figure 5 and Section 3.4 — particularly the central finding that stronger models improve on Hard samples — is an analytical decomposition that reveals interesting patterns about model behavior, but cannot be directly translated into a deployment strategy. A practitioner reading the paper might conclude: "For hard queries (11+ tool calls), I should use Qwen3-235B-A22B-Thinking because it shows the highest TF1 on Hard samples." But to actually implement this, the practitioner would need a method for classifying incoming user queries into Easy/Medium/Hard before selecting which model to route them to — and that classification itself would need to be accurate, cheap, and reliable. The paper provides no such method.

This is a different limitation from the data-generation circularity or the metric-validation gap. It is a deployability gap: the benchmark reveals difficulty-dependent performance patterns but provides no mechanism for exploiting those patterns in a live system. The paper's difficulty taxonomy is retrospective — it classifies samples after the reference trace is known — not prospective.

What evidence exists in the paper. The paper's construction pipeline does include an implicit query complexity signal: the chain-based synthesis (Section 2.2) generates queries of known tool-chain length because the chain is sampled from the dependency graph before the query is generated. But this is a construction artifact, not an evaluation tool. The evaluated models are never asked to predict difficulty; they are only asked to invoke tools. The benchmark does not include a "difficulty prediction" subtask, does not report whether any of the evaluated models can accurately estimate how many tools a query will require, and does not analyze whether models that are aware of their own tool-calling complexity perform differently from models that are not.

Mitigation status. The paper does not address this limitation. It does not discuss the gap between retrospective difficulty binning (post-hoc based on reference traces) and prospective difficulty estimation (pre-deployment based on query text). It does not suggest training a difficulty classifier as future work. The limitation is structural to the benchmark design: FinMCP-Bench evaluates tool use given queries, not the meta-capability of assessing query complexity before invoking tools. Whether this meta-capability matters depends on the deployment context — a system that always uses the same model for all queries does not need difficulty estimation — but the paper's difficulty-wise analysis implicitly invites difficulty-conditioned model selection without providing the means to implement it.

Single Domain, Single Deployment Source, Single Protocol — Generalization Is Unproven

The assumption or constraint. All 613 samples, all 65 tools, and all 10 scenarios come from a single financial deployment: the XiaoGu AI assistant in the Qieman APP operated by Yingmi Fund (Section 2.1). The tools are specific to this platform's MCP implementation; the user queries reflect the demographic and linguistic patterns of Qieman's user base (Chinese financial consumers using a specific app); and the tool dependency graph encodes the workflow conventions of one company's financial advisory processes. The paper evaluates six LLMs, but all evaluations use exactly these tools, these queries, and this reference-trace format.

The paper makes no claim of generalization to other financial deployments, other tool protocols, other languages, or other user populations. But it also provides no analysis of which benchmark properties are deployment-specific and which are likely to transfer. Section 1 frames the benchmark as addressing a gap in "real-world financial tool use," implying broad applicability, but the evidence base is a single real world.

The consequence. A practitioner deploying a financial agent on a different platform — with different tools, different user demographics, different regulatory requirements, or a different MCP implementation — cannot confidently use FinMCP-Bench results to predict model performance. Several specific failure modes are plausible:

Tool vocabulary mismatch. If another platform's MCP tools have different names, descriptions, or input/output schemas than Qieman's 65 tools, model rankings might shift. A model that excels at Qieman's specific tool set might perform poorly on a different tool set with different naming conventions, even if both tool sets serve the same financial functions.
User query distribution shift. Qieman's users may ask financial questions in a particular style (e.g., shorter queries, specific terminology, certain types of investment goals) that differs from users of a different financial platform (e.g., a U.S. robo-advisor, a European banking app, or a B2B financial analytics tool). Models that perform well on Qieman-style queries may not generalize to other user populations.
Workflow convention specificity. The tool dependency graph (Section 2.2) encodes one company's workflow conventions — which tools are typically called in sequence, which are parallelized, which tools serve as prerequisites for which others. A different financial institution might organize the same underlying financial operations into different tool boundaries and different dependency structures. A model that learns to match Qieman's specific workflow patterns may not adapt to a different institution's conventions.
Regulatory and compliance variation. Financial tool use in one jurisdiction (China, where Yingmi Fund operates under CSRC regulation) may involve different mandatory steps, disclosure requirements, or eligibility checks than in another jurisdiction. The benchmark's reference traces encode these jurisdiction-specific requirements. A model evaluated on FinMCP-Bench might appear to make "errors" (missing a compliance tool call) when in fact it correctly follows the conventions of a different regulatory regime.

What evidence exists in the paper. There is no cross-domain, cross-deployment, or cross-protocol evaluation. All six models are evaluated only on FinMCP-Bench. The paper does not:

Report results on a second financial tool-use dataset (from a different platform) to assess whether model rankings transfer.
Evaluate models on a non-MCP tool interface to determine whether performance is MCP-specific or reflects general tool-use capability.
Analyze whether Qieman's tool set covers a representative range of financial tool types (market data retrieval, portfolio analysis, risk assessment, compliance checks, recommendation generation) or is skewed toward particular categories that might favor certain model capabilities.
Discuss how the Chinese-language user base and Chinese financial context might affect generalization to other languages or markets.

Mitigation status. The paper does not claim generalization beyond the evaluated setting, but it also does not explicitly bound its claims. The introduction describes FinMCP-Bench as evaluating "LLMs in realistic and challenging financial scenarios through interactions with real-world Model Context Protocol" (Section 1), and the conclusion hopes it "can serve as a standardized and challenging testbed for advancing research on tool-augmented LLMs in finance" (Section 4). These framings imply that FinMCP-Bench is representative of financial tool use broadly, but the paper provides no evidence that Qieman's 65 tools, 10 scenarios, and user query distribution are representative of the broader financial tool-use landscape. Without such evidence, a practitioner considering whether FinMCP-Bench results apply to their deployment must either (a) assume representativeness without evidence, or (b) treat the results as specific to the Qieman ecosystem and of unknown relevance elsewhere.

The Exact Match Rate Metric May Penalize Valid Alternative Tool Chains, and This Ceiling Is Not Estimated

The assumption or constraint. The Exact Match Rate (EMR) metric, defined in Section 3.2, checks whether the predicted tool invocation structure exactly matches the reference tool organization after ignoring within-group order. For a sample to receive EMR = 1, the model must call exactly the same tools, in exactly the same grouping structure (sequential vs. parallel), as the reference trajectory. The assumption is that the reference trajectory represents the uniquely correct way to organize tool calls for that query — that any deviation from the reference grouping, even if functionally equivalent, constitutes an error.

The consequence. Financial queries often admit multiple valid tool-calling strategies. For example, if a user asks "Compare Fund A and Fund B on returns, risk, and fees," one valid strategy is to call get_fund_details(A) and get_fund_details(B) in parallel (both independent), then call compare_funds(A, B) sequentially (requires outputs from both). Another equally valid strategy is to call get_fund_comparison(A, B) as a single tool that handles everything. A third valid strategy is to call get_fund_returns(A), get_fund_returns(B), get_fund_risk(A), and get_fund_risk(B) all in parallel. All three strategies produce correct answers, but they differ in tool count and grouping structure — only the strategy matching the reference gets EMR = 1.

The consequence is that EMR has an unknown hard ceiling that is lower than 100% even for a hypothetical perfect agent. If, for 20% of benchmark samples, there are multiple valid tool-calling strategies, then even an oracle that always produces a valid strategy would achieve EMR ≤ 80% — not because it makes errors, but because it sometimes chooses a valid strategy different from the specific strategy recorded in the reference. The paper reports EMR values in Table 2 without estimating this ceiling, making the numbers difficult to interpret. When Qwen3-235B-A22B-Thinking achieves a particular EMR on multi-tool samples, we cannot know whether the gap to 100% reflects (a) genuine tool invocation errors, (b) valid-but-different tool organization choices, or (c) some mixture of both.

This limitation affects the paper's claim (Section 3.3) that Qwen3-4B-Thinking achieves higher EMR than Qwen3-30B-A3B-Thinking despite being smaller. If the 4B model happens to share more grouping preferences with Qwen3-235B-2507 (the model that generated the reference trajectories for synthetic samples) than the 30B model does, its higher EMR could partially reflect similarity to the reference generator rather than superior tool-organization capability. Without knowing the EMR ceiling and the degree of grouping ambiguity in the benchmark, this comparison is uninterpretable.

What evidence exists in the paper. The paper provides no analysis of EMR ceiling or alternative valid tool chains:

The expert review process (Section 2.4) evaluates whether the reference trajectory is correct on dimensions like "tool-chain logical consistency" and "tool-chain completeness," but does not ask reviewers to enumerate all valid tool chains for each query. A sample passes review if its reference chain is valid; the existence of other valid chains is not assessed.
The paper does not report, for any subset of samples, how many distinct valid tool-calling strategies exist. There is no "tool-chain ambiguity" statistic analogous to inter-annotator agreement.
The paper does not discuss whether certain sample types (single-tool vs. multi-tool vs. multi-turn) or certain scenarios have intrinsically higher tool-chain ambiguity, which would mean EMR ceilings vary across the benchmark categories — further complicating EMR comparisons across sample types.

Mitigation status. The paper partially mitigates this limitation through the set-based metrics (TR, TP, TF1), which ignore grouping structure and evaluate only whether the right tools were called regardless of organization. A model that calls the right tools in a different valid grouping achieves TF1 = 1.0 even if EMR = 0. But the paper also treats EMR as a primary metric — it is reported alongside TF1 throughout, and the multi-turn analysis (Section 3.3) specifically highlights EMR as revealing that "handling longer conversations with multiple tool calls remains challenging." The severity of this challenge cannot be assessed without knowing how much of the low EMR is due to genuine errors versus valid alternative organizations. The paper's mitigation (reporting TF1 alongside EMR) acknowledges that EMR is strict, but does not provide the ceiling information needed to calibrate EMR values.

No Statistical Uncertainty Quantification for a 613-Sample Test Set

The assumption or constraint. All reported results in Table 2, Figure 4, and Figure 5 are point estimates — single numbers (TF1, EMR, TR, TP) computed over the full 613-sample benchmark without any measure of statistical uncertainty. The paper does not report standard deviations, confidence intervals, bootstrap ranges, or significance tests for any comparison between models, between sample types, between difficulty levels, or between scenarios.

The consequence. The 613-sample test set, when partitioned into subcategories, yields small effective sample sizes. Specific cases where this matters:

The 10 scenarios (Figure 4). The paper reports 10 scenario-wise TF1 scores per model. The smallest scenarios (those not named as having the largest sample counts) may contain as few as 20–30 samples. A model's TF1 on a 25-sample scenario has a wide confidence interval — a difference of 5–10 percentage points between two models on such a scenario could easily be noise. The radar chart (Figure 4) plots these point estimates without error bars, making it visually ambiguous whether the "noticeable dips" for DeepSeek-R1 and Seed-OSS-36B on certain scenarios are statistically reliable or sampling artifacts.
The 3 difficulty bins (Figure 5). The Easy, Medium, and Hard splits contain different numbers of samples, and the paper does not report the bin sizes. If Hard contains the fewest samples (plausible, since most benchmarks have fewer highly complex examples), the Hard TF1 estimates have the largest uncertainty. The paper's central counterintuitive finding — that stronger models improve on Hard — rests on comparing point estimates across bins; if the Hard bin has wide confidence intervals, the improvement may not be statistically distinguishable from flat performance.
The 3 sample types (Table 2). Single-tool (145 samples), multi-tool (249 samples), and multi-turn (219 samples) have different sample sizes. Differences in reported metrics across types (e.g., "multi-turn samples tend to yield the lowest scores overall," Section 3.3) are not tested for significance. The sample sizes are moderate (145–249 per type), so moderate differences may be reliable, but the paper provides no basis for assessing this.
Model comparisons within the Qwen3 family. Section 3.3 reports that "Qwen3-4B-Thinking achieves a higher EMR score than Qwen3-30B-A3B-Thinking, while Qwen3-30B-A3B-Thinking attains a higher TF1 score than Qwen3-4B-Thinking." These are comparative claims about specific model pairs on specific metrics. Without reported uncertainties, the reader cannot assess whether these differences exceed what would be expected from sampling variance across 613 test examples. If the EMR difference between the 4B and 30B models is 2 percentage points, and the standard error of the difference is 3 percentage points, the claim is not supported.

What evidence exists in the paper. None. The paper provides no uncertainty quantification. The quality control section (2.4) mentions that expert reviewers score samples on a 5-point Likert scale, but does not report inter-annotator agreement statistics (Cohen's kappa, Krippendorff's alpha, or raw agreement rate). The evaluation section (3.3–3.4) reports only point estimates in tables and figures.

Mitigation status. The paper does not acknowledge the absence of uncertainty quantification, does not attempt to estimate confidence intervals or conduct significance tests, and does not flag this as a limitation. This is a standard reporting practice in many NLP benchmark papers, but it is nonetheless a limitation — particularly given the small effective sample sizes in scenario-wise and difficulty-wise sub-analyses. A bootstrap-based confidence interval computation over the 613 samples (with stratification by sample type or scenario) would be straightforward to implement and would substantially strengthen the reliability of the reported comparisons.

7. Implications and Future Directions

How This Work Changes the Landscape

FinMCP-Bench does not introduce a new model, training method, or optimization algorithm. Its contribution is methodological infrastructure — but it is the kind of infrastructure that can redirect research attention and change evaluation norms. The paper's impact on the landscape operates at three levels: evaluation philosophy, diagnostic precision, and benchmark construction methodology.

Evaluation philosophy: process over output. The paper's most consequential shift is its explicit decoupling of tool invocation correctness from final answer quality. This is not a minor metric substitution — it is a conceptual reorientation of what it means for a financial agent to be "good." Prior work in financial NLP (CFBenchmark, CFLUE, CFinBench, FinanceReasoning, FinChain, INVESTORBENCH, M3FinMeeting, cited in Section 1) evaluates LLMs by checking whether they produce correct answers to financial questions — multiple-choice selections, numerical computations, extracted facts. FinMCP-Bench argues, through its design rather than through polemic, that for deployed financial agents, how the agent arrives at an answer is independently evaluable and diagnostically informative, even when what answer it gives cannot be objectively graded because financial advice is inherently open-ended.

This shift matters because it opens up evaluation to an entire class of tasks that were previously considered unevaluable by automated means. If "good financial advice" has no single ground-truth answer, the default evaluation approach — compare generated text to a reference answer — fails. But process evaluation — compare the tool invocation trace to a reference trace — works regardless of whether the final recommendation is objectively correct. The paper demonstrates this with 613 samples across 10 financial scenarios, none of which require answer-level ground truth. The implication is that any domain with structured, auditable processes but open-ended outputs — legal reasoning, medical diagnosis, enterprise software configuration, scientific literature review — could adopt the same evaluation philosophy.

Diagnostic precision: revealing capability profiles that aggregate metrics hide. The difficulty-behavior inversion in Figure 5 — stronger models achieving higher TF1 on Hard samples than on Easy ones — is a finding that could not have emerged from an answer-based evaluation. Answer accuracy would monotonically decrease with difficulty, obscuring the precision-recall dynamics that drive the inversion. The paper shows that tool-use evaluation can reveal how models fail, not just that they fail: the Easy-sample precision penalty (over-calling) and the Hard-sample recall benefit (richer constraints disciplining tool selection) are distinct failure and success modes that aggregate metrics collapse into a single number.

This diagnostic capability has downstream implications for model selection in deployment. A practitioner choosing between Qwen3-4B-Thinking and Qwen3-30B-A3B-Thinking — where the 4B model wins on EMR but the 30B model wins on TF1 (Table 2, Section 3.3) — needs to know whether their application cares more about exact structural matching (EMR) or balanced set-level accuracy (TF1). The paper provides this differentiation without prescribing a single "best" model, which is a more honest treatment of the deployment decision than a single leaderboard rank.

Benchmark construction methodology: real data as inductive bias for synthetic amplification. The paper's hybrid construction pipeline — extracting a tool dependency graph from real production traces, using that graph to constrain synthetic chain sampling, generating queries via in-context real examples, and producing reference trajectories via live execution against real tools — establishes a template for building domain-specific tool-use benchmarks that are simultaneously realistic (grounded in authentic user behavior and actual tool APIs) and challenging (amplified to include chains not observed in raw logs). The 50% yield at the automated validation stage and 37% yield at the expert review stage for synthetic multi-tool samples quantify the value of this multi-stage filtering: a purely generative pipeline without execution-based validation and expert review would produce a benchmark substantially contaminated with unanswerable or low-quality samples.

What the paper does not change. It does not provide evidence that tool invocation accuracy correlates with downstream deployment success — the link between TF1/EMR and user satisfaction, answer quality, or task completion remains an unvalidated assumption (a limitation discussed in Section 6). It does not resolve the data-generation circularity concern (Qwen3-235B-2507 builds the benchmark; Qwen3 models top the leaderboard). And it does not establish that FinMCP-Bench results generalize beyond the specific Qieman deployment ecosystem — the 65 tools, Chinese user base, and CSRC regulatory context may produce rankings that do not transfer. These are limits on the paper's scope, not flaws in its philosophy, but they mean the landscape shift is one of potential (the paper demonstrates a new kind of evaluation is possible and informative) rather than of accomplished consensus.

The research directions this work makes more attractive. The paper makes tool-invocation evaluation a credible research target in its own right, separate from answer-quality evaluation. This should accelerate work on:

Tool-use evaluation benchmarks in other high-stakes domains (legal, medical, enterprise) where output is open-ended but process is auditable.
Process-level verifiers for tool use — models trained to evaluate tool invocation correctness rather than answer correctness, analogous to how process reward models evaluate step-level reasoning in math.
Precision-recall diagnostics for agent behavior — disaggregated metrics that reveal whether an agent's errors are due to over-action (calling too many tools) or under-action (missing necessary tools), which have different remedies (prompt engineering for the former, capability improvement for the latter).

The research directions this work makes less attractive. The paper implicitly argues against two practices: (1) evaluating financial LLMs purely on multiple-choice or short-answer benchmarks without tool use, since these benchmarks measure a capability (static financial knowledge) that is increasingly decoupled from the capability that deployed financial agents actually exercise (dynamic tool orchestration); and (2) building synthetic tool-use benchmarks without execution-based validation or expert review, since the paper's 50-63% rejection rates at successive quality filters demonstrate that uncurated synthetic data is substantially contaminated.

Follow-Up Research This Work Enables or Suggests

1. Quantify the EMR ceiling by enumerating valid alternative tool chains for a representative subset. The Exact Match Rate metric penalizes any deviation from the reference tool grouping, but financial queries often admit multiple valid tool-calling strategies (e.g., parallel vs. sequential invocation of independent tools, or using a composite tool vs. multiple atomic tools). The paper does not estimate what fraction of benchmark samples have multiple valid strategies, so EMR values are uncalibrated — the gap between a model's EMR and 100% is an unknown mixture of genuine errors and valid alternative choices. A follow-up study should sample 50–100 FinMCP-Bench queries across scenarios and difficulty levels, have 3–5 financial domain experts independently enumerate all valid tool-chain organizations for each query, and compute both inter-annotator agreement on tool-chain validity and the EMR ceiling (the maximum achievable EMR if a model always produces a valid chain, but not necessarily the specific reference chain). This would calibrate the paper's reported EMR values and potentially reveal whether model rankings on EMR change after accounting for valid alternatives. If the ceiling is, say, 78% rather than 100%, then Qwen3-235B-A22B-Thinking's EMR on multi-turn samples might actually be close to optimal, reframing the paper's conclusion that multi-turn tool orchestration "remains challenging."

2. Cross-institution replication: rebuild the benchmark for a different financial platform and compare model rankings. The single-deployment source (Qieman / Yingmi Fund) is the paper's most significant generalizability concern. A compelling follow-up would partner with a second financial institution — ideally in a different jurisdiction, language, or regulatory regime (e.g., a U.S. robo-advisor, a European wealth management platform, or a Japanese securities firm) — and replicate the benchmark construction pipeline: extract tool dependency graphs from that institution's production logs, synthesize multi-tool and multi-turn samples using the same chain-based and role-playing methods, apply identical quality control, and evaluate the same six LLMs. The central question is whether model rankings transfer. If Qwen3-30B-A3B-Thinking tops both leaderboards, the paper's results are robust to deployment context. If rankings invert (e.g., GPT-OSS-20B leads on the second platform), then FinMCP-Bench measures platform-specific tool-use patterns rather than general financial tool-use capability, and the paper's framing as a "standardized testbed" is overly broad. This study would also reveal which benchmark design choices (tool vocabulary, dependency graph density, query language) are primary drivers of model ranking variation.

3. Train a tool-invocation verifier and compare its judgments against the reference traces. The paper's metrics compare predicted tool calls to reference traces, but reference traces are static — they cannot judge whether a predicted tool invocation that differs from the reference is nevertheless valid. A natural extension is to train a learned verifier — fine-tune one of the evaluated models (or a separate smaller model) on the benchmark's reference traces plus human judgments of alternative valid tool chains (from follow-up #1) — that takes a user query, the available tool set, and a predicted tool invocation trace as input, and outputs a binary valid/invalid judgment. This verifier could then be used to evaluate models on new queries without requiring reference traces, enabling evaluation at scale and on continuously evolving financial scenarios where static benchmarks become stale. The verifier's agreement with human expert judgments of tool-chain validity (on a held-out set of queries with multi-expert annotations) would quantify how well learned evaluation can substitute for reference-based evaluation, and would address the open-ended output problem more directly than the paper's current approach (which still requires reference traces for every sample).

4. Difficulty prediction as a pre-invocation capability test. The paper's difficulty taxonomy (Easy / Medium / Hard) is retrospective — it bins samples based on the reference tool-chain length, which is unavailable before invocation. A follow-up study could add a "difficulty prediction" subtask to FinMCP-Bench: before invoking any tools, the model must predict how many tool calls the query will require (or classify it into Easy / Medium / Hard). This tests a meta-capability that the current benchmark does not evaluate: can the model assess query complexity before committing to a tool-use strategy? Performance on this subtask could be correlated with downstream tool-invocation accuracy — models that correctly predict complexity may plan more effectively, while models that misclassify complex queries as simple may under-call tools. The study could also test whether providing the model with an oracle difficulty label (telling it "this query requires 8 tool calls" before invocation) improves TF1 and EMR, which would reveal whether tool-use errors are due to planning failures (the model doesn't know how many steps are needed) or execution failures (the model knows but cannot execute correctly). The paper's observation that stronger models improve on Hard queries suggests they may implicitly estimate difficulty — this subtask would make that capability explicit and measurable.

5. Stress-test the benchmark against adversarial or out-of-distribution queries. The benchmark's construction pipeline — chain-based synthesis, role-playing dialogue, execution-based trajectory generation — produces samples that, while realistic, follow predictable structural patterns (tool chains are paths through a fixed dependency graph; multi-turn dialogues follow persona-goal scripts). A follow-up should stress-test whether model rankings hold under distribution shift: generate adversarial queries that break common patterns (e.g., queries requiring tool chains with cycles or dead ends that the agent must detect and avoid; queries with ambiguous intent that require clarification before tool invocation; queries referencing tools or financial products not in the training set). If model rankings are stable under these perturbations, FinMCP-Bench measures a robust capability. If they invert — a model that excels at in-distribution benchmark queries collapses on adversarial ones — the benchmark overestimates real-world readiness. This study would also inform whether tool-use evaluation should include "negative samples" (queries that should trigger no tool calls, or should trigger an error response) to test whether models know when not to invoke tools.

6. Ablation of MCP-specific structure versus general function-calling capability. FinMCP-Bench is tied to the Model Context Protocol, but the relationship between MCP compliance and general tool-use capability is unexplored. A follow-up should construct a parallel version of the benchmark where the same 65 tools are described using a simpler, non-MCP function-calling schema (e.g., OpenAI-style function definitions with name, description, and JSON Schema parameters, but without MCP's server discovery, resource templates, or notification mechanisms). Evaluating the same six models on both versions would quantify how much of the performance variation is due to MCP-specific schema complexity versus general tool selection and orchestration capability. If models perform identically, FinMCP-Bench is effectively a general financial tool-use benchmark and the MCP framing is incidental. If performance drops substantially on the non-MCP version (or vice versa), then MCP imposes a learnable interface that some models handle better than others, and FinMCP-Bench results should be interpreted as MCP-specific rather than general.

Practical Applications and Downstream Use Cases

1. Regression testing for MCP server implementations in financial deployments. Financial institutions deploying MCP-based tool servers — wrapping their internal APIs (portfolio lookup, risk assessment, market data retrieval, compliance checking) in the MCP schema — can use FinMCP-Bench as a regression test suite. When the MCP server implementation changes (new tool added, tool schema updated, underlying API modified), re-evaluating the agent model on the benchmark can detect whether the change breaks tool invocation accuracy: a drop in TF1 or EMR signals that the model can no longer correctly orchestrate the updated tools. This is particularly valuable for financial institutions where tool server updates happen frequently (new data sources, changing regulations requiring new compliance checks) and where deployment errors have financial or regulatory consequences. The benchmark's 613 samples across 10 scenarios provide broader coverage than ad-hoc testing on a handful of manually written queries. The paper's quality control dimensions — especially "tool-chain logical consistency" and "answer reliability and traceability" — define a standard that regression tests should maintain.

2. Model selection for financial agent deployments. Organizations building financial LLM agents face a model selection problem: which of the available models (proprietary, open-weight, fine-tuned) should power the agent's tool-use layer? The paper's multi-dimensional evaluation (Table 2: 3 sample types × 4 metrics × 6 models) provides a template for this decision. A deployment team can weigh their priorities: if their application involves mostly single-turn, single-tool queries (e.g., a stock price lookup bot), the metrics that matter are single-tool TR, TP, and TF1 — over-calling precision losses may be acceptable if recall is high. If their application involves complex multi-turn advisory conversations (e.g., a retirement planning assistant), multi-turn EMR is the critical metric, and the paper's finding that Qwen3-4B-Thinking achieves higher EMR than Qwen3-30B-A3B-Thinking despite being smaller is directly actionable: the smaller, cheaper model may be the better choice for exact-match-critical deployments. The paper does not prescribe a single "best" model, but it provides the disaggregated metrics that make model selection informed rather than faith-based.

3. Training data generation for tool-use fine-tuning. The paper's chain-based synthesis pipeline (Section 2.2) — sampling tool chains from a dependency graph, generating user queries with in-context examples, and executing against a live MCP server to produce reference trajectories — is a recipe for generating fine-tuning data for tool-use models. An organization with a proprietary financial tool set can (1) construct a tool dependency graph from their own production logs (or expert specifications), (2) use the paper's method to generate thousands of (query, tool-chain) training pairs, (3) filter via execution-based validation (only retain pairs where a competent model can successfully execute the chain against live tools), and (4) fine-tune a smaller, cheaper model on these traces to imitate the tool-use behavior. The paper's finding that Qwen3-4B-Thinking — a 4B-parameter model — is competitive with much larger models on EMR suggests that tool-use capability may be distillable, and the synthesis pipeline provides the distillation data without requiring expensive human annotation of tool chains.

4. Diagnostic auditing of agent tool-use behavior in production. Deployed financial agents accumulate interaction logs — user queries, tool invocations, generated responses. These logs can be analyzed through the lens of FinMCP-Bench's metrics to detect degradation or anomalous behavior. A drop in Tool Precision over time (the agent starts calling more extraneous tools) could indicate model drift, a change in the user query distribution, or a tool server update that makes previously irrelevant tools appear more attractive. A drop in Tool Recall (the agent starts missing necessary tools) could indicate that new financial products or regulatory requirements have introduced tool dependencies that the agent was not trained on. The difficulty-wise analysis from the paper — the observation that over-calling is the dominant failure mode on simple queries — provides a diagnostic: if production logs show low precision concentrated on short, simple queries, the remedy is likely prompt engineering or post-hoc filtering rather than model retraining. The benchmark establishes baseline performance distributions (per scenario, per difficulty level, per metric) that production monitoring can compare against.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Toolformer: Language Models Can Teach Themselves to Use Tools2023· training-methods← LLM-judged similar
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning2023· training-methods← in-domain predecessor
Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos2026· evaluation← in-domain predecessor
- Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Models2026· evaluation← in-domain predecessor
Learning When to Act or Refuse: Guarding Agentic Reasoning Models for Safe Multi-Step Tool Use2026· alignment← topic thread
- Agentic Reinforced Policy Optimization2025· rl-training← LLM-judged similar
- CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production2026· alignment← in-domain predecessor
- D-CORE: Incentivizing Task Decomposition in Large Reasoning Models for Complex Tool Use2026· training-methods← topic thread
VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications2025· evaluation← similar topic
- ReviewScore: Misinformed Peer Review Detection with Large Language Models2025· evaluation← in-domain predecessor
- OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language World Models2026· evaluation← similar topic
- OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents2025· evaluation← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

BizGenEval: A Systematic Benchmark for Commercial Visual Content Generation2026· evaluation→ in-domain predecessor
- Consistency Amplifies: How Behavioral Variance Shapes Agent Accuracy2026· evaluation→ in-domain predecessor
- When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought2025· evaluation→ similar topic
- ViGoR-Bench: How Far Are Visual Generative Models From Zero-Shot Visual Reasoners?2026· evaluation→ similar topic
- Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs2026· evaluation→ similar topic
In-Context Reinforcement Learning for Tool Use in Large Language Models2026· training-methods→ topic thread
- How Far Can Unsupervised RLVR Scale LLM Training?2026· training-methods→ in-domain predecessor
- Rainbow: Combining Improvements in Deep Reinforcement Learning2000· training-methods→ similar topic
- Off-Policy Deep Reinforcement Learning without Exploration2018· training-methods→ similar topic
- Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning2022· training-methods→ similar topic
VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications2025· evaluation→ similar topic
- CodeClash: Benchmarking Goal-Oriented Software Engineering2025· code→ LLM-judged similar
- Vibe Checker: Aligning Code Evaluation with Human Preference2025· evaluation→ in-domain predecessor
- MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments2025· evaluation→ similar topic
- OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language World Models2026· evaluation→ similar topic
OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents2025· evaluation→ similar topic
- STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence2025· evaluation→ in-domain predecessor
- DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle2025· evaluation→ similar topic

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: We Don't Know How Well LLMs Use Financial Tools in Production

Why This Problem Matters: Financial Tool Use Is High-Stakes and Poorly Understood

Prior Approaches and Their Shortcomings

How the Paper Positions Itself Relative to Existing Work

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

Data Source and Real Sample Curation

Tool Dependency Graph Construction

Chain-Based Multi-Tool Sample Synthesis

Role-Playing-Based Multi-Turn Sample Construction

Quality Control

Sample Categorization and Difficulty Taxonomy

Evaluation Protocol

Evaluation Metrics: Tool Recall, Tool Precision, Tool F1, Exact Match Rate

Synthesis of Design Choices and Their Justifications

4. Key Insights and Innovations

Innovation 1: Tool Invocation Correctness as a First-Class Evaluation Target, Decoupled from Answer Quality

Innovation 2: Difficulty-Behavior Inversion — Why Stronger Models Improve on Harder Samples

Innovation 3: Real Production Data as the Foundation for Synthetic Augmentation, Rather than Synthetic-Only or Real-Only Construction

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Aggregate Model Performance Across Sample Types

Scenario-Wise Analysis

Difficulty-Wise Analysis

Summary of Key Quantitative Patterns

Ablation Studies and Robustness Checks

Critical Assessment

Does the benchmark actually measure tool invocation correctness?

Does the tool invocation evaluation capture what matters for deployment?

Does the benchmark support the claims about model capability?

What experiments would have strengthened the paper?

Summary of experimental strengths and weaknesses

6. Limitations and Trade-offs

Data-Generation Circularity: The Benchmark Was Partially Built by the Model Family That Tops the Leaderboard

Tool Invocation Evaluation Is Not Validated Against Any Downstream Measure of Agent Quality

Difficulty Estimation and Strategy Selection Cost Are Unaccounted For in the Headline Results

Single Domain, Single Deployment Source, Single Protocol — Generalization Is Unproven

The Exact Match Rate Metric May Penalize Valid Alternative Tool Chains, and This Ceiling Is Not Estimated

No Statistical Uncertainty Quantification for a 613-Sample Test Set

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables or Suggests

Practical Applications and Downstream Use Cases

Adjacent in domain

Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos

BizGenEval: A Systematic Benchmark for Commercial Visual Content Generation