ArXiv: 2510.24697

🎯 Pitch

Current LLM-based web agents are astonishingly wasteful—the paper reveals that roughly 96% of their search actions are useless. By training on dense, tree-structured tasks synthesized from Wikipedia tables and filtering for trajectories that maximize entities found per action, WebLeaper creates agents that are not only faster but strictly more accurate, outperforming Claude-4-Sonnet and OpenAI-o3 on several benchmarks while using fewer steps.

1. Executive Summary

This paper introduces WebLeaper, a framework for constructing entity-intensive information-seeking (IS) training tasks and generating efficient solution trajectories, addressing the problem that LLM-based agents suffer from low search efficiency due to sparse target entities in conventional training data. Training on data synthesized from curated Wikipedia tables—using three task variants: Basic (single-source tree-structured queries), Union (multi-source tasks requiring entity-set intersection), and Reverse-Union (obfuscated queries forcing deductive reasoning before search)—and filtering trajectories by Information-Seeking Rate (fraction of required entities retrieved) and Information-Seeking Efficiency (entities discovered per action), WebLeaper yields a compute-efficient information-seeking agent that consistently improves over strong baselines across five benchmarks (BrowseComp, GAIA, xbench-DeepSearch, WideSearch, Seal-0), achieving 38.8% on BrowseComp and 73.2% on GAIA under comprehensive training—surpassing proprietary agents like Claude-4-Sonnet and OpenAI-o3 on several benchmarks—while simultaneously reducing average action rounds, establishing that entity-rich training data with explicit efficiency filtering teaches agents to search both accurately and efficiently, though gains remain concentrated on tasks within the base model's capability range rather than on the hardest problems where even the base model's pass@1 is near zero.

2. Context and Motivation

The Core Problem: Information-Seeking Agents Search Inefficiently

The fundamental problem this paper tackles is that LLM-based information-seeking agents search inefficiently, executing many actions that contribute nothing to solving the task. The paper's preliminary experiments (Section 1, Figure 2) reveal a stark finding: the distribution of valid actions — actions that actually retrieve one of the required target entities — peaks around 0.04 for a competitive GPT-based agent. This means that in typical operation, only about 4% of the agent's actions are productive; the remaining 96% consist of redundant query reformulations, retrieval of irrelevant information, and unnecessarily long search chains. These inefficiencies compound as tasks grow more complex, increasing both computational cost and wall-clock time while simultaneously limiting overall task performance.

This inefficiency is not merely a cost concern. The paper argues — and demonstrates empirically — that low search efficiency directly constrains overall IS performance. When an agent wastes actions on irrelevant content, it has fewer remaining actions (or context budget) to find genuinely needed information before hitting environment constraints like maximum steps or context length. The connection between efficiency and effectiveness is bidirectional: efficient search enables reaching more correct entities, and learning to identify which entities matter accelerates search.

Why This Matters: Real-World Impact and Theoretical Significance

Practical deployment cost. Information-seeking agents underpin a rapidly growing class of commercial systems — OpenAI DeepResearch, Google's Gemini, Perplexity AI, Kimi-Researcher (Section 1) — that are deployed at scale. Each redundant action costs money (API calls, compute), time (sequential action chains increase latency), and context window tokens (irrelevant information consumes the increasingly valuable context budget of long-context LLMs). Making these agents 4×4\times more efficient means either a 4×4\times cost reduction at equal performance, or substantially better performance at equal cost. For production systems handling millions of queries, this efficiency gain translates directly to operational savings and improved user experience through faster responses.

Enabling longer-horizon tasks. Many real-world information-seeking tasks require gathering facts across dozens of sources (e.g., competitive analysis, literature surveys, due diligence research). Inefficient agents simply run out of steps before completing these tasks. The paper's focus on efficiency is therefore not just about doing the same work faster — it is about making feasible a class of long-horizon tasks that current agents cannot complete at all within their step budgets.

Theoretical significance: establishing training data semantics as a design dimension. The paper makes a conceptual contribution that extends beyond information-seeking agents: it argues that the semantic richness of training data — specifically, the number and density of target entities — is a critical and under-explored dimension of synthetic data design. Prior work on agent training primarily focused on increasing task structural complexity through multi-step reasoning chains or long-horizon planning. WebLeaper shifts attention to entity complexity: how many distinct pieces of information the agent must locate within a constrained context. This reframing connects agent training to the broader question of how to embed rich supervision signals within finite training examples, a problem that arises in many RL and imitation learning settings beyond web navigation.

Where Prior Approaches Fall Short

The paper identifies specific limitations in prior work along several axes:

Training tasks have sparse target entities. Existing synthetic IS training datasets (WebSailor, WebDancer, WebShaper) typically construct questions that require finding only a small number of target entities — often single-fact answers or short lists (Section 1). The paper explicitly states:

"prior work often constructs tasks in which the target entities are overly sparse... Such sparsity limits the agent's exposure to informative cues, reducing opportunities to learn to locate relevant information within a constrained context window."

This sparsity has two consequences. First, it provides a weak training signal: if a task only requires finding 3 entities, the agent receives meaningful feedback on only a handful of its actions, leaving most of its trajectory unguided. Second, it creates a measurement bias in efficiency metrics. The paper formalizes this mathematically in Proposition 1 (Section 2.3): the variance of Information-Seeking Efficiency (ISE) scales as Var(ISE)=O(1/n)\text{Var}(\text{ISE}) = O(1/n), where nn is the number of target entities. When nn is small (e.g., 3-5 entities in typical prior work), ISE estimates are noisy and unreliable as training signals or evaluation metrics. When nn is large (e.g., 50-100+ in WebLeaper), ISE becomes a stable, trustworthy measure of search quality. This is not merely a measurement convenience — it is a fundamental requirement for learning efficiency: if you cannot reliably measure efficiency, you cannot reliably train for it.

Focus on search depth over search efficiency. The paper observes that "most existing approaches primarily concentrate on improving the search depth, giving comparatively little attention to search efficiency" (Section 1). Prior work extended the search process through multi-step decomposition (WebResearcher), knowledge graph integration (DeepDive), or increasingly complex verification pipelines, but these approaches multiply the number of actions without systematically optimizing which actions contribute value. The implicit assumption was that deeper search is always better — a claim this paper challenges by showing that adding inefficient actions can actually degrade performance by consuming context and introducing noise.

Shortcut vulnerability in multi-source tasks. An important limitation identified in prior multi-source integration approaches (Section 3.1.3): even when tasks require information from multiple sources, agents can often solve them through simple keyword search without genuine information synthesis. The paper cites WebSailor's approach as an example where tasks intended to require cross-source reasoning could be solved by querying each source independently and pattern-matching. The authors argue this "circumvents the intended synthesis of information, reducing the cognitive load and failing to stimulate true reasoning capabilities." This observation motivates the Reverse-Union task variant, which explicitly blocks such shortcuts through deductive fuzzing.

Reward sparsity in RL for entity-intensive tasks. In Section 3.3, the paper identifies a specific failure mode of standard RL approaches when applied to entity-intensive tasks. Binary success/failure rewards — the dominant approach in prior work — become "so rare that effective learning becomes nearly impossible" when a task requires retrieving dozens of entities; the agent almost never achieves perfect completion during early training, receiving no positive signal whatsoever. This is a more severe version of the standard sparse-reward problem in RL, exacerbated by the entity-intensive task design that WebLeaper itself introduces. The paper's hybrid reward system is a direct response to this self-created challenge.

Model size vs. data quality. An implicit but important gap the paper addresses: many competitive open-source agents rely on large-scale models (e.g., Kimi-K2-Instruct-1T at much larger parameter counts) yet underperform WebLeaper trained on a modest 30B-parameter model (Qwen3-30B-A3B-Thinking-2507). This suggests that data quality and training objective design can compensate substantially for model scale in the information-seeking domain — a finding that parallels the Chinchilla-style insight in pretraining (data quantity matters more than model size at a given compute budget) but applied to the semantic structure of training data rather than its volume.

How This Paper Positions Itself

The paper positions WebLeaper not as an incremental improvement to existing IS agent training, but as a fundamentally different approach to constructing training data that addresses the root cause of inefficiency: the sparsity of target entities.

The tree-structured reasoning formulation is the conceptual foundation. Rather than treating IS tasks as flat question-answer pairs with an unstructured search process, the paper models each task as a tree where nodes are entities and edges are relations (Section 3.1). This formulation enables two critical innovations: (1) it compactly represents a large number of target entities (all nodes in a subtree) within a single structured task, and (2) it makes explicit the reasoning path the agent must follow from known question entities through intermediate entities to final answers. This is a departure from prior work where the "reasoning structure" was implicit in multi-step prompts rather than embedded in the data generation process itself.

Three-tier task complexity is deliberate, not ad hoc. The Basic, Union, and Reverse-Union variants are not independent dataset contributions — they form a curriculum of increasing cognitive demands (Section 3.1.1-3.1.3):

  • Basic establishes the tree-structured entity-density baseline. Tasks are single-source and structurally simple, but contain many more target entities than prior work (often 50-100+ entities from a single Wikipedia table). The paper acknowledges that Basic alone performs worse than the WebSailor-V2-5k baseline (Table 2, -7.64 average) because it creates shortcut-prone overly simple patterns — but it serves as the building block for more complex variants.

  • Union addresses the single-source limitation by algorithmically discovering pairs of reasoning trees that share common subtree relations, enabling questions that require finding entity-set intersections across sources (e.g., "authors who won both the Nobel Prize and the Booker Prize"). This forces the agent to retrieve two separate entity sets and compute their overlap, which cannot be done through a single keyword search. The algorithmic discovery via maximal biclique enumeration (Algorithm 1, Appendix A.5) is a non-trivial contribution that distinguishes WebLeaper from ad-hoc multi-source data augmentation.

  • Reverse-Union addresses the shortcut vulnerability in Union by obfuscating the entry point. Instead of naming the search target directly, it describes an "anchor" entity through third-layer attributes that require deductive reasoning to identify, then uses that anchor's properties as the pivot for the main search. This enforces a two-stage cognitive workflow (deduce anchor, then search) that cannot be shortcut by keyword matching. The paper explicitly positions this as a defense against the circumvention patterns observed in prior work.

Efficiency filtering transforms the training objective. Prior work trained agents on any trajectory that reached the correct answer, regardless of how many extraneous actions were taken. WebLeaper introduces explicit efficiency (ISE) and coverage (ISR) thresholds during trajectory curation (Section 3.2), retaining only trajectories that are both accurate and efficient. This is a philosophical shift: the training objective is no longer "solve the task" but "solve the task with minimal wasted actions." The paper provides evidence that ISR+ISE filtering outperforms either criterion alone on complex benchmarks like GAIA and BrowseComp (Figure 4), though on broad-search benchmarks like WideSearch the differences are within variance — suggesting the efficiency requirement matters most when tasks require precision rather than breadth.

The hybrid reward system bridges entity-intensive and standard tasks. Section 3.3's hybrid reward design — granular F-score for WebLeaper tasks, binary rewards for legacy benchmarks — positions WebLeaper as a compatible augmentation rather than a replacement for existing training pipelines. This is a pragmatic design choice that enables fair comparison: the WebLeaper agent is trained on the same legacy data as baselines plus additional entity-intensive data, so performance differences directly reflect the contribution of the WebLeaper data rather than training recipe differences. The granular F-score itself addresses a recognized limitation in RL for IS: standard exact-match rewards are too brittle for entity-list outputs (penalizing "USA" vs. "United States"), while LLM-as-judge approaches are too expensive for frequent RL updates. The soft precision/recall formulation with type-specific semantic scoring functions is a middle ground that balances accuracy and computational cost.

Connecting to the broader scaling narrative. The paper implicitly positions itself in a lineage that extends from the Chinchilla scaling laws (optimal allocation of pretraining compute) through inference-time scaling analyses to training data efficiency. The key claim is that entity richness in training data functions analogously to data quantity or model size in pretraining: increasing it yields disproportionate improvements in downstream capability, and ignoring it leaves significant performance on the table. The mathematical formalization in Proposition 1 — showing that measurement reliability scales with entity count — provides theoretical grounding for what might otherwise appear as an engineering heuristic. This is an emerging direction in the agent training literature (parallel work like WebSailor-V2 and ASearcher also explores data scaling levers), and WebLeaper's contribution is specifically identifying and exploiting entity density as the scaling dimension.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

WebLeaper is a data synthesis and training framework that produces synthetic question-answer pairs and efficient solution trajectories specifically designed to teach LLMs how to search the web both accurately and with minimal wasted actions. It solves the problem that current information-seeking agents waste approximately 96% of their actions on irrelevant searches by constructing training tasks that embed many more target entities per question — typically 20 to 100+ entities instead of the 3–5 common in prior work — and then filtering training trajectories to keep only those that retrieve a high fraction of required entities using as few steps as possible. The solution takes the form of a pipeline that mines structured Wikipedia tables, assembles them into reasoning trees, algorithmically identifies opportunities to combine trees for multi-source tasks, generates natural-language questions from these trees, collects solution trajectories from a base LLM, filters those trajectories using two novel efficiency metrics, and finally trains an agent through supervised fine-tuning followed by reinforcement learning with a hybrid reward that provides granular feedback on entity-level retrieval quality.

3.2 Big-Picture Architecture (Diagram in Words)

The WebLeaper system has five major components connected in a pipeline:

  1. Wikipedia Table Extraction and Cleaning (Section 3.1.1, Appendix A.4): crawls approximately 2 million Wikipedia tables, filters for quality and structural homogeneity, and transforms well-formed tables into three-layer reasoning trees where the root is the table topic, the second layer is a key column (e.g., person names), and the third layer contains attribute values from remaining columns.

  2. Task Synthesis Engine (Sections 3.1.1–3.1.3, Appendices A.5–A.6): generates three increasingly complex task variants — Basic (single-table tree-structured queries), Union (algorithmically discovered pairs of trees sharing common relations, requiring entity-set intersection), and Reverse-Union (obfuscated queries forcing deductive reasoning before search) — by programmatically constructing reasoning trees and then prompting an LLM to verbalize them into natural-language questions with full entity-answer sets.

  3. Trajectory Collection and Filtering (Section 3.2): executes the synthesized tasks with an open-source base model under the ReAct framework (using Search and Visit tools), collects full action-observation trajectories, and filters them using Information-Seeking Rate (ISR, the fraction of required entities retrieved, threshold $\alpha = 0.3$) and Information-Seeking Efficiency (ISE, entities discovered per Visit action, threshold $\beta = 0.1$), retaining only trajectories that are both accurate and efficient.

  4. Supervised Fine-Tuning (SFT; Section 3.3): trains the base model on the filtered trajectories using standard next-token prediction, producing a policy that already exhibits improved search behavior but can still be optimized further.

  5. Reinforcement Learning with Hybrid Reward (Section 3.3): further optimizes the SFT policy using Group Relative Policy Optimization (GRPO) with a hybrid reward function that provides granular F-score-based feedback on entity retrieval for WebLeaper tasks (using soft precision and recall with type-specific semantic scoring) while retaining standard binary rewards for legacy benchmark data, enabling the agent to refine its search strategies based on dense, entity-level signals.

Information flows as follows: raw Wikipedia tables enter the cleaning pipeline and emerge as structured reasoning trees → the task synthesis engine transforms trees into QA pairs with large entity answer sets → the base model generates thousands of trajectories on these tasks → the ISR/ISE filters discard inefficient or incomplete trajectories → the surviving trajectories train the model via SFT → the SFT-ed model generates new trajectory batches for RL → the hybrid reward evaluates each trajectory at the entity level → GRPO updates the policy to favor trajectories with higher-than-average rewards within each batch.

3.3 Roadmap for the Deep Dive

  • First, the tree-structured reasoning formulation (Section 3.1 preamble): this is the conceptual substrate for everything that follows. Understanding how entities, relations, and subtrees are organized explains why the three task variants differ and what makes each progressively harder.

  • Second, the Basic task synthesis (Section 3.1.1): the simplest variant establishes the entity-density mechanism — how a single Wikipedia table becomes a question with dozens of target entities. This is the building block for Union and Reverse-Union.

  • Third, the Union task synthesis (Section 3.1.2): the algorithmic core of the paper — how maximal biclique enumeration on bipartite graphs discovers pairs of reasoning trees that share common subtree relations, enabling multi-source tasks that require entity-set intersection. The algorithm matters because it automates what would otherwise be manual or heuristic data augmentation.

  • Fourth, the Reverse-Union task synthesis (Section 3.1.3): the most complex variant, which addresses the shortcut vulnerability in Union by obfuscating the starting point. Understanding the "deductive fuzz" and "union-based search construction" stages explains why this variant produces the strongest agent.

  • Fifth, trajectory collection and filtering (Section 3.2): this is where the theoretical metrics (ISR, ISE) become operational filtering criteria. Understanding the Search/Visit tool interface and why ISE only counts Visit actions (not Search) reveals a crucial design insight.

  • Sixth, the hybrid reward system and GRPO optimization (Section 3.3): the RL component that makes entity-density training tractable. The soft precision/recall formulation with type-specific semantic scoring is the key engineering contribution that prevents the sparse-reward problem from derailing learning on entity-intensive tasks.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data synthesis and training methodology paper whose core idea is that constructing training tasks with high entity density — embedding many target entities within a constrained tree-structured reasoning space — and filtering training trajectories by both coverage and efficiency metrics produces information-seeking agents that search both more accurately and more efficiently than those trained on entity-sparse tasks.

The Tree-Structured Reasoning Formulation

Every information-seeking task in WebLeaper is modeled as a reasoning tree — a directed, layered graph where nodes represent entities and edges represent relations between entities. The tree has three layers by construction (though the conceptual framework generalizes to deeper trees):

  • Layer 1 (Root / Question Entities): the entities explicitly named or described in the question that serve as the starting point for search. For example, in the Nobel Prize task, these are "Nobel Prize in Literature" and "1980–1990." The agent knows these entities before beginning search; they are the "given" information.

  • Layer 2 (Subtree Roots / Intermediate Entities): entities that are directly connected to the question entities via a membership or inclusion relation. For a Nobel Prize winners table, these are individual laureate names like "Czesław Miłosz" and "William Golding." Each such entity is the root of its own subtree carrying all its properties. The agent must discover these entities through search; they are not given in the question.

  • Layer 3 (Leaves / Attribute Entities): entities representing properties of the Layer-2 entities, connected via attribute relations like "has_nationality," "has_award_year," or "has_gender." For Czesław Miłosz, these would be "Poland," "1980," and "male." The agent must retrieve these attributes for each discovered Layer-2 entity.

A subtree $S_{i,j}$ is the collection of a single Layer-2 entity plus all its connected Layer-3 entities and their relations. The full reasoning tree $T_i$ is the union of all subtrees that share a common Layer-1 root. Critically, the ground-truth answer set $R$ (the target entities the agent must retrieve) includes all Layer-2 entities and all their Layer-3 attribute entities — not just a few selected facts. For a Nobel Prize table covering 11 years with multiple winners per year, this could easily mean 50–150 distinct entities (10–20 laureates × 3–5 attributes each).

This tree formulation enables entity density because it exploits the natural hierarchical structure of tabular data. A single Wikipedia table with 50 rows and 6 columns compactly encodes 250+ entity-relation triples. By making the entire subtree the answer set, WebLeaper packs substantially more supervision signal into each training task than prior approaches that extracted only one or two facts per table.

Version-I: Basic Task Synthesis

The Basic variant constructs a complete reasoning tree and its corresponding QA task from a single cleaned Wikipedia table.

Table selection and cleaning. The authors crawl approximately 2 million tables from Wikipedia articles. A multi-stage cleaning procedure (detailed in Appendix A.4) discards tables that fail size constraints (fewer than 10 rows or 3 columns; more than 200 rows or 20 columns), contain semantically irrelevant columns (serial numbers, notes, references), or have structural defects (merged cells disrupting the relational structure). The surviving tables are further filtered for isomorphism: only groups of tables sharing the same column headers and structure are retained, because structural homogeneity is a prerequisite for the Union operation's algorithmic discovery (Section 3.1.2).

Layer population. From a single cleaned table, the three layers are populated as follows:

  1. Layer 1 (Question Entities): extracted from the table's title or caption. The paper's example uses "Nobel Prize in Literature" plus the constraint "1980–1990" as the root entities.

  2. Layer 2 (Subtree Roots): an LLM is employed to select "the most representative, non-redundant column of values from the table — typically the primary key" (Section 3.1.1). This column must contain no duplicate entries, ensuring each value uniquely identifies a subtree. For the Nobel table, this is the laureates' names. Each unique value becomes a second-layer entity and the root of a subtree.

  3. Layer 3 (Leaves): all values from the remaining columns of the table, connected to their corresponding Layer-2 entity. For the Nobel table, these would be nationality, award year, gender, and any other columns present. The relations connecting Layer 2 to Layer 3 are the column headers (e.g., "has_nationality," "has_award_year").

Question generation. With the tree populated, an LLM is prompted to generate a natural-language question that names the Layer-1 entities and asks for all Layer-2 entities and their attributes. The paper's running example (Appendix A.6.1) produces:

"Who were the Nobel Prize winners in Literature between 1980 and 1990? Please include their name, country, award year, and gender."

Answer set construction. The target entity set $R$ is the complete collection of all Layer-2 entities and all their Layer-3 attribute values — not merely the names. This means the agent must retrieve every laureate and every requested attribute for each, producing an answer set that might contain 50–100+ distinct entity strings.

Design rationale. The key insight is that by making the entire subtree the answer set rather than extracting a single fact, WebLeaper dramatically increases the entity density of each training task. A task with 10 laureates and 4 attributes each requires retrieving 40+ entities, providing 40+ opportunities for the ISR/ISE metrics to evaluate the agent's search quality at a granular level. This directly addresses the Proposition 1 insight: larger $n$ (number of target entities) yields more stable ISE measurements, making efficiency training feasible.

Limitation acknowledged by the paper. Basic tasks are recognized as structurally simple — they come from single sources and follow a predictable hierarchical pattern. The ablation in Table 2 confirms this: training on Basic data alone produces substantially worse performance than the WebSailor-V2-5k baseline (average -7.64 across five benchmarks), because the model learns shortcuts (find the one relevant table, extract all rows) rather than genuine multi-source information integration. Basic is therefore a building block, not a standalone training dataset.

Version-II: Union Task Synthesis

The Union variant constructs tasks that require integrating information from multiple reasoning trees by finding and synthesizing their overlapping entity sets. This addresses the single-source limitation of Basic while avoiding ad hoc or heuristic data combination.

The core algorithmic problem. Given a collection of $N$ basic reasoning trees $T_{\text{base}} = \{T_1, T_2, \dots, T_N\}$ — each representing a different Wikipedia table — the goal is to find groups of trees whose subtrees share common relational structures, enabling questions about entities that satisfy conditions across multiple trees (e.g., "authors who won both the Nobel Prize and the Booker Prize"). A naive approach of enumerating all possible tree combinations would be combinatorially explosive ($O(2^N)$). Instead, the problem is reformulated as Maximal Biclique Enumeration on a bipartite graph.

Bipartite graph construction. A bipartite graph $G = (U, V, E)$ is constructed where:

  • $U = T_{\text{base}}$ is the set of all basic reasoning trees (one partition).
  • $V$ is the set of all unique standardized relation names that appear within subtrees across all trees in $T_{\text{base}}$ (the other partition). A "relation name" is a standardized version of a column header / edge label connecting Layer-2 to Layer-3 entities (e.g., "has_nationality," "has_award_year").
  • An edge $(T_i, v_j) \in E$ exists if and only if the relation $v_j$ appears in any subtree of tree $T_i$.

In this construction, a biclique is a pair of subsets $(U', V')$ where every tree in $U'$ contains every relation in $V'$. A maximal biclique is a biclique that cannot be extended by adding another tree to $U'$ or another relation to $V'$ without breaking the "every tree has every relation" property. Each maximal biclique represents a group of trees ($U'$) that all share a common set of relations ($V'$), making them semantically compatible for union — they describe different entity sets (e.g., Nobel laureates, Booker winners) but share structural properties (e.g., both have nationality, name, and award year fields).

Formal optimization. The problem is formally stated as:

find maximal (U,V) subject to ∀Ti∈U,V⊆Rel(Ti),∣U∣≥kmin,∣V∣≥mmin\text{find maximal } (U, V) \text{ subject to } \forall T_i \in U, V \subseteq \text{Rel}(T_i), |U| \geq k_{\text{min}}, |V| \geq m_{\text{min}}

where $k_{\text{min}}$ is the minimum number of trees in a valid union group and $m_{\text{min}}$ is the minimum number of common relations. The paper does not specify the exact values of $k_{\text{min}}$ and $m_{\text{min}}$ used in experiments, but the constraints ensure that union groups are neither trivial (single trees) nor structurally vacuous (no shared relations).

What the equation computes: given all basic trees and all their subtree relations, the algorithm searches the bipartite graph for subsets of trees and subsets of relations such that every tree in the subset contains every relation in the relation subset, the subsets are as large as possible (maximal — no tree or relation can be added without breaking the property), and they meet minimum size thresholds. The output is a set of maximal bicliques, each defining a candidate union group.

Why this form: biclique enumeration is a well-studied graph algorithm problem with efficient solvers (the paper references standard algorithms like MICA or Eclat in Algorithm 1, Appendix A.5). Reformulating the combinatorial search over tree combinations as biclique enumeration on a bipartite graph converts an exponential brute-force search into a structured graph traversal problem. The maximality constraint ensures that discovered groups are as inclusive as possible — partial overlaps are subsumed by larger groups — avoiding redundant task generation from overlapping subgroup combinations.

Semantic filtering. After a maximal biclique is identified, a semantic compatibility check is applied to the Layer-2 entities of the candidate trees. For relations connecting Layer 2 to Layer 3, the matching criteria require shared standardized name, data type, and domain. For the Layer-2 entities themselves, the constraint is relaxed to require only matching data type and domain — not identical names — allowing the union of trees with semantically equivalent but differently named root concepts (e.g., "Authors" in one table and "Writers" in another are compatible because both have the same type and domain).

Question generation from unions. Once a maximal union group $\langle U', V' \rangle$ is identified (e.g., the Nobel Prize table and the Booker Prize table sharing "has_nationality" and "has_name"), an LLM is prompted to synthesize a question based on the common features of the selected subtrees. The LLM identifies a meaningful relationship across the trees — typically the intersection of their Layer-2 entity sets — and produces a question that requires finding that intersection. The running example (Appendix A.6.2):

"Which authors have won both the Nobel Prize in Literature and the Booker Prize? For each, provide their name, nationality and the year they won the Nobel."

Required reasoning path. To solve a Union task, the agent must execute three sequential operations that cannot be collapsed into a single search:

  1. Retrieve the first entity set: identify the first concept (Nobel Prize in Literature) and retrieve the complete set of corresponding Layer-2 entities from the first source tree.

  2. Retrieve the second entity set: identify the second concept (Booker Prize) and retrieve the complete set of Layer-2 entities from the second source tree.

  3. Compute the intersection: identify which entities appear in both sets and retrieve the requested attributes for those entities from the relevant source(s).

Why this is harder than Basic: the agent must now navigate to at least two distinct information sources, each potentially requiring multiple search and visit actions. The intersection step requires holding both entity sets in memory (or revisiting sources) and performing a set operation — a higher cognitive load than single-source extraction. The entity density is also typically higher because the final answer set includes entities from the intersection of two sources plus their attributes.

Version-III: Reverse-Union Task Synthesis

The Reverse-Union variant addresses a specific vulnerability of the Union approach: an agent could solve Union tasks through naive keyword search without genuine information synthesis. For example, for the Nobel/Booker intersection question, an agent could simply search "Nobel Prize in Literature winners" and "Booker Prize winners" independently, extract both lists by pattern matching, and compute the overlap without understanding why those authors are connected or reasoning about the relationships between sources. Reverse-Union forces a more robust cognitive workflow by obfuscating the starting point — the agent cannot search for the root concepts directly because they are not named in the question.

Stage 1: Deductive Fuzz. This stage selects a specific "anchor" entity — an entity from the intersection of two trees' Layer-2 sets — and replaces direct naming with descriptive clues derived from its Layer-3 attributes. In the running example (Appendix A.6.3), William Golding (a laureate present in both the Nobel and Booker prize trees) is the anchor. Instead of naming him, the question provides:

"the 1980s prize-winner that wrote a novel about a group of British boys stranded on an uninhabited island"

These clues are third-layer entities from Golding's subtree — "1980s" (award year), "novel about stranded British boys" (a distinguished work, derivable from an attribute or external knowledge) — that uniquely identify him but require the agent to perform deductive reasoning: searching for "author who wrote Lord of the Flies" to identify Golding, then verifying he won the Nobel in the 1980s. This deductive step cannot be shortcut by searching for "Nobel Prize winners" directly, because the question never mentions the Nobel Prize in the initial clues.

Stage 2: Union-Based Search Construction. Once the anchor is established (implicitly, through the clues), the question uses a specific third-layer attribute of the anchor as a pivot for the main search. In the example, Golding's nationality (British, a third-layer attribute from his subtree, extracted from the "has_nationality" relation) becomes the pivot. The question then asks:

"Who are the authors from the same country as the [deduced anchor], and who have also won both this reward and the Booker Prize?"

The agent must:

  1. Deduce that the clues refer to William Golding.
  2. Identify Golding's nationality (British) from his subtree.
  3. Use "British" as a filter on the Nobel/Booker intersection set to find all British authors who won both prizes.
  4. Retrieve the requested attributes for each qualifying author.

Required reasoning path. The full solution requires a two-stage cognitive workflow:

  • Deduction Stage: resolve the descriptive third-layer clues to identify the anchor Layer-2 entity. This requires searching for the described attributes ("novel about stranded British boys" → "Lord of the Flies" → William Golding) and cross-referencing with the time constraint (1980s Nobel winner). The agent must reason from properties to entity identity — the reverse direction of standard search, which goes from entity name to properties.

  • Union Stage: having identified the anchor, extract its relevant attribute (nationality = British), then treat this as a pivot attribute. The agent must search for the Nobel and Booker prize winner sets (as in the Union variant), compute their intersection, and filter the intersection by the pivot attribute (same nationality as the anchor). The final answer set includes only entities that satisfy all three conditions: won Nobel, won Booker, and share the anchor's nationality. In the paper's example, this yields William Golding, Kazuo Ishiguro, and J.M. Coetzee — all British authors who won both prizes.

Why this is the hardest variant. Reverse-Union combines three distinct cognitive demands that each tax different aspects of information-seeking capability:

  1. Deductive reasoning from properties to entities: the reverse of standard search, requiring the agent to infer what entity is described rather than retrieve an entity's description. This is analogous to solving a riddle or answering a "Who am I?" question.

  2. Attribute extraction and repurposing: the anchor entity is not the final answer; its attributes are merely tools (the pivot) for the next search phase. The agent must recognize that nationality is the relevant pivot attribute rather than, say, award year or gender, and use it to construct the next query.

  3. Multi-condition entity filtering: the final answer set must satisfy intersection (Nobel ∩ Booker) and an additional filter (British). This is a relational algebra query over entity sets, requiring the agent to manage intermediate results and apply cascading constraints.

The paper's experimental results (Table 2) confirm that Reverse-Union produces the strongest agent: Reverse-Union-10k† achieves an average improvement of +4.34 over WebSailor-V2-5k across five benchmarks, compared to +3.26 for Union-5k† and -7.64 for Basic-5k†. The Reverse-Union gain is especially pronounced on xbench-DS (+6.00) and WideSearch (+10.92), benchmarks that emphasize multi-step reasoning and broad search respectively — consistent with Reverse-Union training both deductive reasoning and expansive search capabilities.

Information-Guided Trajectory Construction

After tasks are synthesized, the next phase is collecting solution trajectories and filtering them to retain only those that demonstrate both accuracy and efficiency. This is the mechanism that transforms WebLeaper from a data augmentation technique into a training methodology with a specific performance objective (efficient accuracy, not just accuracy).

Agent tools and interaction protocol. The agent operates within the ReAct framework (Yao et al., 2023) — a sequential decision-making loop where at each step $t$, the agent (1) produces a planning thought $\tau_t$, (2) executes a tool-based action $\alpha_t$, and (3) receives an observation $o_t$ from the environment. The trajectory is the full sequence:

HT=(q,τ1,α1,o1,τ2,α2,o2,…,τT,αT,oT)H_T = (q, \tau_1, \alpha_1, o_1, \tau_2, \alpha_2, o_2, \dots, \tau_T, \alpha_T, o_T)

where $q$ is the initial question, $T$ is the total number of steps, and each step adds a thought-action-observation triple.

The agent is equipped with exactly two tools:

  • Search: takes parameters {queries, filter_year}, executes a Google search with the specified query strings and optional temporal filtering, and returns the top relevant URLs with their snippets as the observation. Multiple queries can be issued in a single Search action. The temporal filtering parameter enables constrained searches like "Nobel Prize winners 1980–1990."

  • Visit: takes parameters {urls, goal}, fetches and summarizes the content of the specified URLs, and returns summarized paragraphs as the observation. Multiple URLs can be visited in a single action. The goal parameter provides the Visit action with context about what information the agent is seeking, which helps the summarization focus on relevant content.

The two-tool design reflects the paper's assumption that the agent interacts with a standard web search interface: first discover relevant pages via search engine queries, then read their content. The batching of multiple queries or URLs per action is an efficiency optimization that reduces the total number of action steps, but each batched action still represents a discrete interaction with the environment.

Trajectory generation. The synthesized tasks are executed by an open-source base model — the paper uses Qwen3-30B-A3B-Thinking-2507 as the base — producing a large set of completed trajectories. Each trajectory represents one complete attempt at solving the task, which may succeed (all required entities found), partially succeed (some entities found), or fail (few or no entities found, or incorrect entities retrieved). The trajectories vary naturally in both completeness and efficiency because the base model's search strategy is not yet optimized.

Information-Seeking Rate (ISR) computation. For each trajectory, ISR quantifies how thoroughly the agent retrieved the required entities:

ISR=∣R∩O∣∣R∣=∣R∩O∣n\text{ISR} = \frac{|R \cap O|}{|R|} = \frac{|R \cap O|}{n}

where $R$ is the set of ground-truth target entities defined by the task construction (all Layer-2 and relevant Layer-3 entities), $O$ is the set of entities extracted from the agent's observations across all actions (the union of all unique entities the agent encountered), $|R \cap O|$ is the number of required entities the agent successfully encountered, and $n = |R|$ is the total number of required entities.

What it computes: ISR is the fraction of required entities that the agent's search process encountered at any point — regardless of whether the agent correctly identified them as answers, and regardless of how many steps it took. ISR $\in [0, 1]$, where 1.0 means the agent's trajectory covered every required entity at least once and 0.0 means it missed all of them.

Why this form: ISR measures coverage rather than correctness of the final answer output. This is an important distinction. The trajectory filtering criterion (Section 3.2) uses ISR to ensure that retained trajectories had the right information in the agent's view at some point, whether or not the agent correctly assembled it into a final answer. This is appropriate for the filtering stage because the goal is to select trajectories where the search process was on the right track — the SFT training will then teach the model to extract and report the encountered entities correctly. Using final-answer correctness for filtering would be circular: it would only retain trajectories from a model that is already good at extracting answers, providing no signal for improving the search process that precedes extraction.

The coverage criterion. A trajectory is retained only if $\text{ISR} > \alpha$, where the paper specifies $\alpha = 0.3$ in the training configurations (Section 4.1). This means the trajectory must have encountered at least 30% of the required entities. For a task with $n = 100$ target entities, the agent must have seen at least 30 of them in its observations. This threshold is deliberately permissive — it does not require near-perfect coverage — because the goal is to retain trajectories that demonstrate some effective search behavior while still filtering out trajectories that were completely off-track.

Information-Seeking Efficiency (ISE) computation. ISE quantifies how quickly the agent discovered required entities:

ISE=nT\text{ISE} = \frac{n}{T}

where $n = |R|$ is the total number of required entities (as in ISR) and $T$ is the total number of steps in the trajectory. ISE is measured in units of "entities discovered per step" and is higher when the agent finds all required entities in fewer steps.

What it computes: ISE is the average discovery rate of required entities across the entire trajectory. If a task has 50 required entities and the agent takes 10 steps, ISE = 5.0 — meaning the agent discovered an average of 5 required entities per action step. If it takes 100 steps for the same task, ISE = 0.5.

Why this form: ISE penalizes trajectories that use many actions to achieve the same coverage. A trajectory with ISR = 1.0 (found everything) but ISE = 0.5 (took 100 steps for 50 entities) is less desirable for training than one with ISR = 1.0 and ISE = 5.0 (took 10 steps for the same 50 entities). By filtering on ISE as well as ISR, WebLeaper selects for trajectories that were not just successful but efficiently successful — providing training examples of what good search behavior looks like in terms of action economy.

Critical design detail: ISE is computed only on Visit actions. The paper explicitly states (Section 3.2):

"For ISE, we accumulate the obtained target entities in Visit actions. The reason for not including Search in ISE is that we observe entities found in Search are less precise and would be updated by the following Visit action."

This means that $T$ in the ISE formula is actually the count of Visit actions only, not all actions. Entities discovered during Search actions (from snippets) are not counted toward ISE, only entities found after visiting and reading the full page content. The rationale is that search snippets are noisy — they may mention an entity but provide incomplete or incorrect context — and the agent only truly "discovers" the entity after visiting the page to confirm and extract the full information. This is a pragmatic design choice that prevents inflated ISE scores from trajectories where the agent sees many entity names in snippets but never actually reads about them.

The efficiency criterion. A trajectory is retained only if $\text{ISE} > \beta$, where the paper specifies $\beta = 0.1$ (Section 4.1). For a task with $n = 50$ entities, this means the agent must use at most $n / \beta = 50 / 0.1 = 500$ Visit actions — an extremely permissive bound that filters only catastrophically inefficient trajectories. At $n = 100$ entities and $\beta = 0.1$, the agent could use up to 1000 Visit actions and still pass. The low $\beta$ value suggests that the ISE filter primarily serves to exclude trajectories that get stuck in loops or wander through completely irrelevant pages, rather than to enforce tight efficiency constraints.

Why both ISR and ISE together. The combined filter $\text{ISR} > 0.3$ AND $\text{ISE} > 0.1$ selects trajectories that are simultaneously reasonably complete (at least 30% coverage) and not pathologically inefficient. Trajectories that achieve high coverage only by taking thousands of steps, or that are fast but only cover 5% of required entities, are both excluded. The paper's ablation (Section 4.4, Figure 4) shows that ISR+ISE outperforms ISR-Only and ISE-Only on GAIA (69.9 vs. 67.0 and 68.0) and BrowseComp (27.5 vs. 22.7 and 22.2), suggesting the two criteria provide complementary signals — coverage constraints prevent the model from learning to stop early, while efficiency constraints prevent it from learning to search endlessly.

The Proposition 1 connection. Throughout Section 3.2, the paper implicitly relies on Proposition 1 (formally proved in Appendix A.2): $\text{Var}(\text{ISE}) = O(1/n)$. This proposition states that the variance of ISE is inversely proportional to the number of target entities $n$. When $n = 3$ (typical for prior work), ISE is a noisy, unreliable metric — two trajectories that are genuinely equally efficient might have very different ISE scores due to random variation in which step each of the 3 entities happened to be discovered. When $n = 100$ (WebLeaper's typical range), ISE is a stable measure — the random variation averages out, and trajectories with genuinely different efficiency reliably have different ISE scores. This is why entity-intensive task construction is a prerequisite for efficiency-based trajectory filtering: you cannot reliably measure what you cannot reliably filter on.

The resulting training dataset. After filtering, the retained trajectories form a supervised fine-tuning dataset. Each trajectory is a sequence of thought-action-observation triples with the implicit label that this sequence represents an efficient, reasonably complete solution. The base model is trained on these trajectories using standard next-token prediction, learning to produce the thought, action, and final answer tokens that correspond to efficient search behavior. The paper does not specify the exact number of trajectories retained or the filtering yield (fraction of generated trajectories that pass both criteria).

Reinforcement Learning with Hybrid Reward Systems

While SFT on filtered trajectories teaches the model to imitate efficient search patterns, RL further optimizes the policy by directly rewarding entity retrieval quality. The challenge is that standard RL reward functions fail on entity-intensive tasks for two reasons: binary success/failure rewards are too sparse (the agent almost never gets all entities right during early training), and exact-match verification of entity lists is too brittle (penalizing semantically equivalent variations like "USA" vs. "United States").

The granular F-score for WebLeaper tasks. For the approximately 500 entity-intensive QA pairs reserved for RL (distinct from the SFT data), WebLeaper defines a reward based on soft precision and recall over entities. The core innovation is the semantic scoring function $s(e_o, e_r) \in [0, 1]$ that measures the similarity between a retrieved entity $e_o \in O$ (from the agent's output) and a ground-truth entity $e_r \in R$ (from the task definition).

Type-specific evaluation modalities. Rather than using a single similarity function for all entities, the paper categorizes entities in the ground-truth set $R$ by semantic type (person names, dates, organizations, etc.) and assigns an appropriate evaluation modality to each category. For example:

  • Person names might be evaluated using near-exact match — allowing minor variations like middle initial presence/absence or diacritic normalization, but requiring the core name components to match. This handles "William Golding" vs. "W. Golding" without accepting completely different people.

  • Dates might use normalized date comparison — converting various date formats (ISO, natural language, abbreviated) to a canonical form before comparison.

  • More abstract concepts might require a targeted LLM-as-a-Judge assessment — an LLM call specifically prompted to evaluate whether two strings refer to the same entity in context (e.g., "United Kingdom" vs. "Britain" vs. "UK"). The paper notes this is done "at the individual entity level" and organized by type to keep costs manageable compared to a monolithic judge that evaluates an entire answer list.

The type-specific design balances accuracy (different entity types need different comparison logic) with computational efficiency (expensive LLM-as-judge is only used for entity types where simpler string matching fails).

Soft recall computation. Using the scoring function $s$, soft recall generalizes ISR to account for near-matches:

Rc=1∣R∣∑er∈Rmax⁡eo∈Os(eo,er)R_c = \frac{1}{|R|} \sum_{e_r \in R} \max_{e_o \in O} s(e_o, e_r)

where $|R|$ is the number of ground-truth entities, $e_r$ iterates over each ground-truth entity, $e_o$ iterates over each agent-retrieved entity in $O$, $s(e_o, e_r)$ is the type-specific semantic similarity score, and $\max_{e_o \in O} s(e_o, e_r)$ finds the retrieved entity most similar to each ground-truth entity.

What it computes: for each required entity in the ground truth, find the retrieved entity that best matches it (according to the semantic scorer), take that match score, and average across all required entities. If every ground-truth entity has a near-perfect match in the retrieved set, $R_c \approx 1.0$. If some are missing, $R_c$ is lower. Importantly, multiple retrieved entities matching the same ground-truth entity only help to the extent that the best match score improves — the $\max$ operation prevents double-counting.

Why this form: soft recall addresses the brittleness of exact-match recall. With exact matching, "United States" and "USA" would be scored as 0.0 match, penalizing the agent for finding the correct entity expressed differently. The semantic scorer assigns partial credit — $s(\text{"USA"}, \text{"United States"}) \approx 0.95$ perhaps — so the agent is rewarded for close-but-not-exact matches. This is essential for RL because early training produces many such near-misses, and without partial credit the reward signal would be too sparse for effective learning.

Soft precision computation. Soft precision penalizes the agent for retrieving irrelevant entities:

P=1∣O∣∑eo∈Omax⁡er∈Rs(eo,er)P = \frac{1}{|O|} \sum_{e_o \in O} \max_{e_r \in R} s(e_o, e_r)

where $|O|$ is the number of agent-retrieved entities, and the inner $\max$ finds the best-matching ground-truth entity for each retrieved entity.

What it computes: for each entity the agent retrieved, find how well it matches some required entity (best case), and average across all retrieved entities. If the agent retrieves many entities that have no match in the ground truth (random noise, unrelated facts), $P$ is penalized because those entities have $\max_{e_r} s \approx 0$, dragging down the average.

Why this form: without a precision penalty, the agent could achieve high recall by retrieving everything — flooding its output with hundreds of entities, many irrelevant, knowing that the recall-maximizing $\max$ operation will find matches for the required ones somewhere in the pile. The precision term ensures that extraneous retrievals hurt the reward, incentivizing focused search that retrieves mostly relevant entities.

The WebLeaper reward (weighted F-score). Precision and recall are aggregated into a single reward using a weighted F-score:

RWebLeaper=(1+ω2)P⋅Rcω2P+RcR_{\text{WebLeaper}} = (1 + \omega^2) \frac{P \cdot R_c}{\omega^2 P + R_c}

where $\omega$ is a hyperparameter balancing the importance of precision versus recall. $\omega > 1$ prioritizes recall (aligning with the original ISR objective of thorough coverage), while $\omega < 1$ prioritizes precision (avoiding irrelevant retrievals even at the cost of missing some required entities). The paper does not specify the value of $\omega$ used in experiments.

What it computes: the standard weighted F-score formula, which is the harmonic mean (with weighting) of precision and recall. When $\omega = 1$, this is the standard (unweighted) F1 score. The harmonic mean has the property that both precision and recall must be reasonably high for the F-score to be high — an agent cannot compensate for near-zero recall with perfect precision, or vice versa.

Why this form: the harmonic mean penalizes imbalance between precision and recall more severely than the arithmetic mean. An agent with $P = 0.9$ and $R_c = 0.1$ (high precision but poor coverage) would have an arithmetic mean of 0.5 but a harmonic mean (F1) of approximately 0.18 — a much stronger penalty. This is desirable because information-seeking tasks genuinely require both: finding most of the required entities (recall) without flooding the output with garbage (precision). The weighting parameter $\omega$ allows tuning toward recall-dominant tasks (where missing entities is worse than retrieving extras) or precision-dominant tasks (where extraneous information is costly).

Why not a single end-to-end LLM judge? The paper explicitly considers and rejects this alternative (Section 3.3): "asking a judge model to accurately verify a long list of entities in a single assessment imposes a high cognitive load, leading to inconsistent scores, while running it for every single entity is prohibitively expensive for RL." The entity-level, type-specific scoring approach resolves this tension by using cheap methods (exact/near-exact match) for most entities and reserving expensive LLM calls for the minority of entities where string matching fails. This makes the reward affordable at RL scale (many trajectories per update, many updates per training run).

Hybrid reward integration. The final reward function for a trajectory $H_T$ on task $T$ is conditional on the task's origin:

Rhybrid(HT,T)={RWebLeaper(O,R)if T is from WebLeaperRlegacy(O,R)otherwiseR_{\text{hybrid}}(H_T, T) = \begin{cases} R_{\text{WebLeaper}}(O, R) & \text{if } T \text{ is from WebLeaper} \\ R_{\text{legacy}}(O, R) & \text{otherwise} \end{cases}

where $R_{\text{legacy}}$ is the original reward function for legacy benchmark training data (typically binary success/failure, since legacy tasks have smaller, exact-matchable answer sets). This conditional structure means the agent receives granular entity-level feedback on the entity-intensive WebLeaper tasks while maintaining compatibility with standard evaluation protocols on existing benchmarks.

What it computes: at RL training time, for each trajectory in a batch, check which dataset the task came from. If it is a WebLeaper-synthesized task, compute the entity-level soft F-score reward. If it is a legacy task (from WebSailor-V2, GAIA training data, etc.), use whatever reward function that benchmark or prior work defined. The agent's policy is updated based on this mixed signal.

Why this form: this is a pragmatic compatibility design. Training only on WebLeaper tasks with the granular reward would optimize the agent for entity-intensive retrieval but might cause catastrophic forgetting of capabilities needed for standard benchmarks (where questions may have short, factoid answers). Training on the legacy data with the legacy reward preserves those capabilities. The hybrid reward enables WebLeaper to augment rather than replace existing training pipelines, which also makes the ablation studies cleaner — differences from baselines reflect the WebLeaper data contribution rather than differences in the base training recipe.

Policy optimization with GRPO. The agent's policy $\pi_\theta$ is optimized using Group Relative Policy Optimization (GRPO; Shao et al., 2024), a variant of PPO that replaces the learned value function with a group-relative baseline. For each task $T$ in the RL dataset, the current policy samples a group of $k$ trajectories $\{H_1, \dots, H_k\}$ (the paper does not specify $k$, but typical GRPO implementations use 4–16). Each trajectory receives a reward $R_i = R_{\text{hybrid}}(H_i, T)$.

The advantage for each trajectory is computed by standardizing its reward relative to the group:

A^i=Ri−mean({Rj}j=1k)std({Rj}j=1k)+ϵstd\hat{A}_i = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^k)}{\text{std}(\{R_j\}_{j=1}^k) + \epsilon_{\text{std}}}

where $\epsilon_{\text{std}}$ is a small constant for numerical stability (preventing division by zero if all trajectories in a group have identical rewards).

What it computes: subtract the group mean reward from each trajectory's reward, then divide by the group standard deviation. This produces advantages centered at zero with unit variance within each group. A trajectory with above-average reward in its group gets a positive advantage; a trajectory with below-average reward gets a negative advantage.

Why this form (relative to PPO with a value function): the group-relative baseline serves the same purpose as the value function in standard PPO — reducing variance of the policy gradient estimate by removing the expected reward level — but without the computational cost and instability of training a separate value network. GRPO leverages the fact that for a given task, multiple trajectories sampled from the same policy naturally form a reference distribution. The mean reward in the group estimates the expected reward under the current policy for that task, and the standardization converts rewards to relative advantages. This is computationally efficient (no value network training) and conceptually clean for the information-seeking domain where reward scales can vary dramatically across tasks (a task with 10 required entities vs. 200).

The GRPO loss. The advantage $\hat{A}_i$ from Equation (11) is applied uniformly to every timestep within trajectory $H_i$. The policy is then updated by minimizing a clipped surrogate objective:

LGRPO(θ)=−E{Hi}i=1k∼πθ[1k∑i=1k1∣Hi∣∑t=1∣Hi∣min⁡(ri,t(θ)A^i,clip(ri,t(θ),1−ε,1+ε)A^i)]\mathcal{L}_{\text{GRPO}}(\theta) = -\mathbb{E}_{\{H_i\}_{i=1}^k \sim \pi_\theta} \left[ \frac{1}{k} \sum_{i=1}^{k} \frac{1}{|H_i|} \sum_{t=1}^{|H_i|} \min\left( r_{i,t}(\theta) \hat{A}_i, \text{clip}(r_{i,t}(\theta), 1 - \varepsilon, 1 + \varepsilon) \hat{A}_i \right) \right]

where $r_{i,t}(\theta) = \frac{\pi_\theta(a_{i,t} | s_{i,t})}{\pi_{\text{old}}(a_{i,t} | s_{i,t})}$ is the importance sampling ratio — the probability of the action taken at timestep $t$ under the current policy divided by its probability under the policy that generated the trajectory, $\varepsilon$ is the clipping hyperparameter (typically 0.1–0.2, though the paper does not specify the exact value used), and the outer expectation is over groups of $k$ trajectories sampled from the current policy.

What it computes: this is the standard PPO clipped objective, summed over all timesteps of all trajectories in a group. For each action at each timestep, we compute how much more (or less) likely the current policy is to take that action compared to the old policy that generated the data ($r_{i,t}$). If the trajectory had a positive advantage ($\hat{A}_i > 0$), we want to increase the probability of those actions — but the clip prevents increasing $r_{i,t}$ beyond $1 + \varepsilon$, limiting how aggressively the policy can change in a single update. If $\hat{A}_i < 0$, we want to decrease the probability, clipped at $1 - \varepsilon$. The $\min$ operation ensures that when the advantage and ratio changes have opposite signs, we use the clipped (more conservative) objective. The result is a scalar loss that is minimized via gradient descent on $\theta$.

Why this form: the clipping mechanism is the key PPO innovation that enables stable RL without the line search or trust region constraints of earlier methods like TRPO. By preventing the policy from changing too much in a single update (as measured by the per-action probability ratio), clipping avoids catastrophic updates that could destroy the policy. The group-relative advantage ($\hat{A}_i$) replaces the learned value function advantage, making the whole pipeline simpler and more scalable. For information-seeking tasks, this stability is particularly important because early RL training produces highly variable trajectories (some accidentally find many entities, most find few), and without clipping the policy would oscillate wildly between overfitting to lucky trajectories and forgetting useful behaviors.

Training dynamics. The paper reports training for 135 steps (Figure 6) before terminating "when web access resources were exhausted" — this is a practical limitation of RL for web agents, since each trajectory requires real web API calls (Google Search, page visits) that are rate-limited and potentially costly. The reward curve shows a "stable and continuous upward trajectory," indicating that the hybrid reward provides a sufficiently dense signal for the policy to improve progressively. The RL results (Table 3) show consistent improvements over SFT: +1.0 on BrowseComp, +3.3 on GAIA, +3.0 on xbench-DS, and substantial gains on WideSearch (SR +2.5, Row F1 +8.0, Item F1 +3.1), demonstrating that the entity-level reward signal successfully guides the policy toward more efficient and accurate search behavior beyond what SFT alone achieves.

Summary of Design Choices and Their Justifications

  • Tree-structured reasoning formulation over flat entity lists: compactly represents dozens to hundreds of target entities (all nodes in a subtree) within a single hierarchical structure, enabling both high entity density and explicit reasoning paths that the agent can learn to follow.

  • Wikipedia tables as the raw data source over free text or knowledge graphs: tables naturally encode entity-relation triples with structural consistency (same columns = same relations for all rows), making them ideal for automated tree population without manual annotation. The multi-stage cleaning (size, semantic, structural, isomorphism) ensures quality while the ~2 million table crawl provides scale.

  • Maximal biclique enumeration for Union discovery over manual or heuristic combination: converts the combinatorial search over tree pairs into a well-studied graph algorithm problem with efficient solvers, ensuring that discovered unions are semantically coherent (shared relations) and maximal (no redundant subgroups). The semantic filtering (type and domain matching on Layer-2 entities) prevents incoherent unions from tables with superficially similar column names but incompatible entity types.

  • Deductive fuzz in Reverse-Union over directly naming all sources: forces the agent to perform property-to-entity inference before the main search, blocking the keyword-search shortcut that undermines Union tasks and ensuring the agent practices genuine multi-step reasoning rather than pattern matching.

  • ISR threshold $\alpha = 0.3$ over higher thresholds (e.g., 0.8): deliberately permissive to retain trajectories with some useful search behavior even if incomplete, providing more training data and more diverse examples of "on the right track" search strategies. A higher threshold would retain only near-perfect trajectories, reducing dataset size and potentially filtering out informative partial successes.

  • ISE threshold $\beta = 0.1$ over stricter thresholds: filters only catastrophically inefficient trajectories (loops, random walks) while retaining a range of efficiency levels, providing contrastive examples for the model to learn from. A stricter threshold would retain only the most efficient trajectories, removing contrast and potentially overfitting to a narrow efficiency regime.

  • ISE computed only on Visit actions over including Search actions: avoids inflating efficiency scores from snippet-level entity mentions that don't represent genuine information discovery. The Search → Visit pipeline is naturally two-stage (discover URLs, then read them), and only Visit confirms actual entity extraction. This design choice reflects an understanding of how web search works in practice.

  • Entity-level F-score with type-specific semantic scoring over monolithic LLM judge or exact match: balances accuracy (type-specific comparisons handle semantic variation gracefully) with computational cost (most entities use cheap string matching, only ambiguous types use LLM calls). The soft scoring handles the partial-credit problem that makes binary rewards too sparse for entity-intensive RL.

  • GRPO with group-relative baseline over PPO with learned value function: eliminates the need to train and tune a separate value network while still providing variance reduction through group-relative standardization. This is particularly appropriate for the web agent domain where the state space (web pages, search results, accumulated context) is high-dimensional and training a value function would be expensive and unstable.

  • Hybrid reward ($R_{\text{WebLeaper}}$ + $R_{\text{legacy}}$) over WebLeaper-only reward: preserves compatibility with existing benchmarks and training data, ensuring that the entity-intensive optimization does not catastrophically forget standard QA capabilities. This is both a practical necessity (fair comparison with baselines) and a design principle (augment, don't replace).

  • Basic → Union → Reverse-Union curriculum over training on all variants mixed: the paper does not explicitly state that training follows a curriculum order, but the ablation results (Table 2) show that training on Basic alone degrades performance (shortcut overfitting), Union improves, and Reverse-Union improves further. This suggests that progressively increasing task complexity is more effective than training on the hardest variant directly, though the paper does not ablate curriculum ordering versus mixed training.

  • Two-tool design (Search + Visit) over richer tool sets: mirrors realistic web search interfaces where the primary actions are query formulation and page reading. Adding more tools (click, scroll, form interaction) would increase task complexity but also increase trajectory length and tool-selection difficulty. The minimalist tool set keeps the focus on search strategy (what to search for, what to read) rather than browser automation.

4. Key Insights and Innovations

Innovation 1: Entity Density as a Training Data Design Dimension — Moving from "Harder Tasks" to "Richer Signals"

The paper's most fundamental conceptual contribution is reframing what makes training data effective for information-seeking agents. Before WebLeaper, the dominant paradigm for improving IS agent training was increasing structural complexity: multi-step reasoning chains (WebDancer, WebResearcher), long-horizon planning horizons (ASearcher), or more sophisticated tool-use orchestration (WebShaper). These approaches asked "how can we make the task harder?" — layering additional reasoning steps, longer action sequences, or more tools.

WebLeaper asks a fundamentally different question: "How many pieces of useful information can we pack into each training example?" The key variable is not the number of reasoning steps or the planning depth, but the number of target entities n — the distinct pieces of information the agent must locate and retrieve. The paper's tree-structured reasoning formulation is not merely a convenient data structure; it is the mechanism that enables packing 50–150+ target entities into a single training task, compared to the 3–5 entities typical in prior work (WebSailor, WebDancer). This is a shift from task complexity (harder questions) to data efficiency (more supervision signal per question).

The significance of this reframing is both practical and theoretical. Practically, it means that a modest number of WebLeaper tasks (5,000–10,000) can provide more training signal than far larger datasets of entity-sparse tasks, because each WebLeaper trajectory carries granular feedback on dozens of retrieval decisions rather than binary feedback on a single answer. This is evident in Table 2: 5,000 Union tasks added to the WebSailor-V2-5k baseline produce an average +3.26 improvement across five benchmarks, despite increasing the dataset size by only 5,000 examples. The gain per added example is disproportionately high because each example is semantically denser.

Theoretically, the contribution is the formalization of why entity density matters, via Proposition 1 in Section 2.3: Var(ISE) = O(1/n). This result is more than an appendix curiosity. It establishes that efficiency metrics are unreliable when target entities are sparse — you cannot reliably measure search efficiency on tasks with 3–5 target entities because random variance in which step each entity is discovered dominates the ISE score. This provides a theoretical explanation for why prior work struggled to improve search efficiency: they were trying to optimize a metric that was fundamentally noisy at their task scales. By increasing n to 50–150+, WebLeaper drives ISE variance down proportionally, making efficiency a stable, optimizable quantity. This insight generalizes beyond information-seeking: any training methodology that optimizes per-step efficiency on multi-output tasks requires sufficient output density for the efficiency signal to be reliable.

The paper does not merely claim that entity density helps — it demonstrates the boundaries through negative results. The Basic variant (Table 2, -7.64 average vs. WebSailor-V2-5k) shows that entity density alone is insufficient when tasks are structurally too simple. Basic tasks pack many entities but from a single source with a predictable hierarchical pattern; the model learns a shortcut (find the one table, extract all rows) rather than genuine information-seeking strategy. This negative result distinguishes the contribution from a simplistic "more entities = better" claim: entity density must be combined with structural complexity (multi-source integration in Union, deductive reasoning in Reverse-Union) for the richer signal to translate into improved capability. This is reminiscent of the finding in curriculum learning that increasing difficulty along only one dimension (e.g., more entities) can lead to degenerate solutions if other dimensions (structural variety) are not also scaled.

Innovation 2: The Efficiency-Coverage Co-Filter as a Training Objective — Selecting for "How" Not Just "What"

Prior agent training pipelines uniformly treated any trajectory that reached the correct answer as equally valuable for supervised fine-tuning. Whether the agent found the answer in 5 efficient steps or 100 meandering steps — if the final output was correct, the trajectory was included. This is the standard approach in behavioral cloning: imitate all successful trajectories, regardless of their efficiency.

WebLeaper introduces a fundamentally different training objective: the model should learn not just from what answer a trajectory produced, but from how efficiently it produced it. The dual filter — ISR > 0.3 (coverage) and ISE > 0.1 (efficiency) — selects trajectories based on the process quality rather than merely the outcome correctness. This is not a minor filtering tweak; it changes the nature of the supervision signal. In standard behavioral cloning, the model learns "when in this state, any action that eventually leads to success is acceptable" — which includes inefficient actions, redundant searches, and unnecessary page visits. With the co-filter, the model learns "when in this state, only actions that maintain both coverage and efficiency are acceptable."

The innovation is in recognizing that for information-seeking tasks — where the action space is vast (all possible search queries and page visits) and the reward is delayed (entities accumulate over many steps) — trajectory quality varies enormously even among successful outcomes, and this variance is a learning signal that prior work discarded. The ablation in Figure 4 supports this: ISR+ISE outperforms ISR-Only and ISE-Only on GAIA (69.9 vs. 67.0 and 68.0) and BrowseComp (27.5 vs. 22.7 and 22.2), demonstrating that the two criteria provide complementary signals. ISR-only filtering retains trajectories that are thorough but potentially inefficient; ISE-only filtering retains trajectories that are fast but potentially incomplete. The combination selects for trajectories that balance both — and this balanced signal produces a measurably better policy.

The significance extends beyond the specific thresholds (α = 0.3, β = 0.1). These values are deliberately permissive, filtering only trajectories that are catastrophically incomplete or pathologically inefficient. The fact that even coarse filtering on process quality produces substantial gains (Table 2, Union +3.26 average) suggests that the raw distribution of trajectories from standard models contains a long tail of low-quality-but-successful trajectories that dilute the training signal. The co-filter removes this tail, and the remaining trajectories — even if not perfectly efficient — provide a cleaner signal for the model to learn from. This is a general insight for imitation learning from noisy demonstrations: when trajectories vary in quality along a dimension orthogonal to binary success, filtering on that dimension can yield outsize improvements even with conservative thresholds.

A subtle but important design choice reinforces the process-over-outcome philosophy: ISR measures coverage (did the required entities appear in the agent's observations at any point?) rather than extraction correctness (did the agent correctly report them as final answers?). This means the filter selects trajectories where the search process was on the right track, regardless of whether the final answer assembly succeeded. The SFT training then teaches the model to both search efficiently (from the filtered trajectories' action sequences) and extract correctly (from the observation-to-answer mapping). This decoupling of search quality from answer extraction is a deliberate architectural choice that prevents the filter from being circular — it doesn't require the model to already be good at extraction to produce training data for improving search.

Innovation 3: Deductive Fuzz as a Shortcut-Breaking Mechanism — Forcing Genuine Reasoning Through Information Asymmetry

A recurring challenge in training agents on synthetic tasks is that models learn to exploit surface patterns and keyword shortcuts rather than developing genuine reasoning capabilities. The paper explicitly identifies this failure mode in Section 3.1.3: an agent trained on Union tasks (e.g., "authors who won both the Nobel Prize and the Booker Prize") can succeed by searching for "Nobel Prize winners" and "Booker Prize winners" independently and pattern-matching — a strategy that requires no understanding of why these prizes are related or what it means for an author to win both. This is a form of reward hacking on the training distribution — the model achieves high accuracy on Union tasks by learning a shallow heuristic rather than the intended multi-source information synthesis.

The innovation in Reverse-Union is not merely making tasks harder; it is designing tasks that are structurally impossible to solve through the shallow heuristic that works on simpler variants. The deductive fuzz removes the most direct search key (the name of the prize or source table) and replaces it with descriptive clues that require property-to-entity inference. An agent cannot search "1980s prize-winner wrote novel about stranded British boys" as a keyword query and directly retrieve "William Golding" — it must decompose this into sub-queries, synthesize partial results, and perform abductive reasoning to identify the anchor entity. The paper calls this "Deductive Fuzz" but the deeper concept is information asymmetry: the task provides sufficient information to identify the anchor (if you reason correctly) but withholds the most direct search key, forcing the agent to bridge the gap through multi-step inference.

This is fundamentally different from prior approaches to preventing shortcuts in synthetic training data. One common approach is adversarial filtering — generate tasks, test whether the model can solve them with simple heuristics, and discard those it can. Reverse-Union takes a constructive approach: design tasks where the shortcut is structurally impossible because the necessary search key is never provided in searchable form. This is a stronger guarantee — adversarial filtering can only catch shortcuts that the current model exhibits, while structural impossibility prevents any model from exploiting the shortcut regardless of its capabilities.

The results validate that this design choice produces genuinely different capabilities. In Table 2, Reverse-Union-10k† achieves the highest average improvement (+4.34 over WebSailor-V2-5k), and the gains are concentrated on benchmarks that demand reasoning: xbench-DS (+6.00) and WideSearch (+10.92 — the largest single-benchmark gain in the table). These benchmarks, unlike GAIA which primarily tests fact-finding, explicitly require the kind of multi-step deduction and broad synthesis that Reverse-Union's forced reasoning path teaches. The contrast with Union-5k† (+3.26 average, +2.33 on xbench-DS, +8.55 on WideSearch) shows that adding the deductive fuzz contributes roughly +1 additional average gain and disproportionately helps on reasoning-heavy benchmarks — exactly what the design intended.

This innovation connects to a broader challenge in AI training: the simplicity bias of neural networks (they prefer to learn the simplest function that fits the training data) and how training data design can counteract rather than reinforce this bias. By structurally eliminating the simple solution, Reverse-Union forces the model up the complexity ladder. This is a training data design principle — not "make tasks harder" but "make the easy solution impossible" — that generalizes beyond information-seeking to any domain where synthetic data risks teaching shallow heuristics.

Innovation 4: The Hybrid Reward as a Compatibility Layer — Granular Optimization Without Catastrophic Forgetting

Multi-task RL for agents faces a fundamental tension: specialized rewards improve performance on specific capabilities but risk degrading general capabilities through catastrophic forgetting. The standard solutions — reward shaping, multi-task loss weighting, experience replay — add complexity and hyperparameters. WebLeaper's approach is elegantly minimal: conditional reward computation based on the data source's origin, with no changes to the RL algorithm, no auxiliary losses, and no replay buffer management.

The hybrid reward (Equation 10) is R_hybrid = R_WebLeaper for WebLeaper-synthesized tasks and R_legacy for legacy benchmark training data. This single conditional makes the entire training pipeline compatible with existing agent RL frameworks — GRPO operates exactly as it would on any mixed dataset, with the reward function being the only WebLeaper-specific component. The innovation is not the hybrid reward itself (conditional rewards are common in multi-task RL) but the recognition that entity-intensive tasks require a qualitatively different reward structure (granular F-score vs. binary success) and that this difference can be cleanly isolated to the reward function without modifying the policy architecture, the RL algorithm, or the legacy data processing pipeline.

The practical significance is substantial: WebLeaper can be integrated into any existing IS agent training pipeline by adding its synthesized data to the training mixture and inserting the conditional reward computation — no other changes required. This is why the paper's comprehensive setting experiments (Figure 1, Table 3) can directly compare WebLeaper-augmented training against strong baselines: the only difference is the presence of WebLeaper data in the training corpus, making the ablation clean and the contribution unambiguous.

But the deeper insight is about reward design for compositional capabilities. Information-seeking requires a composition of skills: formulating queries, selecting pages to visit, extracting relevant information, synthesizing across sources, and verifying completeness. Binary success rewards provide a single scalar signal for the entire composition, which is extremely sparse for complex tasks. The entity-level F-score decomposes the reward into per-entity contributions, providing dense feedback that credits individual retrieval decisions even when the overall task is incomplete. This decomposition is only possible because WebLeaper's task synthesis produces tasks with known, enumerable target entity sets — a property that legacy benchmarks (GAIA, BrowseComp) do not share. The hybrid reward is thus not just a compatibility hack; it is a way to provide dense, compositional feedback on tasks where the structure is known (synthesized data) while maintaining sparse, holistic feedback on tasks where the structure is unknown (benchmark data).

The RL results in Table 3 confirm that this hybrid approach avoids catastrophic forgetting while enabling specialized improvement. SFT+RL improves over SFT-only on every benchmark: +1.0 BrowseComp, +3.3 GAIA, +3.0 xbench-DS, and substantial WideSearch gains (SR +2.5, Row F1 +8.0, Item F1 +3.1). The improvements are not concentrated on WebLeaper-like entity-intensive tasks at the expense of standard benchmarks — GAIA and BrowseComp gains are consistent with or larger than the overall trend, indicating that the entity-level training signal generalizes to standard evaluation settings. The stable upward reward curve in Figure 6 further suggests that the hybrid reward provides a well-behaved optimization landscape without the instability that often plagues multi-objective RL.

This innovation should be understood as a pragmatic contribution to the engineering of agent training pipelines — solving the integration problem that arises whenever a specialized training technique must coexist with general-purpose benchmarks — rather than a theoretical advance in RL. Its significance is in enabling the adoption of entity-intensive training without requiring practitioners to abandon their existing training infrastructure.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. Five benchmarks, evaluated on their standard test/validation splits. GAIA uses the 103-sample text-only validation subset; all other benchmarks (BrowseComp, xbench-DeepSearch, Seal-0, WideSearch) use the complete test sets. Originating works: GAIA (Mialon et al., 2023), BrowseComp (Wei et al., 2025), xbench-DeepSearch (Xbench-Team, 2025), Seal-0 (Pham et al., 2025), WideSearch (Wong et al., 2025).

  • Base model. Qwen3-30B-A3B-Thinking-2507, a ~30B-parameter open-source model. Chosen as a representative mid-scale model that leaves headroom for improvement via training (unlike near-saturated proprietary models) while being competitive enough for meaningful baseline comparison. All training uses the Megatron framework.

  • Metrics. For BrowseComp, GAIA, xbench-DS, and Seal-0: Pass@1 accuracy, scored via LLM-as-a-judge evaluation following each benchmark's specified protocol. For WideSearch: Success Rate (SR, fully retrieving all target results), Row F1, and Item F1 — computed using a combination of string matching and LLM-as-a-judge evaluation per the official protocol.

  • Baselines. Proprietary agents: Claude-4-Sonnet (Anthropic, 2025), OpenAI-o3 (OpenAI, 2025a), OpenAI DeepResearch (OpenAI, 2025b). Open-source agents: ASearcher-Web-32B (Gao et al., 2025), DeepDive-32B (Lu et al., 2025), DeepDiver-V2-38B (Team, 2025b), MiroThinker-32B-DPO-v0.2 (Team et al., 2025b), Kimi-K2-Instruct-1T (Team et al., 2025a), WebExplorer-8B (Liu et al., 2025), WebDancer-QwQ-32B (Wu et al., 2025a), WebSailor-32B (Li et al., 2025c), WebShaper-QwQ-32B (Tao et al., 2025).

  • Generation budget / compute accounting. The paper does not use a FLOPs-equivalent compute budget across methods — comparison is at the level of final benchmark accuracy, not compute-matched. Inference hyperparameters are fixed at temperature 0.6 and top-p 0.95 across all WebLeaper evaluations. Training compute is reported indirectly via the number of training samples (5,000 WebSailor-V2 base + 5,000 or 10,000 WebLeaper variants) and RL training steps (135 steps for the comprehensive setting).

  • Cross-validation / statistical protocol. WebLeaper scores in Table 1 and Figure 1 are averaged over three runs with different random seeds. Baseline scores are typically single-point estimates from either the original papers or single evaluations; the paper does not report standard deviations or confidence intervals for WebLeaper results, nor does it report statistical significance tests between WebLeaper and baselines.

Main Quantitative Results

Base Setting: WebLeaper vs. Open-Source Baselines (Table 1)

WebLeaper achieves state-of-the-art performance among open-source agents on all five benchmarks. On GAIA, WebLeaper-Reverse-Union B reaches 67.0% and WebLeaper-Union B reaches 69.9% — both exceeding Claude-4-Sonnet at 68.3% and the best open-source baseline (MiroThinker-32B-DPO-v0.2 at 64.1%) by 3.0–5.8 percentage points. On BrowseComp, WebLeaper-Reverse-Union B (23.0%) and WebLeaper-Union B (22.1%) substantially exceed the best open-source baseline (WebExplorer-8B at 15.7%) by 6.4–7.3 points — a >40% relative improvement — while also surpassing Claude-4-Sonnet at 12.2%.

On xbench-DeepSearch (xbench-DS), WebLeaper-Reverse-Union B achieves 66.0%, compared to the best open-source baseline (WebSailor-32B at 53.3%) — a 12.7-point absolute gain. On Seal-0, WebLeaper-Reverse-Union B reaches 37.2%, versus WebSailor-32B at 21.3% (+15.9 points). On WideSearch, WebLeaper-Reverse-Union B achieves Success Rate 4.0% (vs. Claude-4-Sonnet at 2.3% and the best open-source Kimi-K2-Instruct-1T at 1.1%), Row F1 25.8 (vs. 34.0 for OpenAI-o3, though o3 is proprietary), and Item F1 40.8 (vs. 57.9 for Claude-4-Sonnet). Note that WebLeaper trails proprietary models on WideSearch Item F1, but leads open-source agents on all WideSearch metrics.

The comparison between WebLeaper variants is also informative: Reverse-Union B generally outperforms Union B on BrowseComp (23.0 vs. 22.1), xbench-DS (66.0 vs. 62.3), and Seal-0 (37.2 vs. 35.1), but slightly underperforms on GAIA (67.0 vs. 69.9). This suggests Reverse-Union's deductive reasoning training helps most on benchmarks emphasizing multi-step planning and reasoning (BrowseComp, xbench-DS, Seal-0), while Union's clean multi-source integration may better serve fact-finding tasks like GAIA.

Notably, WebLeaper-Reverse-Union B uses only 10K WebLeaper samples added to 5K WebSailor-V2 base data and a 30B-parameter model, yet surpasses Kimi-K2-Instruct-1T — a much larger model — on BrowseComp (23.0% vs. 14.1%), GAIA (67.0% vs. 57.7%), and xbench-DS (66.0% vs. 50.0%). This is the key efficiency-of-training-data signal: entity-rich data compensates for model scale.

Comprehensive Setting: WebLeaper vs. All Competitors (Figure 1, Table 3)

Under the comprehensive training setting — where WebLeaper data is mixed into the Tongyi-DeepResearch-30B-A3B training corpus for both SFT and RL stages — WebLeaper achieves:

  • GAIA: 73.2% (averaged over 3 runs), exceeding Claude-4-Sonnet (68.3%), OpenAI DeepResearch (67.4%), and OpenAI-o4-mini (28.3% — though note this is o4-mini, not o3).
  • BrowseComp: 38.8%, exceeding Claude-4-Sonnet (12.2%), OpenAI-o4-mini (28.3%), and all open-source agents listed (the highest open-source baseline in Table 1 was WebExplorer-8B at 15.7%).
  • xbench-DeepResearch: 72.0%, slightly exceeding DeepSeek-V3.1 (71.0%) and GLM-4.5 (70.0%).
  • WideSearch: SR 3.0% (vs. Claude-4-Sonnet 2.3%, DeepSeek-R1 0.4%), Item-F1 48.5% (vs. Claude-4-Sonnet 57.9% — trailing here), and presumably Row F1 and Item F1 as reported in Figure 1's bar chart format (the exact Row F1 value is only partially visible in Figure 1 but appears around 31.0 based on Table 3's SFT+RL results).

The comprehensive setting results are importantly not directly comparable to the base setting results in Table 1, because the comprehensive setting uses a larger, more diverse training corpus (Tongyi-DeepResearch-30B-A3B) that includes additional proprietary data beyond the WebSailor-V2-5k + WebLeaper mixture. The gain from comprehensive vs. base is therefore partially attributable to WebLeaper data and partially attributable to the larger base training corpus. The paper does not report a comprehensive baseline without WebLeaper data, making it impossible to isolate WebLeaper's contribution in this setting. Table 3 provides the closest ablation: SFT (comprehensive corpus without RL) vs. SFT+RL (comprehensive corpus with WebLeaper-guided RL), showing RL adds +1.0 to +3.3 points across benchmarks. But the SFT baseline already includes WebLeaper data during supervised fine-tuning, so this isolates the RL contribution of WebLeaper's reward design, not the contribution of WebLeaper data overall.

Capability Gains from Entity-Intensive Task Synthesis Variants (Table 2)

Table 2 ablates the three WebLeaper task variants against training solely on WebSailor-V2 data (5K and 10K samples). All variants are mixed with the WebSailor-V2-5k base (noted with †). Results across all five benchmarks, reported as average across all benchmarks in the final column:

  • WebSailor-V2-5k (baseline): 44.05
  • WebSailor-V2-10k (doubled baseline data): 45.21 (+1.16)
  • Basic-5k†: 36.41 (−7.64) — substantially worse than baseline, consistent with the paper's claim that Basic tasks are too simple and induce shortcut learning
  • Union-5k†: 47.31 (+3.26) — consistent improvement across all benchmarks; the gain over adding 5K more WebSailor data (45.21) is +2.10, meaning Union data is more valuable per sample than additional baseline data
  • Reverse-Union-10k†: 48.39 (+4.34) — the strongest variant; the gain over Union-5k† is +1.08 with double the WebLeaper data (10K vs. 5K), so the per-sample efficiency gain is smaller but the absolute performance is highest

Per-benchmark patterns reveal where each variant's design matters:

  • On BrowseComp, Basic-5k† drops to 20.67 (−4.50 from WebSailor-V2-5k's 25.17), Union-5k† reaches 27.50 (+2.33), Reverse-Union-10k† reaches 27.67 (+2.50). The near-equivalence of Union and Reverse-Union on BrowseComp is interesting — both improve similarly, suggesting BrowseComp's primary challenge is multi-source integration rather than deductive fuzz.
  • On GAIA, Basic-5k† catastrophically drops to 40.78 (−26.91 from baseline's 67.69). This is the paper's strongest evidence of Basic's shortcut problem: GAIA's questions require diverse information sources, and the single-source pattern of Basic actively hurts transfer. Union-5k† recovers to 69.90 (+2.21 above baseline), while Reverse-Union-10k† actually slightly underperforms at 66.99 (−0.70). This is the only benchmark where Reverse-Union does not improve over baseline, suggesting GAIA's evaluation may not reward deductive reasoning as much as it rewards clean multi-source fact aggregation.
  • On xbench-DS, Basic-5k† drops to 58.33 (−1.67), Union-5k† reaches 62.33 (+2.33), Reverse-Union-10k† reaches 66.00 (+6.00 — the second-largest single-benchmark gain). xbench-DS explicitly tests deep research reasoning, aligning with Reverse-Union's deductive + union design.
  • On WideSearch, Reverse-Union-10k† achieves 44.07 (+10.92 over baseline 33.15 — the largest single-benchmark absolute gain in the entire table). Union-5k† achieves 41.70 (+8.55). WideSearch's emphasis on broad, comprehensive retrieval directly benefits from entity-dense training, and the larger gain from Reverse-Union suggests that training for deductive search also improves broad search strategy. Basic-5k† slightly underperforms at 32.26 (−0.89).
  • On Seal-0, the pattern is consistent: Basic-5k† at 30.03 (−4.20), Union-5k† at 35.14 (+0.82), Reverse-Union-10k† at 37.24 (+3.01). The progressive improvement tracks task complexity.

The key takeaway from Table 2 is that entity density alone is not sufficient — Basic has high entity density but degrades performance because of structural simplicity — and that task complexity must co-vary with entity density for the richer training signal to translate into generalizable capability. The paper's three-tier curriculum (Basic → Union → Reverse-Union) is validated by the monotonic improvement in average performance (−7.64 → +3.26 → +4.34) as task complexity increases, though the dependence on data quantity confounds the Union vs. Reverse-Union comparison (5K vs. 10K samples).

Impact of Information-Guided Trajectory Construction (Figure 4)

The ablation in Figure 4 compares three trajectory filtering strategies applied to the Union-5k data: ISR-Only (coverage filter only), ISE-Only (efficiency filter only), and ISR+ISE (both filters). On GAIA, ISR+ISE achieves 69.9 versus ISR-Only at 67.0 and ISE-Only at 68.0, suggesting both criteria contribute complementary signals. On BrowseComp, ISR+ISE reaches 27.5 versus ISR-Only at 22.7 and ISE-Only at 22.2 — the largest gap, indicating BrowseComp's complex browsing tasks are where joint filtering matters most. On WideSearch, all three strategies perform similarly (42.7, 41.8, 41.7), with differences within the margin of variance. This is consistent: WideSearch's broad retrieval tasks benefit from entity-dense training regardless of trajectory filtering quality, because the evaluation metrics (wide coverage) align with the entity-density training objective even without explicit efficiency filtering.

The non-obvious finding is the asymmetry between ISR-Only and ISE-Only: on GAIA (69.9 vs. 68.0) and BrowseComp (27.5 vs. 22.2), the joint filter substantially outperforms either alone, but ISE-Only never outperforms ISR-Only on GAIA and BrowseComp, while on WideSearch they are equal. This suggests that coverage (ISR) provides a more essential training signal than efficiency (ISE) — filtering out incomplete trajectories harms learning more than filtering out inefficient ones — but the combination provides the best of both. The paper does not explore whether different α and β thresholds would change this balance, which is a notable gap given the conservative default thresholds (α = 0.3, β = 0.1).

Efficiency-Effectiveness Trade-off (Figure 5)

Figure 5 plots performance (accuracy or SR/F1) against average action rounds for WebLeaper vs. WebSailor-V2 across four benchmarks. On BrowseComp, WebLeaper achieves ~40% performance at ~54 average action rounds, while WebSailor-V2 achieves ~28% at ~60+ rounds — WebLeaper is simultaneously 12 points more accurate and ~6 steps faster. On WideSearch, WebLeaper achieves ~33% at ~56 rounds versus WebSailor-V2 at ~26% at ~60+ rounds. On GAIA, WebLeaper achieves ~69% at ~20.5 rounds versus WebSailor-V2 at ~62% at ~24 rounds. On xbench-DS, WebLeaper achieves ~70% at ~22 rounds versus WebSailor-V2 at ~60% at ~26 rounds.

In all four cases, WebLeaper is in the upper-left quadrant (higher performance, fewer action rounds) — a genuine Pareto improvement over the baseline. The magnitude of the efficiency gain is 10–15% fewer action rounds, while the accuracy gain ranges from 5–10 points on GAIA/xbench-DS to 7–12 points on BrowseComp/WideSearch. This is the most direct evidence for the paper's central claim that "enhancing search efficiency is a powerful lever for boosting the overall capabilities of IS agents" (Section 6), because it demonstrates that the efficiency gains from WebLeaper training are not achieved by trading off accuracy — both improve simultaneously.

The limitation: Figure 5 does not report these results in a table format with specific numeric values, making precise comparison difficult. The points on the scatter plot must be estimated visually. Additionally, the x-axis measures "average action rounds" which includes both Search and Visit actions — the paper's earlier distinction about ISE counting only Visit actions is not reflected in this visualization, so we cannot tell whether efficiency gains come from fewer Searches, fewer Visits, or both.

Reinforcement Learning Results (Table 3, Figure 6)

Under the comprehensive training setting, adding RL with the WebLeaper hybrid reward to the SFT-trained model yields:

  • BrowseComp: 37.80 → 38.8 (+1.0)
  • GAIA: 69.9 → 73.2 (+3.3)
  • xbench-DS: 69.0 → 72.0 (+3.0)
  • WideSearch SR: 1.5 → 4.0 (+2.5)
  • WideSearch Row F1: 23.0 → 31.0 (+8.0)
  • WideSearch Item F1: 45.4 → 48.5 (+3.1)

The gains are largest on WideSearch — the benchmark most aligned with entity-dense training — with Row F1 improving by a striking 8.0 points. This suggests the entity-level F-score reward in RL provides especially strong signal for broad retrieval tasks where partial credit (finding some but not all entities) is the typical training outcome. GAIA and xbench-DS both improve by ~3 points, demonstrating that the granular reward generalizes beyond entity-list tasks to standard QA evaluation. BrowseComp shows the smallest gain (+1.0), possibly because BrowseComp's evaluation already rewards thorough search, so additional RL optimization on entity coverage provides diminishing returns.

Figure 6 shows the training reward curve over 135 RL steps. The reward increases from approximately 0.38 at step 1 to approximately 0.63 at step 135, with the smoothed curve indicating a stable upward trajectory. The experiment was terminated at step 135 due to web access resource exhaustion, not due to convergence — the curve is still rising at termination, suggesting further RL steps would yield additional improvement. This is both a positive (the method is continuing to improve) and a limitation (the reported results are not at convergence, and the final performance ceiling is unknown).

Ablation Studies and Robustness Checks

Basic vs. Union vs. Reverse-Union task synthesis (Table 2): Basic-5k† degrades average performance by −7.64 relative to WebSailor-V2-5k, Union-5k† improves by +3.26, Reverse-Union-10k† improves by +4.34. This validates the progressive complexity design: entity density alone (Basic) is harmful when structural complexity is too low, entity density with multi-source integration (Union) is beneficial, and adding deductive fuzz (Reverse-Union) provides the strongest benefit, particularly on reasoning-heavy benchmarks.

ISR vs. ISE vs. ISR+ISE trajectory filtering (Figure 4): On GAIA (69.9 vs. 67.0 vs. 68.0) and BrowseComp (27.5 vs. 22.7 vs. 22.2), the joint filter outperforms either criterion alone. On WideSearch, all three strategies are equivalent within variance. The joint filter provides the strongest signal precisely on benchmarks where both coverage and efficiency are necessary for success.

SFT vs. SFT+RL (Table 3): Adding RL with the hybrid reward yields consistent improvements (+1.0 to +8.0 across all metrics). The largest gains are on WideSearch Row F1 (+8.0), where entity-level partial credit during RL provides dense feedback that SFT trajectories (filtered but not optimized for entity-level precision) do not.

WebSailor-V2-5k vs. WebSailor-V2-10k (Table 2): Doubling the baseline training data from 5K to 10K samples improves average performance by +1.16 (44.05 → 45.21). Adding 5K Union samples instead (+3.26 gain) is more effective than adding 5K more baseline samples (+1.16 gain), demonstrating that WebLeaper data quality yields higher marginal value per sample than additional in-distribution data. This is the closest the paper comes to a data efficiency claim, though it is confounded by the fact that Union increases both data quantity and data diversity (new task types), so the +2.10 differential may reflect diversity rather than entity density per se.

Comprehensive vs. Base setting (Table 1 vs. Figure 1): The comprehensive setting (larger training corpus) produces substantially higher absolute scores: GAIA 73.2 vs. 67.0–69.9, BrowseComp 38.8 vs. 22.1–23.0, xbench-DS 72.0 vs. 62.3–66.0. The paper presents these as validation that WebLeaper data remains effective in larger-scale training, but does not provide a comprehensive-setting baseline without WebLeaper data, making it impossible to attribute the gain to WebLeaper specifically vs. the larger training corpus. This is a significant missing ablation.

RL reward curve (Figure 6): The hybrid reward produces a stable, monotonically increasing training curve over 135 steps, with no evidence of reward hacking or training instability. The smoothed curve shows consistent improvement without plateaus, indicating the optimization landscape is well-behaved. However, the experiment was terminated due to resource constraints rather than convergence, so the reward ceiling is unknown.

Tool call distribution analysis (Appendix B, Figure 9): WebLeaper trajectories involve a high number of tool calls (Search, Visit, and total), with density distributions showing many instances exceeding 50 total actions. This confirms that WebLeaper's entity-intensive tasks induce extensive environment interaction. However, this is a descriptive statistic, not a causal ablation — we cannot tell whether the high action counts reflect thorough search (desirable) or inefficiency that survived the ISE filter (undesirable).

Entity count distribution (Appendix A.3, Figure 8): A significant portion of WebLeaper training samples contain at least 100 entities, confirming the entity-density design goal. The distribution shape is not fully characterized (mean, median, variance not reported), and the paper does not ablate how performance scales with different entity count ranges — e.g., whether 50-entity tasks are sufficient or 100+ is necessary for the efficiency benefits.

Missing ablation: Union data quantity. The Union variant is tested at 5K samples, Reverse-Union at 10K. The +4.34 gain for Reverse-Union-10k vs. +3.26 for Union-5k could partially reflect the doubled data quantity rather than the Reverse-Union design. A Union-10k ablation would disambiguate this.

Missing ablation: ISR and ISE threshold sensitivity. The paper uses α = 0.3 and β = 0.1 but does not vary these thresholds to test sensitivity. It is unknown whether stricter filtering (α = 0.7, β = 0.5) would improve or degrade performance by reducing training data quantity, or whether more permissive thresholds (α = 0.1) would already capture most of the benefit.

Missing ablation: GRPO vs. alternative RL algorithms. The paper uses GRPO without comparing to standard PPO (with a value function) or to simpler alternatives like rejection sampling fine-tuning (generate trajectories, filter by reward, SFT on the best). It is unknown whether the group-relative baseline specifically matters for the IS domain or whether any RL algorithm with the hybrid reward would perform similarly.

Missing ablation: ω (F-score weight) sensitivity. The paper introduces the weighted F-score parameter ω in Equation 9 but does not report its value or test sensitivity. If ω strongly favors recall, the precision penalty may be negligible and the reward effectively reduces to soft recall only.

Missing comparison: WebLeaper data vs. simply scaling baseline data. Table 2 compares 5K WebSailor-V2 against adding WebLeaper variants, but does not test whether adding, say, 50K more WebSailor-V2 samples (10x the baseline) would match or exceed the WebLeaper gain. This is the critical counterfactual: is entity density more valuable than simply more data from the same distribution? The +1.16 gain from 5K→10K WebSailor suggests diminishing returns, but 10K→50K might reveal different scaling behavior.

Critical Assessment

Do the experiments demonstrate that entity-intensive training improves search efficiency?

Yes, with qualifications. Figure 5 shows WebLeaper achieving higher accuracy with 10–15% fewer average action rounds across four benchmarks — a genuine joint improvement in both dimensions. Table 2 shows that Union and Reverse-Union variants improve over the WebSailor-V2 baseline by +3.26 and +4.34 average points respectively.

However, the evidence for efficiency specifically (as distinct from overall capability) has limitations. Figure 5 reports "average action rounds" as the efficiency metric, but this aggregates Search and Visit actions. The paper's own metric design distinguishes between them (ISE counts only Visits), so the aggregated metric may obscure whether efficiency gains come from fewer searches (better query formulation), fewer visits (better page selection), or both. The ablation on ISR vs. ISE filtering (Figure 4) shows that ISR (coverage) matters more than ISE (efficiency) on GAIA and BrowseComp, and both are equivalent on WideSearch — suggesting the efficiency signal may be less critical than the coverage signal for downstream performance. The conservative ISE threshold (β = 0.1) means the efficiency filter was extremely permissive, filtering only pathologically inefficient trajectories. The observed efficiency gains in Figure 5 may therefore derive more from the entity-dense SFT training teaching the model to search purposefully (because each task requires finding many entities within a budget) than from the explicit ISE filtering.

Do the experiments demonstrate that Reverse-Union prevents shortcut learning?

Partially. Table 2 shows Reverse-Union-10k† outperforms Union-5k† by +1.08 on average, with the largest gains on xbench-DS (+3.67) and WideSearch (+2.37) — benchmarks that test multi-step reasoning and broad retrieval. This is consistent with Reverse-Union's design forcing deductive reasoning before search. However, the data quantity is confounded (10K vs. 5K), and the paper does not provide qualitative evidence (example trajectories, error analysis) showing that Reverse-Union-trained agents actually exhibit the intended two-stage cognitive workflow (deduce anchor → search) rather than some other learned strategy that also works. The claim that Reverse-Union prevents shortcut learning is supported by the performance data but not directly verified through behavioral analysis.

Additionally, on GAIA, Reverse-Union-10k† slightly underperforms Union-5k† (66.99 vs. 69.90), and on BrowseComp they are essentially tied (27.67 vs. 27.50). If Reverse-Union's deductive fuzz universally prevented shortcuts, we would expect consistent improvement over Union. The fact that the gain is benchmark-dependent suggests Reverse-Union's benefit is specific to tasks where deductive reasoning is actually required for success — which is precisely the intended design, but also means the "prevents shortcut learning" claim is conditional on the evaluation distribution containing tasks unsolvable by shortcuts.

Do the experiments demonstrate that the hybrid reward enables effective RL on entity-intensive tasks?

Yes, but with limited evidence. Table 3 shows SFT+RL improves over SFT-only across all benchmarks, with the largest gains on WideSearch (+2.5 SR, +8.0 Row F1, +3.1 Item F1) — the benchmark most aligned with entity-intensive training. The training curve in Figure 6 shows stable improvement over 135 steps without instability. However, the paper does not compare the hybrid reward against alternative reward designs: binary success reward on WebLeaper tasks (does granular F-score actually outperform binary reward, or would binary work fine with sufficient entity density?), exact-match F1 (does soft scoring meaningfully outperform standard F1?), or LLM-as-judge monolithic evaluation (is the claimed cost-accuracy tradeoff empirically validated?). Without these comparisons, the claim that the hybrid reward is necessary for RL on entity-intensive tasks is asserted rather than demonstrated.

The SFT baseline in Table 3 already includes WebLeaper data during supervised fine-tuning, so the RL gain (+1.0 to +8.0) isolates the benefit of the RL stage with the hybrid reward. But this also means the total contribution of WebLeaper data (SFT + RL) vs. no WebLeaper data is not measured in Table 3. The comprehensive setting's SFT baseline (37.80 BrowseComp, 69.9 GAIA, 69.0 xbench-DS) is already quite high, and we don't know what a comprehensive SFT baseline without WebLeaper data would score.

Do the experiments genuinely demonstrate state-of-the-art performance?

For open-source agents, yes, with the caveat of training data confounds. Table 1 shows WebLeaper achieving the highest open-source scores on all five benchmarks. However, the base setting models are trained on WebSailor-V2-5k + WebLeaper variants, while baseline open-source agents are trained on different data mixtures. The comparison is therefore between entire training pipelines, not between WebLeaper data specifically and other data synthesis approaches. A fairer comparison would train all baselines on the same WebSailor-V2-5k base data plus their respective synthetic data augmentations, controlling for total training data quantity.

For proprietary agents, the comparison is even less controlled. The comprehensive setting WebLeaper (73.2 GAIA) exceeds Claude-4-Sonnet (68.3 GAIA) and OpenAI DeepResearch (67.4), but these proprietary agents are evaluated with unknown inference budgets, unknown tool configurations, and possibly different evaluation protocols or data splits (the paper uses GAIA's text-only validation subset, but doesn't confirm proprietary baselines use the same split). The proprietary agents may also be optimized for different trade-offs (safety, latency, cost) that reduce raw benchmark scores. The claim "surpasses proprietary agents" is technically true for the numbers reported but should be understood as a comparison of specific evaluation outcomes, not a controlled head-to-head measurement.

What genuine weaknesses exist in the experimental design?

No comprehensive-setting baseline without WebLeaper data. The comprehensive setting (Figure 1, Table 3 SFT+RL) achieves the strongest results, but the contribution of WebLeaper data specifically cannot be isolated because there is no comprehensive training run without WebLeaper data. The SFT baseline in Table 3 already includes WebLeaper data, and the RL gain is +1.0–8.0 points. The total WebLeaper contribution (SFT + RL stages) relative to a hypothetical comprehensive baseline is unknown.

Data quantity confounds in variant comparison. Union-5k vs. Reverse-Union-10k confounds variant type with data quantity. Union-10k and Reverse-Union-5k are missing ablations that would cleanly separate the effect of task design from data volume.

RL reward design not ablated. The claimed advantages of the granular F-score reward (soft scoring, type-specific modalities, weighted harmonic mean) over simpler alternatives are asserted but not tested empirically.

Efficiency metrics (ISR, ISE) not validated as predictors of downstream performance. The paper introduces ISR and ISE as trajectory quality metrics, filters on them, and shows the resulting models perform well. It does not test whether ISR/ISE scores of individual trajectories correlate with the usefulness of those trajectories for training — e.g., by binning trajectories by ISR/ISE and measuring per-bin contribution to downstream performance. This would validate that the filtering is selecting causally better training data rather than coincidentally correlated with quality.

Limited statistical reporting. WebLeaper results are averaged over three runs (stated in Figure 1 caption), but no standard deviations, confidence intervals, or significance tests are reported. Baseline scores are single-point estimates. The differences between WebLeaper and the next-best baseline are small in several cases (e.g., GAIA comprehensive: 73.2 vs. 68.3, a 4.9-point gap; xbench-DS: 72.0 vs. 71.0, a 1.0-point gap). Without variance estimates, we cannot assess whether these differences are statistically reliable, particularly on benchmarks with small test sets (GAIA validation: 103 samples, where a 5-point difference requires only 5 additional correct answers).

Terminated RL training. The RL experiment stopped at step 135 due to resource exhaustion while the reward curve was still rising. The reported RL results are therefore not at convergence, and the final performance ceiling of the approach is unknown. This also means the hybrid reward's long-term stability (does it eventually overfit or exhibit reward hacking at higher step counts?) is untested.

Single base model. All experiments use Qwen3-30B-A3B-Thinking-2507. The generalizability of WebLeaper's entity-density training to other model families (Llama, DeepSeek, Gemma) and scales is untested. The paper claims the approach should generalize, but provides no evidence.

Missing latency analysis. Figure 5 reports average action rounds as the efficiency metric, but action rounds do not equal wall-clock time. Search actions (Google API calls) have very different latency profiles than Visit actions (page fetching and summarization), and batching multiple queries or URLs per action further complicates the mapping from "action rounds" to actual response time. The paper's efficiency claims are about action economy, not end-to-end latency, which matters more for user-facing deployment.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in Any Headline Metric

The assumption or constraint. The entire WebLeaper framework relies on the ability to synthesize entity-intensive tasks from Wikipedia tables, collect trajectories by executing those tasks with a base model, and filter those trajectories using ISR and ISE metrics computed from ground-truth answer sets R. This pipeline requires significant upfront computation before any training begins: crawling ~2 million Wikipedia tables, running a multi-stage cleaning procedure, employing an LLM to select primary key columns and generate natural-language questions, executing thousands of trajectories through an open-source model with real web search API calls, and computing per-trajectory ISR/ISE scores against the constructed ground-truth entity sets. Additionally, the ISE metric requires computing n = |R| — the total number of required entities for each task — which itself requires the task synthesis engine to have enumerated the full answer set. The paper makes no attempt to account for this data synthesis cost in any efficiency comparison. The 4×4\times efficiency gains claimed in the introduction and the "higher performance with fewer action rounds" results in Figure 5 measure only inference-time action economy, not the total cost of producing the training data that enables that economy.

The consequence. A practitioner deciding whether to adopt WebLeaper vs. simply scaling up baseline training data cannot evaluate the true cost-effectiveness. The paper's Table 2 ablation shows that adding 5K Union samples to 5K WebSailor-V2 data yields a +3.26 average improvement across benchmarks, while adding 5K more WebSailor-V2 data yields only +1.16. At first glance, WebLeaper data appears more valuable per sample. But if synthesizing those 5K Union samples costs the equivalent of generating 50K WebSailor-V2 samples (including table extraction, task synthesis, trajectory collection, and filtering), then the "per-sample" comparison is misleading — the correct comparison is total end-to-end cost. The paper provides no information about synthesis costs, making it impossible to determine whether WebLeaper is genuinely more cost-effective or merely shifts the computation from inference-time to pre-training data preparation.

Furthermore, the trajectory filtering step requires ground-truth entity sets R for ISR computation (Section 3.2: ISR = |R ∩ O| / |R|). These entity sets are available because WebLeaper synthesizes the tasks — the answer set is defined during construction. But this means the filtering methodology is fundamentally tied to the synthetic task format. For real-world IS tasks where ground-truth entity sets are unavailable, ISR-based filtering cannot be applied, limiting WebLeaper's trajectory curation approach to synthetic data pipelines. The efficiency gains demonstrated in Figure 5 are the result of training on filtered synthetic data, not a method that can be applied to arbitrary in-the-wild trajectories.

What evidence exists in the paper. The paper acknowledges the synthesis effort only indirectly. Section 3.1.1 mentions "We crawled approximately 2 million tables from Wikipedia and applied a multi-stage cleaning procedure." Appendix A.4 describes the table cleaning stages in detail but provides no timing or cost estimates. The hybrid reward discussion in Section 3.3 explicitly considers computational cost for LLM-as-judge evaluation in RL — "running it for every single entity is prohibitively expensive for RL" — but no analogous cost analysis exists for the data synthesis pipeline itself. Figure 5 and Table 2 report efficiency as inference-time action rounds and accuracy as Pass@1, with no amortization of synthesis costs. The paper does not report total GPU-hours, API call volumes, or dollar costs for the data synthesis phase.

Mitigation status. The paper does not attempt to address this limitation. No synthesis cost analysis is provided, no amortization of synthesis costs into the efficiency metrics is performed, and the ability to apply ISR-based filtering to non-synthetic data is not discussed. The authors position WebLeaper as a training data synthesis framework, implicitly accepting that the synthesis cost is a one-time investment amortized over many downstream inference queries. However, without quantifying this investment, the practical value proposition remains incomplete.

All Results Are on a Single Base Model Architecture — Generalizability Across Model Families and Scales Is Unverified

The assumption or constraint. Every experiment in the paper uses exactly one base model: Qwen3-30B-A3B-Thinking-2507 (Section 4.1). This is a 30B-parameter model from Alibaba's Qwen family, with a specific architecture (Mixture-of-Experts with 3B active parameters) and a specific pretraining distribution. The paper states (Section 4.1): "We employ Qwen3-30B-A3B-Thinking-2507 as the base model" and does not test on any other model family or scale. The comprehensive setting experiments (Figure 1, Table 3) further introduce the Tongyi-DeepResearch-30B-A3B training corpus, which is also tied to the same model family and laboratory.

The consequence. The paper's central claim — that entity-intensive task synthesis improves IS agent efficiency and effectiveness — could be confounded by model-specific properties. Several candidate confounds exist:

  • The base model's initial search strategy quality. The paper's Figure 2 shows the valid action rate distribution for "the agent based on the GPT model" — not Qwen3-30B-A3B. We do not know the valid action rate for Qwen3-30B-A3B, and the 4% peak shown in Figure 2 may not characterize the base model used in all downstream experiments. If Qwen3-30B-A3B already has a higher valid action rate from its pretraining, the efficiency gains from WebLeaper training may be smaller on models with better innate search behavior — or larger on models with worse innate behavior.

  • The base model's table-reading capability. WebLeaper's task synthesis relies heavily on structured Wikipedia tables as the raw data source. A model with strong tabular reasoning capabilities (from pretraining on table-heavy data) might learn more effectively from tree-structured tasks than a model predominantly trained on free text. If Qwen3-30B-A3B has specific strengths in tabular data processing, the WebLeaper gains may not transfer to models with different pretraining distributions.

  • The 30B scale. The paper's comparisons in Table 1 show WebLeaper-30B outperforming Kimi-K2-Instruct-1T — a much larger model — on several benchmarks. This is presented as evidence that entity-rich data compensates for model scale. But the comparison is confounded: Kimi-K2-Instruct-1T is trained on different data, with a different training recipe, and potentially evaluated under different conditions. The claim that "data quality compensates for scale" would require comparing WebLeaper training at multiple model scales within the same architecture family (e.g., Qwen 7B, 30B, 70B) to establish a scaling trend — and then showing that the WebLeaper-trained 30B matches or exceeds the baseline-trained 70B. No such within-family scaling experiment is conducted.

  • The Mixture-of-Experts architecture. Qwen3-30B-A3B uses a MoE design with only 3B active parameters. It is unknown whether WebLeaper's training benefits (or costs) are specific to MoE architectures. MoE models may route different types of search behaviors to different experts, and the entity-intensive training objective might disproportionately improve certain expert pathways in ways that a dense model would handle differently.

What evidence exists in the paper. The single-model limitation is not acknowledged in the main text. Section 4.1 (Training Configurations) specifies the base model without discussing generalizability. Appendix A.4's data cleaning rationale and the tree-structured task design are presented as general methods, not tied to any model-specific properties. The baselines in Table 1 include models from diverse families (Qwen, Kimi-K2, Llama-based derivatives, unknown architectures), but these are independent agents trained on different data — they do not constitute a controlled comparison of WebLeaper across architectures.

Mitigation status. The paper does not address this limitation. No experiments use models from other families (Llama, DeepSeek, Gemma, Mistral), other scales (7B, 13B, 70B), or other architectures (dense vs. MoE). The authors' belief that "this model is representative of the capabilities of many contemporary LLMs" (Section 4, implied by their choice) is asserted without evidence. Future work on replication across model families would be necessary to establish generalizability.

The Hardest Problems Remain Essentially Unsolved — Test-Time Training Cannot Compensate for Fundamental Capability Gaps

The assumption or constraint. WebLeaper operates entirely within the training data preparation and RL fine-tuning paradigm — it improves the agent's search strategy and entity retrieval efficiency for tasks within the base model's capability range. The base model must already be capable of generating trajectories that achieve ISR > 0.3 (the coverage threshold in Section 3.2) to produce any training data at all. If the base model's pass@1 on a task class is near zero — it cannot generate any trajectory that finds even 30% of the required entities — then WebLeaper's trajectory collection and filtering pipeline produces no usable training data for that task class, and no amount of RL fine-tuning will help.

The consequence. WebLeaper should be understood as amplifying existing search capability, not creating it from nothing. This has several implications for deployment scenarios:

  • Distribution shift in production. If a deployed WebLeaper agent encounters question types that are semantically or structurally outside the distribution of Wikipedia-table-derived tasks (e.g., real-time news queries, proprietary database lookups, multi-modal information retrieval, or open-ended qualitative research questions without enumerable answer sets), the entity-intensive training may not transfer. The agent's search strategy was optimized on tasks with known answer structures derived from tabular data; facing unstructured information landscapes may reveal brittle learned behaviors.

  • The difficult tail of benchmark distributions. While WebLeaper achieves strong aggregate scores on benchmarks like GAIA (73.2%) and BrowseComp (38.8%), these scores are averages across all test questions. The paper provides no breakdown by question difficulty or question type. If the aggregate improvement comes disproportionately from easier questions within each benchmark — questions where the base model's initial search strategy was already somewhat effective — and WebLeaper provides minimal benefit on the hardest subset, then the efficiency gains are concentrated on the less-challenging part of the distribution. This would still be practically valuable (most user queries may be in the easier regime), but it would mean WebLeaper does not address the hardest IS challenges.

  • The WideSearch Item-F1 gap. On WideSearch (Table 1), WebLeaper-Reverse-Union C achieves Item F1 of 48.5, compared to Claude-4-Sonnet's 57.9 — a substantial 9.4-point disadvantage. WideSearch's Item F1 metric specifically evaluates the completeness of broad entity retrieval. The fact that WebLeaper trails a proprietary model on this metric, despite outperforming it on Success Rate (4.0 vs. 2.3) and trailing open-source models on Row F1, suggests that WebLeaper's entity-intensive training improves focused, structured retrieval but does not close the gap on the broadest, most comprehensive retrieval tasks — exactly the regime where entity density training should theoretically help most. This is a potential negative signal that entity density during training does not directly translate to entity completeness during broad evaluation.

What evidence exists in the paper. The capability bound is partially acknowledged. Section 3.2's filtering thresholds (α = 0.3, β = 0.1) implicitly define the capability floor: the base model must recover at least 30% of required entities in at least some trajectories. If it cannot, no training data is produced. The Benchmarks description (Section 4.1) lists five benchmarks of varying difficulty but does not stratify results by difficulty level within each benchmark. Figure 5 shows WebLeaper outperforming the baseline across all four benchmarks, but only at the aggregate level — the paper does not report per-difficulty-bin performance. The WideSearch Item-F1 gap is visible in Table 1 and Figure 1 but is not discussed in the text.

Mitigation status. The paper does not directly address the capability bound limitation. There is no analysis of performance on the hardest decile or quartile of each benchmark. There is no experiment testing whether WebLeaper training on easier synthesized tasks transfers to harder real-world tasks. The paper's conclusion that "enhancing search efficiency is a powerful lever for boosting the overall capabilities of IS agents" (Section 6) is stated without qualification about the capability floor beneath which efficiency training provides no benefit. Future work on combining WebLeaper's entity-intensive training with methods that expand the base model's capability range (e.g., retrieval-augmented pretraining, knowledge graph integration, or multi-modal training) would address this limitation.

The Deductive Reasoning Gains from Reverse-Union Are Not Directly Validated — Behavioral Change Is Inferred, Not Measured

The assumption or constraint. The paper claims that the Reverse-Union task variant prevents shortcut learning by forcing the agent to perform deductive reasoning before search. Section 3.1.3 states: "Reverse-Union prevents agents from succeeding with simple keyword searching and mandates a more robust, multi-step reasoning process." The supporting evidence for this claim is entirely performance-based: Reverse-Union-10k† achieves higher average benchmark scores than Union-5k† (+4.34 vs. +3.26 in Table 2), with the largest gains on xbench-DS and WideSearch.

However, why Reverse-Union improves performance is not empirically established. The paper provides no behavioral analysis demonstrating that Reverse-Union-trained agents actually exhibit the intended two-stage cognitive workflow (deduce anchor entity from descriptive clues, then use anchor attributes as pivots for union search). It is possible — and the paper does not rule out — that Reverse-Union training improves performance through a different mechanism. For example, the obfuscated question phrasing in Reverse-Union may simply expose the model to more diverse linguistic formulations of search tasks during training, improving its query formulation robustness without teaching genuine deductive reasoning. Or the higher entity density in Reverse-Union tasks (which combine union search with additional attribute-based filtering) may provide more supervision signal per task, and the performance gain reflects data efficiency rather than shortcut prevention.

The consequence. Without behavioral validation, the claimed causal mechanism — "deductive fuzz blocks shortcuts, forcing genuine reasoning" — remains a hypothesis consistent with the data but not proven by it. This matters for two reasons:

  • Reproducibility and transfer. If the mechanism is genuinely shortcut prevention, other practitioners seeking similar benefits should design tasks that structurally eliminate direct search keys. If the mechanism is something else (e.g., linguistic diversity, higher entity density), the design principle generalizes differently. The paper's framing pushes the community toward Reverse-Union-style obfuscation as a general shortcut-prevention technique, but if the true mechanism is different, efforts to replicate or extend the approach may fail or pursue the wrong design axis.

  • Robustness to adversarial evaluation. If Reverse-Union training genuinely teaches deductive reasoning, the resulting agent should be robust to adversarial task variations designed to probe for shortcuts — e.g., Union-style tasks with subtly misleading keyword matches, or Reverse-Union tasks with different types of descriptive clues. The paper does not test this. An agent that learned a different-but-effective strategy (e.g., better query decomposition for any type of obfuscated question) might perform well on the current benchmarks but fail on targeted adversarial evaluations.

What evidence exists in the paper. The paper provides extensive quantitative evidence in Table 2 and Figure 5 showing that Reverse-Union training yields the strongest performance. Section 3.1.3 and Appendix A.6.3 provide detailed walkthroughs of the intended Reverse-Union reasoning path with worked examples. However, the paper provides no qualitative analysis of trained agent behavior: no example trajectories showing the agent executing the deductive-then-union workflow, no error analysis distinguishing between failures at the deductive stage vs. the union search stage, and no ablation testing whether agents trained on Reverse-Union tasks are robust to adversarial shortcut probes. The paper's only behavioral evidence is the aggregate performance gain, which is consistent with the claimed mechanism but equally consistent with several alternatives.

The paper's earlier handling of Basic provides a useful contrast: the authors explicitly diagnose Basic's failure as shortcut learning ("tasks generated under this setting tend to be overly simple, allowing the model to infer complete answers from only a few information sources" — Section 4.3) and use this diagnosis to motivate Union and Reverse-Union. But for Reverse-Union, the diagnosis of success (shortcut prevention) is assumed from the design rather than verified from behavior. The paper was careful to explain why Basic failed; it is less careful to explain how Reverse-Union succeeds.

Mitigation status. The paper does not provide behavioral validation. The discussion in Section 4.3 attributes Reverse-Union's gains to "a certain degree of reasoning complexity" that "particularly enhances the model's planning and decision-making capabilities" — an inference from design and performance, not an observation of behavior. No trajectory analysis, attention visualization, or qualitative error breakdown is provided for any variant. This is a significant gap because the paper's conceptual contribution — shortcut prevention through structural task design — cannot be fully validated by performance metrics alone.

The Comprehensive Setting Baseline Without WebLeaper Data Is Missing — The Largest Reported Gains Cannot Be Cleanly Attributed

The assumption or constraint. The paper's strongest results appear in the comprehensive training setting (Figure 1, Table 3): WebLeaper achieves 73.2% on GAIA, 38.8% on BrowseComp, 72.0% on xbench-DeepResearch, and 48.5 Item F1 on WideSearch. These numbers are presented as evidence that "our approach generalizes well and remains effective even when evaluated under the comprehensive and realistic training setting" (Section 4.2). However, the comprehensive setting involves mixing WebLeaper data into the Tongyi-DeepResearch-30B-A3B training corpus — a substantially larger and more diverse dataset than the WebSailor-V2-5k base used in the base setting experiments. The paper does not report a comprehensive setting baseline without WebLeaper data. The SFT baseline in Table 3 (37.80 BrowseComp, 69.9 GAIA, 69.0 xbench-DS) already includes WebLeaper data during supervised fine-tuning — only the RL stage is ablated (SFT vs. SFT+RL). There is no comprehensive SFT baseline without WebLeaper data at all.

The consequence. The headline numbers in Figure 1 cannot be decomposed into "gain from larger training corpus" vs. "gain from WebLeaper data specifically." It is possible that:

  • The Tongyi-DeepResearch-30B-A3B corpus alone (without WebLeaper data) already achieves, say, 70% on GAIA and 35% on BrowseComp, and WebLeaper data adds +3.2 and +3.8 points respectively. This would be a modest but real contribution.

  • Alternatively, the Tongyi-DeepResearch-30B-A3B corpus alone already achieves 72% on GAIA and 37% on BrowseComp, and WebLeaper adds only +1.2 and +1.8 points — a much smaller contribution that might not justify the synthesis cost.

  • Or the Tongyi-DeepResearch-30B-A3B corpus alone underperforms the base setting baselines (e.g., 65% GAIA), and the comprehensive setting's strong performance is entirely driven by WebLeaper data. This would be the strongest case for WebLeaper.

Without the ablation, we cannot distinguish among these scenarios. The paper's base setting results (Table 1) do provide a clean WebLeaper-vs.-baseline comparison: Union B and Reverse-Union B achieve 69.9% and 67.0% on GAIA vs. the best open-source baseline (MiroThinker-32B-DPO-v0.2) at 64.1%. These gains are +3.0 to +5.8 points, and the comparison is controlled (same WebSailor-V2-5k base data, same Qwen3-30B-A3B model). But the comprehensive setting gains over the strongest proprietary baselines (73.2% vs. Claude-4-Sonnet's 68.3%) cannot be cleanly attributed to WebLeaper, because the Tongyi-DeepResearch corpus contribution is unknown.

This matters for the paper's central narrative: that entity-intensive training is a general principle that any practitioner can adopt. If the comprehensive setting gains are heavily dependent on the proprietary Tongyi-DeepResearch corpus (which is not publicly described or released), then the claim of "consistently achieving improvements in both effectiveness and efficiency over strong baselines" is confounded by access to proprietary training resources. If the gains are primarily from WebLeaper data, the base setting results (Table 1) already demonstrate this adequately, and the comprehensive setting adds only a demonstration of compatibility, not a demonstration of additional WebLeaper-specific value.

What evidence exists in the paper. Section 4.1 describes the comprehensive setting as mixing "WebLeaper data into the corpus of Tongyi-DeepResearch-30B-A3B, covering both the supervised fine-tuning and reinforcement learning stages, to examine its overall impact on performance." The phrase "overall impact" is ambiguous — it could mean the impact of the entire comprehensive training pipeline (including the Tongyi-DeepResearch corpus) or the marginal impact of adding WebLeaper to that pipeline. Table 3's SFT baseline is described as the comprehensive SFT model before RL, but its training data (whether it includes WebLeaper data or not) is not precisely specified. The paper states the SFT baseline "already includes WebLeaper data during supervised fine-tuning" only indirectly through the description of the comprehensive setting pipeline.

The paper's own caveat in Section 4.1 — "It is worth noting that this serves only as a supplementary setting applied in certain experimental sections. Unless otherwise specified, we adopt the base WebLeaper experimental configuration by default." — acknowledges that the comprehensive setting is supplementary, but this acknowledgment is easily missed given that Figure 1 (the paper's most prominent results visualization) presents comprehensive setting numbers front and center, and the abstract does not distinguish between base and comprehensive settings when reporting the headline 73.2% GAIA and 38.8% BrowseComp scores.

Mitigation status. The paper does not provide a comprehensive setting baseline without WebLeaper data. The authors acknowledge the supplementary nature of the comprehensive setting but do not acknowledge the missing ablation or the consequent inability to attribute gains. The base setting results in Table 1 are cleanly ablated and sufficient to demonstrate WebLeaper's contribution over the WebSailor-V2 baseline, but they do not support the strongest-reported numbers in Figure 1. A comprehensive baseline ablation would significantly strengthen the paper's claims about WebLeaper's contribution to state-of-the-art performance.

The RL Training Was Terminated Before Convergence — The Reward Ceiling and Long-Term Stability Are Unknown

The assumption or constraint. The reinforcement learning experiments in Section 4.6 and Figure 6 were terminated at 135 training steps because "web access resources were exhausted" (Figure 6 caption). The reward curve in Figure 6 shows a "stable and continuous upward trajectory" from approximately 0.38 at the start to approximately 0.63 at step 135, with no evidence of plateau. The smoothed reward curve is monotonically increasing and shows no sign of diminishing returns. This means the reported RL results in Table 3 (SFT+RL scores) represent a snapshot of an unconverged training process, not the final performance of a converged policy.

The consequence. The reported RL gains in Table 3 — +1.0 BrowseComp, +3.3 GAIA, +3.0 xbench-DS, +2.5 WideSearch SR, +8.0 WideSearch Row F1, +3.1 WideSearch Item F1 — are lower bounds on what the hybrid reward RL training would achieve at convergence. But this also means:

  • The true value of the hybrid reward design is under-measured. The paper makes a case for the hybrid reward as a key innovation, but the most convincing evidence — convergence to a high-performing policy — is not provided. An unconverged snapshot could dramatically underestimate (if the policy would improve significantly with more training) or overestimate (if the policy would eventually overfit or exhibit reward hacking) the RL contribution.

  • The scaling relationship between RL steps and performance is unknown. The reward curve shows roughly linear improvement over 135 steps. Does this continue linearly to step 500? Does it plateau at step 200? Does it degrade after step 300 due to over-optimization of the reward signal? Without the full curve, practitioners cannot estimate the RL compute budget needed to achieve a target performance level. The paper's own experience — exhausting web access resources at step 135 — suggests this is not just a theoretical concern: web API rate limits and costs are a practical constraint on RL for web agents, and knowing the required step count is essential for resource planning.

  • The hybrid reward's robustness to over-optimization is untested. The paper identified verifier over-optimization as a key concern in related work (Section 2's discussion of prior work on search and verifier reliability), but does not test whether the entity-level F-score reward is susceptible to similar over-optimization at high step counts. The smooth reward curve through step 135 is encouraging but insufficient to guarantee robustness — over-optimization often emerges after extended training as the policy discovers reward-exploiting strategies.

What evidence exists in the paper. The paper is transparent about the termination reason: "We terminated the experiment at 135 steps when web access resources were exhausted and evaluated the results at this point." (Figure 6). The reward curve is presented in full, clearly showing the upward trajectory and the termination point. The paper acknowledges the practical constraint without framing it as a limitation. The SFT+RL results in Table 3 are reported as-is, without caveats about the unconverged training state.

Mitigation status. The paper does not discuss the implications of unconverged training for result interpretation. The reported RL gains are treated as final results, not as lower bounds. The authors do not estimate the additional RL steps (and associated web API costs) that would be needed to reach convergence, nor do they discuss alternative training approaches (e.g., offline RL from a fixed trajectory dataset, or simulated web environments) that could reduce dependence on live web access. This is a practical limitation that directly affects practitioners attempting to replicate or extend the approach.

7. Implications and Future Directions

How This Work Changes the Landscape

A reframing of training data design: from task complexity to entity richness. This paper does not introduce a new agent architecture, a new RL algorithm, or a new search mechanism. Its contribution is a reframing of what matters in synthetic training data for information-seeking agents: not merely how hard the question is (multi-step reasoning depth, action horizon length), but how many pieces of useful information each training example contains. The tree-structured reasoning formulation that packs 50–150+ target entities into a single task is the mechanism that makes this reframing operational, but the conceptual shift is the insight that entity density functions as a scaling dimension — one that prior work overlooked entirely while optimizing structural complexity.

This matters because the field of agent training has been implicitly operating under an assumption that more training examples (or harder training examples) is the primary lever for improvement. The current leading open-source approaches — WebSailor, WebDancer, DeepDive — each scale by generating more trajectories or deeper reasoning chains. WebLeaper demonstrates that a modest number of entity-dense training examples (5,000–10,000) can outperform a larger quantity of entity-sparse examples, because each dense example provides granular feedback on dozens of retrieval decisions rather than binary feedback on a single answer. Table 2 makes this concrete: adding 5K Union samples to WebSailor-V2-5k yields a +3.26 average gain across five benchmarks, while adding 5K more WebSailor-V2 samples yields only +1.16. The entity-dense data is roughly 3× more valuable per sample.

The conceptual contribution is not that WebLeaper's specific datasets are valuable — it is that entity richness should be a first-class design dimension in synthetic data pipelines, alongside task complexity, domain diversity, and data volume. Proposition 1 (Var(ISE) = O(1/n)) provides theoretical grounding for why this dimension matters: efficiency measurement becomes reliable only when n (entity count) is sufficiently large. This formalizes what would otherwise be an intuition ("more entities per task = better training signal") and connects it to a mathematical property (variance scaling). The implication is that any synthetic data pipeline for information retrieval, multi-hop QA, or fact-gathering should track and optimize entity density as a quality metric — not just task difficulty.

Resolving the depth-vs-efficiency tension. Prior work on information-seeking agents faced an implicit tension: deeper search improves recall but multiplies actions, increasing cost and the risk of context-window exhaustion. The dominant research response was to make search deeper (more steps, more verification, more source integration), accepting the efficiency cost as necessary. WebLeaper demonstrates that this is a false choice — the same training methodology can simultaneously improve both depth (higher accuracy on complex multi-source tasks) and efficiency (fewer average action rounds). Figure 5 shows WebLeaper achieving higher accuracy with 10–15% fewer action rounds than WebSailor-V2 across all four benchmarks, occupying the upper-left quadrant (better accuracy, lower cost) in every case.

This is not a minor Pareto improvement. It means the efficiency-accuracy tradeoff that prior work accepted as inherent may instead be an artifact of training on entity-sparse data. When tasks contain few target entities, the model never learns action economy — any action that eventually leads to the answer is equally rewarded during behavioral cloning. When tasks contain dozens of entities packed into a constrained context, the model must learn to budget its actions to find everything before hitting step limits or context exhaustion. The efficiency gain is therefore not a separate optimization objective but an emergent property of entity-dense training — the model learns to search efficiently because entity-dense tasks require efficiency to succeed.

Making entity-level evaluation practical for RL. The hybrid reward system addresses a recognized bottleneck in applying RL to complex generative tasks: the tension between reward density (needed for effective learning) and reward accuracy (needed to prevent reward hacking). Standard binary success rewards are accurate (the agent either solved the task or didn't) but too sparse for entity-intensive tasks where perfect completion is rare. LLM-as-judge evaluation could provide dense feedback but is too expensive for RL-scale training. The entity-level F-score with type-specific semantic scoring is an engineering solution that navigates this tension: cheap string matching for most entities, targeted LLM calls for ambiguous types, and a soft scoring function that provides partial credit for near-matches.

This contribution should be understood as making entity-intensive RL tractable rather than as a theoretical advance. Prior to this work, the sparse-reward problem would have made RL on tasks requiring 50+ entity retrievals nearly impossible — the agent would almost never receive positive feedback during early training. The granular F-score converts "retrieve all 50 entities correctly" (binary, almost never achieved) into "retrieve 30 of 50 entities with reasonable precision" (continuous, achievable early in training, and improvable). This is the same principle that drives reward shaping in other RL domains, but applied to the specific structure of information retrieval tasks: the reward decomposes into per-entity contributions rather than treating the entire answer as a single outcome.

Which research directions become more attractive, and which become less so.

More attractive: Entity-aware training data design across all agent domains — not just web search but code generation (how many distinct test cases does each training example exercise?), GUI automation (how many distinct UI elements must be correctly manipulated?), and multi-agent coordination (how many distinct information exchanges must occur?). Any domain where the "answer" can be decomposed into independently evaluable sub-components can potentially benefit from entity-density thinking.

More attractive: Process-level reward design for compositional tasks. The hybrid reward's success suggests that decomposing holistic task success into per-component scores is a general strategy for making RL practical in compositional domains. For code generation, this might mean per-test-case rewards; for dialogue, per-turn or per-constraint-satisfaction rewards; for planning, per-subgoal-achievement rewards. The key insight — that cheap evaluation methods can cover most cases, with expensive methods reserved for ambiguity — is a design pattern that transfers broadly.

More attractive: Shortcut-proof synthetic data generation through structural impossibility rather than adversarial filtering. Reverse-Union's approach — design tasks where the naive keyword search strategy is structurally impossible because the required search key is never provided — generalizes to any domain where models exploit surface patterns. For code generation, this suggests designing tasks where the function signature or API name cannot be directly searched; for math reasoning, tasks where the problem template cannot be pattern-matched to known solutions. This is a stronger guarantee than adversarial filtering (which only catches shortcuts the current model exhibits) and is under-explored as a data design principle.

Less attractive, based on this paper's results: Unconstrained scaling of action horizon as an IS improvement strategy. Figure 5 shows that WebLeaper achieves higher accuracy with fewer actions, not more. The idea that "longer search chains = better answers" is undermined by the finding that many actions in baseline agents are unproductive (the 4% valid action rate in Figure 2). Future IS agent research should prioritize action quality over action quantity — making each search and visit count — rather than engineering the agent to sustain longer action sequences.

Less attractive: Model scale as the primary differentiator in IS agent performance. The fact that a 30B-parameter model trained on entity-dense data outperforms a much larger model (Kimi-K2-Instruct-1T) on several benchmarks (Table 1: BrowseComp 23.0% vs. 14.1%, GAIA 67.0% vs. 57.7%, xbench-DS 66.0% vs. 50.0%) suggests that data quality can compensate substantially for model scale in this domain. This does not mean scale is irrelevant — the paper uses a 30B model, not a 3B model, and the base model must already have non-trivial search capability — but it shifts the research emphasis from "how to train larger models" toward "how to construct better training data for available models."

Follow-Up Research This Work Enables

1. Behavioral validation of the Reverse-Union mechanism: does obfuscation actually prevent shortcuts, or does it provide a different benefit? The paper claims Reverse-Union forces deductive reasoning by structurally eliminating keyword-search shortcuts. The evidence is performance-based (Table 2: Reverse-Union-10k† outperforms Union-5k† by +1.08 average). A direct behavioral test would instrument the agent to log whether it executes the two-stage workflow (deduce anchor → use anchor attributes as pivot → perform union search) on Reverse-Union evaluation tasks. If trained agents consistently produce the intended workflow, the shortcut-prevention mechanism is validated. If they instead develop a different-but-effective strategy (e.g., iteratively refining search queries based on partial results without explicit anchor deduction), the mechanism is different from what the design intended, and the design principle generalizes differently. An adversarial evaluation could construct Reverse-Union-style tasks where the intended workflow is blocked (e.g., the descriptive clues are ambiguous between multiple anchors, or the pivot attribute is not uniquely identifying) and test whether Reverse-Union-trained agents degrade gracefully or fail catastrophically — distinguishing between robust deductive reasoning and brittle pattern exploitation.

2. Entity density scaling laws: how does performance scale with n (number of target entities per task) at fixed total training data volume? The paper demonstrates that entity-dense tasks (50–150+ entities) outperform entity-sparse tasks (3–5 entities typical in prior work), but does not map the curve — e.g., does moving from 10 to 50 entities per task provide most of the benefit, with diminishing returns beyond 50? Or does the benefit scale roughly linearly with n? A controlled experiment would fix total training data volume (e.g., 10K tasks), vary average entity count per task (10, 25, 50, 100, 200), and measure downstream benchmark performance. The prediction from Proposition 1 is that ISE measurement stability should improve with n, but the learning benefit may saturate once ISE is "stable enough" (perhaps at n = 30-50). Understanding this scaling relationship would tell practitioners the minimally sufficient entity density for their training pipeline, avoiding unnecessary synthesis complexity for tasks where 200 entities provide no marginal benefit over 50. This experiment would also reveal whether the Basic variant's failure (Table 2, -7.64 average) is primarily due to low entity density, low structural diversity, or both — by testing Basic-like single-source tasks at varying entity counts.

3. Does the entity-level F-score reward outperform simpler alternatives, and which component matters most? The hybrid reward system combines several design choices: soft semantic scoring (rather than exact match), type-specific evaluation modalities (rather than a single comparison method), weighted F-score aggregation (rather than binary success or unweighted F1), and hybrid integration with legacy rewards (rather than a unified reward). Which of these choices is necessary, and which is incidental? A systematic ablation would compare the full hybrid reward against: (a) binary success reward on WebLeaper tasks (does granularity matter at all?); (b) exact-match F1 (does soft scoring actually improve over counting exact string matches?); (c) unweighted F1 (does the ω weighting matter, and what is the optimal ω for the IS domain?); (d) uniform LLM-as-judge for all entities (is the claimed cost-accuracy tradeoff empirically supported?); and (e) WebLeaper-only reward without legacy data (does the hybrid integration prevent catastrophic forgetting, or would RL on entity-dense tasks alone transfer acceptably?). The paper's RL results (Table 3) show +1.0 to +8.0 gains from RL with the full hybrid reward, but this does not establish which components are load-bearing. A practitioner wanting to implement a minimal version of the reward needs to know the minimum viable design.

4. Cross-architecture and cross-scale replication: does entity-density training transfer across model families and sizes? The paper uses a single base model (Qwen3-30B-A3B-Thinking-2507, a 30B MoE architecture). A replication study would apply the WebLeaper pipeline (same data synthesis, same trajectory filtering, same SFT+RL recipe) to at least three model families (e.g., Llama-3, DeepSeek-V2, Gemma) at two scales each (e.g., 8B and 30B or 70B). Key questions: Does entity-density training provide consistent relative improvement across architectures, or is it specific to the Qwen family's pretraining distribution? Does the improvement scale with base model size (suggesting entity-dense training is complementary to scale) or does it diminish (suggesting larger models already learn efficient search from pretraining)? The paper's comparison against Kimi-K2-Instruct-1T (Table 1) hints that data quality can compensate for scale, but within-family scaling experiments would provide cleaner evidence. This is particularly important because the paper's Proposition 1 justification — that larger n enables more stable efficiency measurement — is architecture-agnostic, so the method should transfer. Confirming this empirically would establish entity density as a universal training data design principle rather than a Qwen-specific trick.

5. Open-domain entity-intensive task synthesis without Wikipedia table dependence. WebLeaper's task synthesis is fundamentally tied to structured Wikipedia tables — tables provide the entity-relation triples, the hierarchical organization, and the clean attribute structure that the three-layer reasoning tree requires. This limits the task distribution to domains well-covered by Wikipedia tables (historical data, awards, sports statistics, geographical information) and excludes domains where information is primarily unstructured (news, legal documents, scientific papers, social media, proprietary databases). A natural extension is to develop entity-intensive task synthesis from unstructured text sources — using information extraction to identify entity clusters and relations, constructing reasoning trees from extracted triples, and generating questions from these synthetic trees. This would require solving the entity alignment problem (different text sources may refer to the same entity differently) that Wikipedia tables avoid by design, but would dramatically expand the domain coverage. A strong follow-up would construct Union and Reverse-Union tasks from news article clusters (e.g., "Find all companies mentioned in both the Wall Street Journal and Financial Times as having revenue above $10B in Q3 2024") and measure whether WebLeaper-style training on unstructured-source tasks transfers to standard benchmarks. Negative results — e.g., that entity density benefits require the structural cleanliness of tables — would refine our understanding of the boundary conditions for this technique.

6. Direct combination of WebLeaper's entity-dense training with verifier-guided search at inference time. The paper trains agents for efficient search through SFT on filtered trajectories and RL with entity-level rewards — modifying the agent's policy to search better. A complementary approach, studied extensively in the LLM reasoning literature but not explored here, is to use a learned verifier at inference time to guide search (e.g., beam search against a process reward model). A natural combination: train the WebLeaper agent and train a separate process reward model that scores partial search trajectories by their ISR (fraction of required entities found so far). At inference time, use the verifier to decide whether to continue searching or submit an answer, enabling dynamic action budgeting per query rather than fixed step limits. This would address one of the paper's unstated limitations: the SFT+RL agent still operates with a predetermined action paradigm, whereas entity-dense tasks could teach a verifier to recognize when "enough" entities have been collected. A strong experiment would compare WebLeaper with fixed action budget vs. WebLeaper with verifier-guided early stopping, measuring both accuracy and average action count — testing whether learned search efficiency (from SFT) can be further improved by learned search termination (from a verifier).

Practical Applications and Downstream Use Cases

1. Production web research assistants (e.g., competitor analysis, due diligence, market research). These use cases require gathering dozens to hundreds of specific facts across multiple sources — exactly the capability WebLeaper's entity-dense training optimizes. A company evaluating potential acquisition targets needs to find all key executives, funding rounds, regulatory filings, product launches, and partnership announcements across the target and its competitors. Current IS agents (whether proprietary like OpenAI DeepResearch or open-source like WebSailor) would execute many redundant searches and page visits, running up API costs and latency. A WebLeaper-trained deployment could achieve equivalent or better coverage (higher ISR) with 10–15% fewer actions (Figure 5's efficiency gain), directly translating to lower per-query Google Search API costs and faster report generation. The entity-level precision and recall training (via the F-score reward) is particularly relevant here: missing a key executive or a regulatory filing is a higher-stakes error than retrieving an extra irrelevant fact, so ω > 1 (recall-weighted F-score) aligns with the deployment objective of minimizing false negatives.

2. Automated literature surveys and systematic reviews. Systematic reviews in medicine, social science, and computer science require identifying all papers meeting specific inclusion criteria across multiple databases (PubMed, arXiv, Semantic Scholar, etc.) — a process that is essentially a Union task: find the intersection of "papers about topic X" across multiple sources. Entity-dense training methods like WebLeaper could be adapted to generate training tasks from bibliographic databases (PubMed queries with known result sets) and train review agents that execute efficient multi-database searches. The Reverse-Union variant's deductive fuzz translates naturally to "grey literature" searches where the target concepts are described indirectly (e.g., find studies on interventions for "the condition characterized by X, Y, Z symptoms" without knowing the condition's technical name). The key practical benefit is completeness guarantees: the entity-level F-score reward during RL incentivizes the agent to verify that all required entities have been found (high recall), which is the defining quality metric for systematic reviews. Trajectory filtering by ISR > 0.3 would need adjustment to ISR > 0.9 or higher for this use case, raising the question of whether the SFT data volume would remain sufficient — a domain-specific adaptation challenge.

3. Data annotation and knowledge base population pipelines. Many organizations maintain internal knowledge bases (product catalogs, customer databases, competitive intelligence repositories) that must be periodically updated by scraping and synthesizing web information. Current approaches often use brittle rule-based scrapers or manual curation. A WebLeaper-style agent could be fine-tuned on synthetic tasks constructed from the existing knowledge base: each table in the database becomes a reasoning tree, the query becomes "find the current values for all attributes of all entities in category X," and the ground-truth answer set is the current database contents (enabling automated ISR computation for filtering and reward). The agent would learn to efficiently search the web for updates to each attribute, verify consistency across sources, and flag discrepancies. The practical benefit is cost reduction in knowledge base maintenance: training one agent that handles diverse update tasks versus maintaining per-source scrapers. The entity-density focus means the agent is explicitly trained to handle vast attribute sets (all columns across all rows) rather than single-attribute extraction, aligning with the batch nature of knowledge base updates.

4. Efficient training data generation for self-improvement loops. The paper demonstrates but does not explore the implication that WebLeaper's trajectory filtering (ISR + ISE) produces high-quality training data that can be used to further fine-tune the agent in iterative self-improvement cycles. In each cycle, the current-best agent generates trajectories on new synthetic tasks, the ISR/ISE filters select the most efficient and complete trajectories, and those trajectories train the next iteration of the agent via SFT. Because the filtering criteria are auto-evaluable (only requiring the ground-truth entity set R, which is known from task synthesis), this loop requires no human annotation. The practical benefit is sustained improvement without additional data collection cost: the synthesis pipeline generates new tasks (from untapped Wikipedia tables or other structured sources), the agent generates and filters its own trajectories, and the process repeats. The paper's RL results (Table 3: SFT+RL consistently outperforms SFT) suggest that iterative refinement beyond a single RL stage would yield further gains, though the resource exhaustion at RL step 135 indicates that web API costs are the binding constraint, not model training compute.

When to Prefer This Method

The paper positions WebLeaper as an augmentation to existing IS agent training pipelines — "mix our synthesized data and hybrid reward into your existing training corpus" — rather than as a replacement for specific alternatives. The base setting experiments (Table 1) always include the WebSailor-V2-5k baseline data alongside WebLeaper variants, and the hybrid reward (Equation 10) explicitly preserves legacy reward functions for non-WebLeaper tasks. There is no head-to-head comparison where WebLeaper replaces (rather than augments) an existing training methodology, and the paper does not articulate conditions under which a practitioner should choose WebLeaper over, say, simply doubling their baseline training data.

The closest the paper comes to a decision rule is in Table 2, where adding 5K Union samples yields +3.26 average gain vs. adding 5K more WebSailor-V2 samples yielding +1.16. This suggests — but does not state — that when operating under a fixed training data synthesis budget, entity-dense tasks provide higher marginal value per sample than additional entity-sparse tasks from the same distribution. However, this comparison does not account for the relative cost of synthesizing a Union task (table extraction → biclique enumeration → LLM question generation → trajectory collection → dual-metric filtering) vs. generating another WebSailor-V2 sample (LLM question generation → trajectory collection → standard filtering). Without cost normalization, the per-sample efficiency advantage could be negated by higher per-sample synthesis cost.

The paper also does not provide guidance on when to deploy the comprehensive vs. base setting — the comprehensive setting achieves much higher absolute scores (73.2 GAIA vs. 69.9) but requires access to the proprietary Tongyi-DeepResearch-30B-A3B training corpus. A practitioner with access to large proprietary training corpora would likely adopt the comprehensive approach; a practitioner building from only public data would use the base setting. This distinction is not presented as a deliberate tradeoff by the paper but is an implicit consequence of the training data differences.

Given these constraints, the decision to prefer WebLeaper-style entity-dense training over standard entity-sparse training reduces to a single empirically testable question: "Is my task domain one where target information can be decomposed into enumerable, independently evaluable entities?" If yes (structured fact-finding, list-retrieval QA, knowledge base population, multi-attribute entity extraction), the entity-density training paradigm is likely beneficial — the paper's results across five diverse benchmarks support this. If no (open-ended summarization, creative generation, opinion synthesis, tasks without clear "correct answer sets"), the entity-level filtering and reward mechanisms have no ground truth to operate on, and the approach is inapplicable. The paper does not test this boundary, leaving it as a hypothesis for practitioners to validate in their own domains.