Repurposing Synthetic Data for Fine-grained Search Agent Supervision

🎯 Pitch

Search agents trained with standard reinforcement learning treat all wrong answers identically, but the paper shows that simply rewarding how many intermediate entities (like 'Titanic' or 'Leonardo') a model identifies turns near-miss failures into powerful learning opportunities. By repurposing the scaffolding entities already baked into synthetic data as a dense reward, E-GRPO boosts accuracy by up to 6.8 points on deep research benchmarks while simultaneously teaching the agent to solve tasks with fewer tool calls.

1. Executive Summary

This paper proposes Entity-aware Group Relative Policy Optimization (E-GRPO), a novel reinforcement learning framework that repurposes the ground-truth entities discarded during synthetic data generation as a dense, fine-grained reward signal for training LLM-based search agents. Evaluating on 11 QA and deep research benchmarks—including multi-hop datasets like HotpotQA and challenging tasks like GAIA and BrowseComp—with Qwen2.5-7B and Qwen3-30B models, E-GRPO augments standard GRPO's sparse outcome-based reward with a partial-credit bonus proportional to the entity match rate in the agent's reasoning (e.g., awarding α·0.5 to a "near-miss" that correctly identifies Leonardo and Titanic but fails on the director, rather than the flat 0 penalty GRPO assigns). The method consistently and significantly outperforms the GRPO baseline—achieving, for instance, a 2.8-point average improvement on QA tasks for the 7B model and a 6.8-point Pass@3 gain on GAIA for the 30B model—while simultaneously learning more efficient reasoning policies that require fewer tool calls, establishing that fine-grained entity-aware supervision yields both superior accuracy and sample efficiency only when the entity-matching bonus is balanced against the primary correctness objective (with α = 0.3 proving optimal before performance degrades at α = 0.5).

2. Context and Motivation

The Core Problem: Training Search Agents with Sparse, Outcome-Based Rewards Discards Valuable Learning Signals

The fundamental problem this paper addresses is deceptively simple: when training LLM-based search agents with reinforcement learning, we systematically throw away information that could dramatically improve learning efficiency. Specifically, the dominant training paradigm—Group Relative Policy Optimization (GRPO) and its variants—uses a reward signal that depends solely on whether the agent's final answer matches the ground truth. Every incorrect answer, regardless of why it was incorrect or how close it got, receives an identical penalty. This is the reward sparsity problem, and it is particularly acute for search agents because their trajectories involve dozens of reasoning steps and tool calls, making it exceptionally difficult for a single scalar outcome signal to provide meaningful gradient information about which intermediate behaviors were productive and which were not.

To understand why this matters, consider what happens during a typical GRPO training step. The agent is given a complex question—say, "Who was the director of the 1997 film starring the actor who won the Academy Award for Best Actor for the film 'The Revenant'?" It generates a group of, for instance, eight complete rollouts, each potentially spanning 10–20 tool calls and reasoning segments. GRPO then assigns a reward of 1 to rollouts whose final answer matches "James Cameron" and a reward of 0 to everything else. The policy gradient is then computed from the relative advantage of each rollout within its group—correct rollouts get a positive signal, incorrect ones get a negative signal.

Now consider three different failure modes (illustrated in the paper's Figure 2):

Complete failure: The agent misunderstands the question entirely, searches for irrelevant information, and outputs "Baz Luhrmann." It matched zero entities from the ground-truth set (which includes key entities like "Leonardo" for the actor and "Titanic" for the film).
Near-miss: The agent correctly identifies that the actor is Leonardo and the film is Titanic, but makes a reasoning error at the final step and outputs "Gus Van Sant." It matched 1 out of 2 ground-truth entities—a 50% entity match rate.
Success: The agent traces through all reasoning correctly and outputs "James Cameron." It matched all entities.

Under standard GRPO, rollouts 1 and 2 receive identical rewards of 0. The policy sees no difference between a trajectory that got almost everything right and one that was completely lost. This is the "near-miss" problem: the model is penalized for partially correct reasoning just as harshly as for complete nonsense, destroying a rich source of learning signal about which steps in the reasoning chain were actually correct.

The paper's critical insight is that this problem is not merely theoretical—it manifests concretely in how the synthetic training data for search agents is constructed, and the solution is hiding in plain sight within that same data.

Why This Problem Matters: The Central Role of Synthetic Data and the Entities It Discards

The importance of addressing the near-miss problem is magnified by how search agents are trained in practice. The dominant paradigm for creating training data is entity-centric synthetic data generation, exemplified by methods like ASearcher (Gao et al., 2025) and SailorFog-QA (Li et al., 2025b). These pipelines systematically construct complex, multi-hop questions by manipulating entities—the factual building blocks of the answer.

The paper provides a concrete example in Figure 1 (left). The process starts with a simple seed question like "Who was the director of the 1997 film starring Leonardo?" Through operations like fact injection (replacing "Leonardo" with a descriptive phrase) and fuzzing (making entity references more ambiguous), the system constructs a progressively more complex question. Each transformation introduces or modifies an entity, and all of these entities are tracked as ground truth throughout the generation process. The final question—"Who was the director of the 1997 film starring the actor who won the Academy Award for Best Actor for the film 'The Revenant'?"—embeds a trail of entities: Leonardo (the actor), Titanic (the 1997 film), and James Cameron (the final answer).

Here is the crucial observation that motivates the paper: after the synthetic question-answer pair is generated, the intermediate entities are discarded. Prior work uses only the final question and answer for training—the entity trail, which represents the factual backbone connecting the question to the answer, is treated as a byproduct of the generation process and thrown away.

This is a remarkable waste for two reasons. First, these entities represent precisely the kind of intermediate supervision that could distinguish near-misses from complete failures in RL training. If we knew which entities the agent successfully identified during its reasoning, we could assign partial credit proportional to that subset. Second, using these entities requires zero additional annotation cost—they are already generated as part of the data synthesis pipeline. The paper's core contribution is the recognition that these discarded entities are not waste material but rather an untapped, computationally free source of fine-grained supervision.

The practical impact of squandering this signal is substantial. As the paper demonstrates (Figure 1, right, and Section 3.1), there is a strong positive correlation between the number of ground-truth entities an agent identifies during reasoning and whether its final answer is correct. On a subset of SailorFog-QA questions, correct rollouts had higher average entity match rates than incorrect ones in roughly 80% of questions (1,939 vs. 487). Furthermore, the distribution of entity match rates reveals a bimodal pattern for incorrect rollouts: a large peak at 0.0 (complete failures) and a notable spread across the mid-to-high range (near-misses). This directly validates that entity match rate captures granular reasoning quality that a binary outcome reward cannot.

Where Prior Approaches Fall Short: The Inadequacy of Existing Solutions to Reward Sparsity

The problem of sparse rewards in reinforcement learning is not new, and several approaches have been developed in other domains. The paper systematically explains why these existing solutions are ill-suited for training search agents.

Process Reward Models (PRMs) are prohibitively expensive for web search. In mathematical reasoning and code generation, a standard solution to reward sparsity is to train a Process Reward Model—a separate neural network that evaluates the correctness of each intermediate reasoning step (Fan et al., 2025; Anonymous, 2025; Zhang et al., 2025b). The PRM provides dense, step-level feedback that guides the policy toward productive reasoning. However, as the paper argues (Section 1), this approach does not transfer to web search for two reasons:

Annotation cost: Training a PRM requires labeled data for individual reasoning steps. In mathematics, this can be done through automated verification of intermediate derivations or through Monte Carlo rollouts from each step. In open-ended web search, there is no equivalent of an "intermediate derivation"—the correctness of a search query, a browsing decision, or a snippet extraction is context-dependent and subjective. Human annotation of step-level correctness across thousands of diverse search trajectories would be economically infeasible.
Distributional complexity: The web is dynamic, noisy, and vast. A PRM trained on today's search results would need constant retraining as websites change, new content appears, and search engine behavior evolves. The paper characterizes this as ill-suited for "the open-ended nature of web search" where "the sheer scale and dynamic nature of web content render the annotation required for a PRM prohibitively expensive."

Tree-based sampling and complex advantage estimation are computationally intractable. Another approach for mitigating reward sparsity is to improve the quality of advantage estimates through sophisticated sampling strategies. Methods like TreeRPO (Yang et al., 2025) and TreeRL (Hou et al., 2025) construct search trees over partial reasoning trajectories, using Monte Carlo Tree Search or similar algorithms to derive more informative step-level advantages. While effective in domains with short, well-structured reasoning chains (e.g., math problems with 5-10 steps), the paper argues this approach breaks down for search agents:

Trajectory length: Search agent rollouts routinely involve dozens of tool calls—search queries, page visits, information extraction steps—each interleaved with free-form reasoning. Constructing and evaluating search trees over such long, branching trajectories would multiply the already substantial computational cost of RL training.
Tool call overhead: Each node in such a tree would require executing actual search engine calls and web page fetches. The latency and computational cost of these external interactions make tree-based exploration infeasible at the scale needed for RL training.

Existing GRPO variants do not address the fundamental sparsity problem. The paper acknowledges that the GRPO family has seen numerous refinements. Methods like DAPO (Yu et al., 2025) improve exploration through higher clipping bounds and KL-divergence removal. ARPO (Dong et al., 2025) adapts the framework with entropy-based rollout mechanisms for multi-turn web search. However, the paper makes the crucial observation that all of these variants remain fundamentally constrained by the same outcome-based reward formulation (Section 5, Related Work). They modify how the gradient is computed from a group of rollouts, but they do not change what information is available in the rewards themselves—every incorrect rollout is still treated as equally undesirable, regardless of how much correct reasoning it contained.

The paper's position is that this is not a minor inefficiency that can be optimized away with better clipping or normalization schemes. It is a structural limitation of the outcome-based reward paradigm. No amount of improvement to the advantage estimation machinery can recover information that was never present in the reward signal to begin with. What is needed is a fundamentally richer reward that captures reasoning quality at a granularity finer than the final answer—and the paper argues that the discarded entities from synthetic data generation provide exactly this signal at negligible additional cost.

How This Paper Positions Itself: Bridging Data Synthesis and Policy Optimization

The paper's positioning is distinctive because it identifies value in a resource that prior work treated as waste. The intellectual move is to recognize that the entity-centric nature of synthetic data generation and the reward sparsity problem in GRPO training are two sides of the same coin—and that connecting them solves both problems simultaneously.

The key positioning claims (Section 1, contributions) are:

First, the paper identifies the "near-miss" problem as a specific, empirically validated failure mode of GRPO-based training. The analysis in Section 3.1 is not merely anecdotal; it quantifies the correlation between entity matching and accuracy across a large sample of questions, establishes that the entity match rate distributions for correct and incorrect rollouts are separable, and demonstrates that a substantial fraction of incorrect rollouts are "near-misses" occupying the mid-to-high entity match rate range. This provides the empirical foundation for the method.

Second, the paper proposes that entities from synthetic data generation can be repurposed as a fine-grained reward signal. This is the conceptual innovation. Prior work used only the final question-answer pairs from synthesis pipelines for post-training (Dong et al., 2025; Wu et al., 2025a; Li et al., 2025b). The paper, by contrast, argues that "these discarded entities" should be viewed "not as a byproduct, but as an untapped source of fine-grained, factual supervision" (Section 5). The paper explicitly positions itself as "the first, to our knowledge, to recognize this" (Section 5), framing the contribution as bridging the gap between the data generation process and the RL alignment phase.

Third, the method requires no additional annotation, model training, or complex sampling. Unlike PRMs (which require training a separate verifier model) and tree-based methods (which multiply computational cost), E-GRPO's entity-aware reward is computed via exact string matching against a pre-existing entity set. The paper emphasizes this efficiency as a key differentiator (Section 5): the reward is "both fine-grained and computationally efficient, requiring no additional annotation, model training, or complex sampling." This positions E-GRPO not as a more sophisticated version of existing dense-reward approaches, but as a fundamentally different category of solution—one that derives richness from repurposing information that already exists in the training pipeline rather than from adding new computational infrastructure.

Fourth, the paper explicitly scopes its contribution as algorithmic validation rather than state-of-the-art chasing. The authors state in Section 4.1 that their "study aims to validate the effectiveness of E-GRPO at the algorithmic level, not merely to pursue state-of-the-art performance." They use limited training data (11K SFT samples, 1K RL samples) and modest model sizes (7B and 30B parameters) to demonstrate that the method works across different architectures and environments. This positions the paper as establishing a principle—that entity-aware rewards improve GRPO training—rather than as a system paper claiming dominance over all baselines.

Fifth, the method is evaluated across a deliberately broad range of settings to demonstrate robustness. The experiments span two distinct training environments (a closed-world local knowledge base and the live open web), two model architectures (dense Qwen2.5-7B and MoE Qwen3-30B), and eleven evaluation benchmarks spanning single-hop QA, multi-hop QA, and challenging deep research tasks like GAIA and BrowseComp. This breadth is intentional: it addresses the concern that the method might only work on data from the specific synthesis pipeline that generated the entities, or only in simplified training environments, or only for certain types of questions.

The Broader Significance: A New Category of Supervision Signal

Stepping back from the specific mechanism, the paper can be understood as opening up a new dimension in the design space of RL reward functions for language agents: process-level supervision derived from structural properties of the training data itself, rather than from external verifiers or human annotation.

This is significant because it suggests a general principle that may extend beyond the specific entities-and-string-matching implementation in this paper. Many synthetic data generation pipelines impose structures on their outputs—knowledge graphs in SailorFog-QA, transformation histories in ASearcher, dependency trees in compositional question generators. These structures encode intermediate correctness criteria that are typically discarded when the data is reduced to (question, answer) pairs. The paper demonstrates that preserving and exploiting one such structure (entity trails) yields measurable improvements. The natural question is whether other structural byproducts of data synthesis—logical subgoals, reasoning step templates, or information-seeking sub-queries—could similarly be repurposed as dense reward components.

In this light, the paper is not merely proposing a better GRPO variant but rather pointing toward a more general methodology: align the reward signal with the structure that is already implicit in how the training data was constructed.

3. Technical Approach

3.1 Reader orientation

This paper presents a reinforcement learning algorithm — not a standalone system or application, but a modification to how policy gradients are computed during training. Specifically, it is an extension of Group Relative Policy Optimization (GRPO) that changes the reward function to incorporate entity-matching information.

The core problem it solves is that standard GRPO gives the same penalty (reward of 0) to all incorrect answers regardless of how much correct reasoning they contain. The solution is to assign partial credit to incorrect rollouts proportional to how many ground-truth entities from the synthetic data generation process the agent successfully identified in its reasoning — turning a binary reward into a dense, graduated signal at negligible additional computational cost.

3.2 Big-picture architecture (diagram in words)

The E-GRPO training system has five major components that interact in a loop:

Synthetic Data Pipeline (external input) — produces training questions paired with ground-truth entities that form the factual backbone of the correct answer. These entities are a byproduct of the question-generation process (e.g., ASearcher or SailorFog-QA) and are normally discarded after the (question, answer) pair is extracted.
Search Agent (policy $\pi_\theta$ ) — the LLM being trained, operating in a ReAct loop. Given a question, it produces multi-turn rollouts of interleaved reasoning (thinking) and tool calls (search, visit), terminating with a final answer. Each rollout is a sequence of tokens that includes thoughts, actions, and observations.
Entity Matcher (reward component) — for each rollout, performs exact string matching between the agent's reasoning text (specifically the thinking blocks) and the ground-truth entity set from the data pipeline. Produces an entity match rate $\gamma_i$ , normalized within the rollout group to $\hat{\gamma}_i \in [0, 1]$ .
Entity-Aware Reward Function — computes the reward $R_i$ for each rollout by combining the binary outcome (1 if answer is correct) with the normalized entity match rate (partial credit $\alpha \cdot \hat{\gamma}_i$ if incorrect but no format/overlength errors). This replaces the standard GRPO reward of $R_i \in \{0, 1\}$ with a richer signal.
GRPO Policy Optimizer — takes the group of rollouts, their entity-aware rewards, and computes group-relative advantages $\hat{A}_{i,j}$ via normalization, then updates the policy $\pi_\theta$ using a clipped surrogate objective (identical in structure to standard GRPO but with the enriched rewards feeding into the advantage calculation).

The training loop proceeds as follows: a batch of questions is sampled from the synthetic dataset → the current policy generates $G$ rollouts per question → the entity matcher computes match rates from the thoughts in each rollout → rewards are assigned → advantages are computed → the policy is updated via gradient ascent on the GRPO objective → repeat.

3.3 Roadmap for the deep dive

First, the entity matching mechanism — what counts as a "matched entity," how the match rate is computed and normalized, why the matching is confined to thoughts rather than the full trajectory. This is the foundation on which the entire reward signal is built.
Second, the entity-aware reward function — how the normalized match rate is combined with the outcome signal, the role of the $\alpha$ hyperparameter, and how format errors and overlength rollouts are handled. This is the central modification to GRPO.
Third, the advantage computation and policy objective — how the enriched rewards flow into the standard GRPO machinery, what modifications to the GRPO framework are adopted from prior work (DAPO), and how the overall training objective is structured.
Fourth, the data synthesis context — a brief account of how the ground-truth entities are constructed in the two generation methods used (ASearcher and SailorFog-QA), since the entity sets are the input to the entire reward mechanism.
Fifth, the training pipeline — cold-start SFT, RL data generation, hyperparameters, and the practical design choices that enable the method to work.
Sixth, critical design decisions — why exact string matching rather than LLM-based matching, why thoughts-only rather than full-trajectory matching, and how the normalization scheme ensures stable training across questions of varying difficulty.

3.4 Detailed, sentence-based technical breakdown

This is primarily an algorithmic contribution paper whose core idea is that the ground-truth entities generated during synthetic data creation can be repurposed as a dense, fine-grained reward signal for GRPO training, enabling the policy to learn from "near-miss" rollouts that standard outcome-based rewards treat identically to complete failures.

Entity Matching: Defining and Computing the Match Rate

The entity matching mechanism is the foundation of E-GRPO — it transforms a set of ground-truth entity strings and an agent's reasoning text into a scalar that quantifies how much factual progress the agent made before producing its final answer. Every other component of the method depends on this signal being meaningful, so the design choices here are critical.

What counts as a matched entity. For a given synthetic question-answer pair $(q, gt)$ , the data generation pipeline produces a set of $m$ ground-truth entities:

$E_q = \left\{ e^{(1)}, e^{(2)}, \ldots, e^{(m)} \right\}$

where each $e^{(k)}$ is a short, definite string — a named entity that forms part of the factual backbone connecting the question to the correct answer. Examples from the paper include "Leonardo" (the actor), "Titanic" (the film), "Tegetthoff" (a ship name), "International Polar Year" (a scientific initiative), and "Royal Geographical Society's Founder's Medal" (an honor).

For a specific rollout $H^{(i)}$ from the agent, let $T^{(i)}$ be the collection of all thinking blocks in that rollout — the free-form reasoning text the agent generates at each step, excluding tool call specifications and tool responses. The set of matched entities is:

$E^{(i)}_{\text{matched}} = \left\{ e \in E_q \mid \exists t \in \{1, \ldots, T_i\}, e \text{ is mentioned in } \tau^{(i)}_t \right\}$

where "mentioned" means the entity's full string appears as an exact substring match somewhere in the thought text.

What it computes: for a given rollout, this operation scans every thought segment and records which ground-truth entities appear verbatim. The output is a subset of $E_q$ — potentially empty (no entities found), potentially the full set (all entities found), or any intermediate subset.

Why exact string matching rather than semantic matching: the paper identifies two practical problems with using an LLM-based judge for this task (Appendix B.2). First, computational latency: running an LLM to semantically parse entity presence in long reasoning traces — which can span thousands of tokens across dozens of steps — would introduce substantial latency into the RL training loop, where the matching must be performed for every rollout in every batch. Exact string matching is computationally trivial. Second, reward hacking vulnerability: in preliminary experiments, the authors observed that agents learned to produce verbose, superficially relevant text that would mislead a semantic LLM judge into erroneously assigning partial credit without actually containing the correct entities. The rigidity of exact matching acts as a safeguard against this exploitation — the agent cannot "talk its way" into unearned entity credit, since only the literal entity string triggers a match.

Why thoughts-only rather than full-trajectory matching: the agent's full trajectory includes both its own reasoning (thinking blocks) and the observations returned by tools (search snippets, web page extracts). The paper's analysis (Appendix B.2, Figure 5) reveals that matching across the entire trajectory produces significantly more "false positives": incorrect rollouts that still achieve a high entity match rate because the entity appeared in a search result or web page that the agent retrieved but failed to incorporate into its reasoning. The paper provides a qualitative explanation: "a key entity is present in the observation returned by a tool (e.g., a search snippet), but the agent fails to extract and incorporate this information into its reasoning process." Rewarding based on trajectory-level matching would grant unearned credit for merely encountering information — a noisy signal that fails to penalize the genuine reasoning failure of not understanding or acting on what was retrieved. Confining matching to thoughts ensures "the reward is directly coupled to the model's ability to identify and internalize key information."

Definition of entity match rate. The raw entity match rate for rollout $i$ is the fraction of ground-truth entities that were matched in its thoughts:

$\gamma_i = \frac{|E^{(i)}_{\text{matched}}|}{|E_q|} = \frac{|E^{(i)}_{\text{matched}}|}{m}$

What it computes: a scalar between 0 and 1 representing the proportion of factual building blocks the agent successfully identified during reasoning. $\gamma_i = 0$ means no ground-truth entities were found; $\gamma_i = 1$ means all of them were found.

Why a fractional match rate rather than a binary flag: a binary "found some entities / found none" signal would collapse the distinction between a rollout that found one entity and one that found all but one. The continuous match rate preserves granularity — it captures the intuition that a rollout matching 3 out of 4 entities had better factual coverage than one matching only 1, even though both are "partial matches." This granularity is what enables the reward function to differentiate near-misses from complete failures.

Group-level normalization. Raw match rates are not directly comparable across questions because different questions have different numbers of entities, different entity difficulties, and different base model capabilities. To put all rollouts on a common scale, the paper normalizes each rollout's match rate within its group:

$\hat{\gamma}_i = \begin{cases} \frac{\gamma_i}{\gamma_{\max}} & \text{if } \gamma_{\max} > 0 \\ 0 & \text{otherwise} \end{cases}$

where the group maximum is:

$\gamma_{\max} = \max_{j \in \{1, \ldots, G\}} \gamma_j$

What it computes: for a group of $G$ rollouts on the same question (typically $G = 8$ in the paper's experiments), the normalized match rate rescales each rollout's raw rate so that the best-performing rollout in the group has $\hat{\gamma} = 1.0$ and all others are expressed as a fraction of that best performance. If no rollout matched any entity ( $\gamma_{\max} = 0$ ), all normalized rates are 0.

Why group-level normalization is necessary and sufficient: the normalization serves two purposes. First, it creates a consistent 0-to-1 scale for the entity-aware reward component, which is essential for stable advantage computation across groups — without normalization, a question with 10 entities would naturally produce much larger raw match rate differences than a question with 2 entities, making the reward signal's magnitude dependent on question structure rather than reasoning quality. Second, by using the group maximum rather than an absolute standard (e.g., the total number of entities), the normalization is adaptive to the current policy's capabilities — if the policy is still learning to identify any entities on a particular question, a rollout matching 2 out of 1 is normalized to 1.0, providing a positive reward that encourages the behavior even though it is not perfect. This makes the reward signal meaningful throughout training, from early stages where entity matching is rare to later stages where it is common.

Why normalize by the maximum rather than standardize (z-score): standardizing within the group (subtracting mean, dividing by standard deviation) would produce positive match rates for rollouts above the group average and negative rates for those below — this is inappropriate for a reward bonus because it would penalize rollouts that are simply average within their group, even if they matched several entities. The max-normalization ensures that the entity bonus is always non-negative (it is either zero or a positive fraction of the best performance), which aligns with the intuition that matching entities is always good, just better in some rollouts than others. The entity bonus is meant to reduce the penalty on near-misses, not to create a new penalty on intermediate-quality rollouts.

Entity-Aware Reward Function

The entity-aware reward function is the central modification that distinguishes E-GRPO from standard GRPO. It replaces the binary $\{0, 1\}$ reward with a more expressive scheme that differentiates the quality of incorrect rollouts.

Standard GRPO reward. In standard GRPO training for search agents, the reward is defined purely by outcome correctness — typically assessed by comparing the agent's extracted final answer (from <answer> tags) against the ground-truth answer using an LLM judge. The reward is simply:

$R_i^{\text{GRPO}} = \begin{cases} 1 & \text{if the final answer is correct} \\ 0 & \text{otherwise} \end{cases}$

What this computes: a binary scalar per rollout that indicates whether the task was solved. Every incorrect rollout — regardless of how much correct intermediate reasoning it contained — receives identical zero reward. This is the reward sparsity that E-GRPO addresses.

Why this is problematic for search agents specifically: search agent trajectories are long (the paper uses up to 40 tool calls), involve diverse information-gathering sub-tasks (searching for the actor, browsing the film's page, verifying the director), and have many intermediate decision points. A binary outcome signal provides no gradient about which of these sub-tasks were performed correctly — the policy receives the same negative signal whether it completely misunderstood the question or correctly executed every step except the final answer extraction. This makes credit assignment extremely difficult and slows learning.

E-GRPO reward function. The proposed reward is:

$R_i = \begin{cases} 1 & \text{if } H^{(i)} \text{ is correct} \\ \alpha \cdot \hat{\gamma}_i & \text{if } H^{(i)} \text{ is wrong} \\ 0 & \text{if error occurs in } H^{(i)} \end{cases}$

where:

$\alpha \in [0, 1]$ is a hyperparameter balancing the value of answer correctness versus entity matching,
$\hat{\gamma}_i \in [0, 1]$ is the normalized entity match rate computed as described above,
"error" refers specifically to format errors and overlength rollouts (defined below).

What it computes: for each rollout, the reward is $1$ if the answer is correct (regardless of entity match rate — a correct answer always gets full credit), $\alpha \cdot \hat{\gamma}_i$ if the answer is wrong but the rollout was properly formatted and within length limits (providing partial credit proportional to entity-matching performance), or $0$ if the rollout had format violations or exceeded tool-call/length limits (providing no credit regardless of entity matching, since such rollouts cannot be reliably evaluated and may destabilize training).

Operational meaning: for the near-miss example in Figure 2 — where the agent identified 1 out of 2 entities (matched "Leonardo," missed "Titanic") and produced the wrong final answer — the normalized match rate within its group would depend on the best-performing rollout. If another rollout in the group matched both entities, then $\hat{\gamma} = 0.5$ and the near-miss receives $R = \alpha \cdot 0.5$ . With the default $\alpha = 0.3$ , this is $R = 0.15$ . The complete failure (0 entities matched) receives $R = 0$ . The correct rollout receives $R = 1$ . The reward spectrum is now $\{0, 0.15, 1\}$ rather than $\{0, 0, 1\}$ — the near-miss is unambiguously distinguished from the complete failure.

Why the reward for correct answers is always 1 regardless of entity match rate: the entity match rate is a proxy for factual correctness, but the final answer correctness is the ground truth. If the agent produces the correct answer, it has achieved the primary objective — providing a reward less than 1 because it took an unusual path to get there (e.g., low entity match rate) would penalize valid reasoning strategies that happen not to surface entities in the thought text. The entity bonus is designed to differentiate failures, not to down-weight successes.

Why $\alpha$ is a hyperparameter rather than fixed at 1: setting $\alpha = 1$ would mean that an incorrect rollout matching all entities receives the same reward (1.0) as a correct rollout. This would eliminate the distinction between being factually thorough and being actually correct, potentially creating a reward landscape where the agent optimizes for entity coverage at the expense of answer accuracy. Conversely, setting $\alpha$ too low (close to 0) reduces to standard GRPO. The paper's ablation study (Figure 4) sweeps $\alpha \in \{0.0, 0.1, 0.3, 0.5\}$ and finds that performance peaks at $\alpha = 0.3$ across all four deep research benchmarks, with $\alpha = 0.5$ showing degradation — confirming that an excessively strong entity-matching bonus can distract from the primary correctness objective.

Handling format errors. Rollouts that do not adhere to the required ReAct format (detailed in Appendix A) are assigned a reward of $0$ . The format requires specific XML-like tags (thinking, response, <tool_call>, </tool_call>, <tool_response>, </tool_response>, <answer>, </answer>) with tool calls as valid JSON. The paper justifies the strict zero-reward penalty by noting that "our RL training is preceded by a cold-start SFT phase that ensures the model is already familiar with the required output format" — format errors at the RL stage represent policy degradation that should be strictly discouraged.

Handling overlength rollouts. Rollouts that exceed the token limit (32K context length) or the tool-call budget (40 calls) are also assigned a reward of $0$ . The paper reports a critical practical finding: "we observed in preliminary experiments that directly optimizing on these rollouts can lead to policy collapse." However, these rollouts are not simply discarded — they "contribute to the advantage normalization (i.e., computing the group's mean and standard deviation) [but] are excluded from the final loss computation." This means that an overlength rollout still affects the relative advantages of other rollouts in its group (by shifting the group statistics), but its own tokens do not contribute to the policy gradient. The rationale: excluding them entirely from the group statistics would artificially inflate the advantages of well-formed rollouts (by removing low-reward competitors), while including them in the loss would encourage the policy to produce more overlength rollouts — a delicate balance that the chosen strategy navigates.

What happens in all-wrong groups under standard GRPO vs. E-GRPO: this is one of the paper's key arguments for the method's value. Under standard GRPO, if all $G$ rollouts in a group are incorrect, every rollout receives $R_i = 0$ . The group mean is 0 and the standard deviation is 0, making the advantage $\hat{A}_{i,j}$ undefined (division by zero) — standard GRPO provides no gradient in this case. Under E-GRPO, even in an all-wrong group, the rewards will vary: rollouts that matched more entities receive $\alpha \cdot \hat{\gamma}_i > 0$ while complete failures receive 0. The group now has a positive mean and non-zero variance, producing well-defined advantages that encourage the policy to shift toward the higher-match-rate behaviors. This addresses a fundamental failure mode of outcome-based rewards: when the policy is not yet capable of producing correct answers on a question, standard GRPO cannot learn from that question at all, while E-GRPO can still learn to improve entity-matching behavior.

Advantage Computation and Policy Optimization Objective

With the entity-aware rewards computed, the remaining machinery is structurally identical to standard GRPO — the rewards feed into advantage computation, which feeds into the policy gradient. However, the paper adopts several practical modifications from prior work (specifically DAPO, Yu et al., 2025) that are important for training stability.

Group-relative advantage. For each rollout $i$ and each token position $j$ within that rollout, the advantage is computed as:

$\hat{A}_{i,j} = \frac{R_i - \text{mean}(\{R_k\}_{k=1}^G)}{\text{std}(\{R_k\}_{k=1}^G)}$

What it computes: the advantage for token $j$ in rollout $i$ is the standardized deviation of rollout $i$ 's reward from the group mean. A positive advantage means rollout $i$ performed better than the group average; a negative advantage means it performed worse. Crucially, the advantage is constant across all tokens in the rollout — every token in a good rollout receives the same positive advantage, every token in a bad rollout receives the same negative advantage. This is the standard GRPO choice (no per-token credit assignment within a rollout).

Why use the group mean and standard deviation rather than a learned value function: group-relative advantage eliminates the need for a separately trained critic network, which is a significant practical simplification. The group serves as a mini-batch baseline — by comparing rollouts against their peers on the same question, the advantage captures question-relative performance: a rollout that achieves $\alpha \cdot 0.3$ is evaluated against other rollouts on the same question, not against an absolute threshold. This is particularly important for E-GRPO because the entity match rate's absolute magnitude depends on question difficulty — a 50% match rate on a very hard question is impressive, while the same 50% on an easy question is mediocre. The group-relative comparison automatically adjusts for this.

Why standard deviation normalization rather than raw reward differences: dividing by the standard deviation ensures that the advantage magnitude is adaptive to the group's reward spread. When the group has high variance (some rollouts are much better than others), the advantages are scaled down, preventing any single rollout from dominating the gradient. When variance is low (all rollouts performed similarly), the advantages are magnified, ensuring the policy still receives a meaningful learning signal even when the absolute reward differences are small. This is the standard justification from GRPO, but it gains additional importance under E-GRPO because the entity-aware rewards create smaller reward spreads than binary rewards — without standardization, the gradient signal would be attenuated.

GRPO policy objective. The policy is optimized by maximizing the following objective over the training distribution:

$J(\theta) = \mathbb{E}_{(q, gt) \sim \mathcal{D}, \{H^{(i)}\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}} \left[ \frac{1}{\sum_{i=1}^G |H^{(i)}|} \sum_{i=1}^G \sum_{j=1}^{|H^{(i)}|} \min\left( r_{i,j}(\theta) \hat{A}_{i,j}, \text{clip}\left(r_{i,j}(\theta), 1 - \varepsilon_{\text{low}}, 1 + \varepsilon_{\text{high}}\right) \hat{A}_{i,j} \right) \right]$

where:

$\mathcal{D}$ is the distribution of training questions with their ground-truth answers,
$G$ is the number of rollouts per question (group size, $G = 8$ in the paper's experiments),
$H^{(i)}$ is the $i$ -th rollout with length $|H^{(i)}|$ tokens,
$r_{i,j}(\theta) = \frac{\pi_\theta(H^{(i)}_j \mid q, H^{(i)}_{j-1})}{\pi_{\theta_{\text{old}}}(H^{(i)}_j \mid q, H^{(i)}_{j-1})}$ is the per-token importance sampling ratio — the ratio of the new policy's probability to the old policy's probability for token $j$ ,
$\hat{A}_{i,j}$ is the group-relative advantage computed from the entity-aware rewards,
$\varepsilon_{\text{low}}$ and $\varepsilon_{\text{high}}$ are the clipping parameters (asymmetric, see below),
The outer expectation is over questions sampled from the training distribution and rollouts sampled from the old policy $\pi_{\theta_{\text{old}}}$ .

What it computes: this is a clipped surrogate objective — the standard PPO/GRPO machinery. For each token, the contribution to the objective is the importance-weighted advantage, clipped to prevent the policy from changing too much in a single update. The $\min$ operation ensures that the objective is pessimistic: when the advantage is positive (good token), it clips the ratio to prevent the policy from increasing the token's probability beyond $1 + \varepsilon_{\text{high}}$ ; when the advantage is negative (bad token), it clips the ratio to prevent the policy from decreasing the token's probability beyond $1 - \varepsilon_{\text{low}}$ .

Modifications from DAPO (Yu et al., 2025): the paper adopts two specific modifications from the DAPO framework that are described in Section 3.2 under "Implementation Details":

KL-free objective: the standard GRPO objective includes a KL-divergence penalty term $-\beta \cdot \text{KL}(\pi_\theta \parallel \pi_{\text{ref}})$ that regularizes the policy toward a reference model (typically the SFT checkpoint). Following DAPO, E-GRPO removes this term entirely. The justification is that the clipping mechanism alone provides sufficient regularization against catastrophic policy drift, and removing the KL penalty simplifies the objective and eliminates the $\beta$ hyperparameter.
Clip-higher method (asymmetric clipping): the standard PPO/GRPO objective uses symmetric clipping bounds $\varepsilon$ (e.g., clipping to $[1 - \varepsilon, 1 + \varepsilon]$ ). DAPO proposes increasing the upper clipping bound to $\varepsilon_{\text{high}} > \varepsilon_{\text{low}}$ , which "better encourage[s] policy exploration" by allowing the policy to increase probabilities for good tokens more aggressively than it decreases probabilities for bad tokens. The paper states that this asymmetric clipping is applied but does not specify the exact values of $\varepsilon_{\text{low}}$ and $\varepsilon_{\text{high}}$ in the main text (they are likely specified in Appendix C, which covers hyperparameters).

Why the objective is structurally identical to standard GRPO but produces different gradients: the objective function $J(\theta)$ does not contain any entity-aware terms — the entity information enters only through the rewards $R_i$ that are used to compute the advantages $\hat{A}_{i,j}$ . This is a deliberate design choice that keeps the method simple: the only change to the training pipeline is in the reward computation step. The rest of the GRPO machinery (importance sampling, clipping, advantage normalization) operates identically regardless of whether the rewards are binary or entity-aware. This means E-GRPO can be implemented as a drop-in replacement for the reward function in any existing GRPO training codebase, without modifying the optimizer, the model architecture, or the rollout generation procedure.

Total training objective behavior: during training, for each question in a batch, the old policy generates $G = 8$ rollouts. The entity matcher computes $\hat{\gamma}_i$ for each. Rewards are assigned. If the group contains a correct rollout (reward 1) and some near-misses (rewards 0.1–0.3), the advantages will be positive for the correct rollout and negative for the near-misses, but the near-misses with higher entity match rates will have less negative advantages than complete failures — they are penalized, but less severely. The gradient update will push the policy away from complete-failure behaviors more strongly than away from near-miss behaviors, preserving the partially correct reasoning patterns that the near-misses exhibited. This is the mechanism by which E-GRPO avoids "re-learning steps it had already mastered," as the paper claims in the introduction.

Construction of Ground-Truth Entity Sets from Synthetic Data

The entity-aware reward signal depends entirely on having access to ground-truth entity sets for each training question. The paper describes how these sets are obtained from the two synthetic data generation methods used in the experiments (ASearcher and SailorFog-QA), emphasizing that the entities are already produced by the generation pipeline and simply need to be retained rather than discarded.

ASearcher entity construction (Appendix B.1). The ASearcher method (Gao et al., 2025) constructs complex questions through an iterative process of entity-focused operations applied to a seed question. The two key operations are:

Injection: replacing a named entity with descriptive facts about that entity. For example, in Figure 1 (left), "Leonardo" is injected — the name is removed and replaced with the descriptive phrase "the actor who won the Academy Award for Best Actor for the film 'The Revenant'."
Fuzzing: substituting a specific entity with a more ambiguous, general description. The paper gives the example of "Titanic" being replaced with the description "the 1997 film starring Leonardo."

At each step of the iterative construction, an entity is selected and either injected or fuzzed. The paper states that the ground-truth entity set is simply "all selected and modified entities for a question" — every entity that was manipulated during the question's construction. This is natural because the construction process itself tracks which entities are being transformed; retaining them requires no additional computation.

What this produces: for the example in Figure 1 where the seed question is "Who was the director of the 1997 film starring Leonardo?" and the final question is "Who was the director of the 1997 film starring the actor who won the Academy Award for Best Actor for the film 'The Revenant'?", the ground-truth entity set would include at minimum "Leonardo" (the injected entity) and potentially "Titanic" (the fuzzed entity) and "James Cameron" (the answer entity), though the paper's exact set depends on which entities were explicitly tracked during the multi-step construction.

SailorFog-QA entity construction (Appendix B.1). The SailorFog-QA method (Li et al., 2025b) takes a different approach. It begins by constructing a complex knowledge graph via a random walk from a seed entity, creating intricate couplings between multiple entities. Questions are then generated by sampling subgraphs from this knowledge graph and applying information obfuscation — replacing specific entity attributes with vague descriptions. The paper states that "the node set of the sampled subgraph is regarded as the ground-truth entity set."

What this produces: the entities are the nodes of the knowledge subgraph that was used to generate the question — these are the factual anchors around which the question is constructed. The paper uses SailorFog-QA data for both the cold-start SFT phase (11K samples) and the RL training data in the Web environment (1K samples generated via the SailorFog-QA pipeline).

Entity quality control. The paper argues that entity noise is a minor concern because the question generation process inherently ensures quality: "injected facts strictly adhere to the selected entity, and generated questions are consistently centered around the sampled subgraph, [so] the resulting entity sets are highly precise." Furthermore, the paper notes that even if some irrelevant entities slip through, the normalization scheme provides robustness: "since any irrelevant entity is likely to be missed by all rollouts within a group, it does not change their relative performance and thus does not affect the normalized reward signal." In other words, a spurious entity that no agent ever mentions contributes zero to all match rates, leaving the normalized rates and resulting rewards identical to what they would be without that entity.

Why the entity sets are "free" in terms of additional annotation cost: this is the paper's central efficiency argument. The entities are a structural byproduct of the synthesis process — the same algorithms that construct the questions need to know which entities they are manipulating. Prior work (Dong et al., 2025; Wu et al., 2025a; Li et al., 2025b) used only the final question-answer pairs and discarded the entity trails. The paper argues that "repurposing this 'waste' material" requires no human labeling, no additional model training, and no extra generation steps — it simply requires preserving information that the data pipeline already computed and then performing computationally trivial string matching at training time.

Robust matching extension (Appendix H). Recognizing that exact string matching is brittle to natural language variations like abbreviations, synonyms, or spelling differences, the paper explores a "Robust Matching" strategy. This involves prompting an advanced LLM to generate 5-10 plausible variations for each ground-truth entity (accounting for synonyms, abbreviations, spelling variations, and formality), then performing exact string matching against this expanded candidate set. The prompt (provided in Appendix H) requests variations "in a valid JSON list format like ['phrasing 1', 'phrasing 2', ...]" to enable programmatic extraction. The results (Appendix H, Figure 7) show that robust matching produces a stronger correlation with correctness — the density peak at match rate 1.0 is higher for correct samples, and the distribution for incorrect samples is more spread out. Training with robust matching yields slight performance gains on three of four deep research benchmarks (Appendix H, Table 4), but the main experiments use simple exact matching throughout for its computational efficiency.

Training Pipeline and Hyperparameters

The paper's training pipeline has two phases: a cold-start supervised fine-tuning (SFT) phase that teaches the model the agentic format and basic search behaviors, followed by the RL phase where E-GRPO (or the GRPO baseline) is applied.

Cold-start SFT (Section 4.1, Appendix C). The base models (Qwen2.5-7B-Instruct or Qwen3-30B-A3B-Instruct-2507) are fine-tuned on 11K samples from SailorFog-QA. The SFT hyperparameters (Appendix C) are:

Batch size: 32
Learning rate: $5 \times 10^{-6}$ with cosine decay
Linear warmup: from $1 \times 10^{-10}$ for 0.2 epochs
Total training: approximately 4.8 epochs

The paper justifies the SFT phase as following Dong et al. (2025): it "mitigates reward collapse and ensures the model understands the agentic format before RL." The format is critical because during RL, format errors are penalized with a zero reward, so the model must already be proficient at producing correctly formatted outputs before the RL phase begins.

RL training data. For the RL phase, two distinct 1K-sample datasets are generated, one for each training environment:

Local environment: data synthesized using the ASearcher method over a Wikipedia 2024 corpus. The agent's search and visit tools are simulated via information retrieval (dense passage retrieval following Karpukhin et al., 2020) over this local corpus.
Web environment: data generated using the SailorFog-QA pipeline. The agent interacts with the live web using Google Search and the Jina API for page fetching.

The paper emphasizes that "both methods are anchored in entities from Wikipedia despite the distinct environments they use" — the entity sets are derived from Wikipedia entities in both cases, ensuring that the entity matching mechanism is applicable regardless of the specific generation method. For both datasets, "all ground-truth entities generated during the synthesis process [are retained] to enable E-GRPO."

RL hyperparameters (Appendix C):

Context length: 32K tokens
Tool call budget: 40 calls (rollouts exceeding this are treated as overlength errors)
Training batch size: 64
PPO batch size: 64 ("exactly on policy optimization" — meaning the entire batch is generated from the current policy before each update, with no replay buffer)
Group size $G$ : 8 (8 rollouts per question)
Learning rate: $2 \times 10^{-6}$
Training duration: approximately 5 epochs (80 steps in the main experiments, extended to 120 steps in Appendix G)
Entity matching weight $\alpha$ : 0.3 (default, determined by ablation in Figure 4)

Inference hyperparameters (Appendix C):

Temperature: 0.6
Top-p: 0.95
Tool call budget: 40 (same as training)

Model configurations. The paper trains models in three configurations:

Local-7B: Qwen2.5-7B-Instruct, trained and evaluated in the Local environment
Web-7B: Qwen2.5-7B-Instruct, trained and evaluated in the Web environment
Web-30B: Qwen3-30B-A3B-Instruct-2507 (a Mixture-of-Experts model with 30B total parameters, 3B active), trained and evaluated in the Web environment

For each configuration, three training variants are produced: SFT-only, GRPO (after SFT), and E-GRPO (after SFT). This enables controlled comparison of the algorithmic enhancement (GRPO vs. E-GRPO) as well as the benefit of RL over SFT alone.

Answer evaluation for reward computation. During training, the correctness of a rollout's final answer is determined by extracting the text within <answer> and </answer> tags and comparing it against the ground-truth answer. The paper uses Qwen2.5-72B-Instruct as an LLM judge for this comparison — this is the same judge used for evaluation on the test benchmarks, ensuring consistency between training signal and evaluation metric.

Why only 1K RL samples: the paper explicitly states that the goal is "to validate the effectiveness of E-GRPO at the algorithmic level, not merely to pursue state-of-the-art performance" and that "limited data [is used] to ensure training efficiency while still enabling performance comparison." This is methodologically significant because it suggests that the observed gains from E-GRPO (e.g., +4.0 points over SFT on Local QA benchmarks, +2.8 points over GRPO) are achieved with a relatively small RL dataset — the entity-aware reward may provide particular benefit in low-data regimes where every training sample's learning signal matters more.

Critical Design Decisions and Their Justifications

Several design choices in E-GRPO are non-obvious and have important consequences for the method's effectiveness. The paper discusses these explicitly, often in the Appendix.

Decision 1: Using exact string matching rather than LLM-based semantic matching. The justification has two components (Appendix B.2). Efficiency: matching entities in long reasoning traces using an LLM would introduce "significant computational latency, impeding the throughput of the RL training loop." The method must be applied to every rollout in every training batch — for a batch of 64 rollouts with up to 40 thinking steps each, an LLM-based matcher would be a substantial bottleneck. Robustness against reward hacking: "advanced LLMs can perform semantic matching, [but] they are also more susceptible to exploitation by the policy model." The paper reports observing reward-hacking behavior where "the agent learned to extend its thoughts with verbose, superficially relevant phrases that, while not containing the correct entities, would mislead the LLM judge into erroneously assigning partial credit." Exact matching provides a hard, non-gameable signal — the agent either produces the entity string or it does not.

Decision 2: Matching entities only in thoughts, not in the full trajectory. As discussed in the entity matching section above, the justification is that trajectory-level matching would grant unearned credit for information the agent retrieved but failed to internalize. The paper's empirical analysis (Appendix B.2, Figure 5) confirms that trajectory-level matching produces significantly more "false positive" high-match-rate incorrect rollouts, making it a noisier supervision signal. The thought-only matching ensures that the reward reflects the agent's understanding, not merely its retrieval success.

Decision 3: Group-level normalization by the maximum rather than by some absolute standard. The core tradeoff is between stability and calibration. Absolute normalization (e.g., dividing by the total number of entities) would ensure that rewards are directly comparable across questions — a 50% match rate always means half the entities were found. However, this would make the reward signal highly dependent on question difficulty and the policy's current capabilities. Early in training, when the policy rarely matches any entities, absolute rewards would be uniformly near zero, providing little differentiation between rollouts. Max-normalization ensures that the best rollout in each group always gets $\hat{\gamma} = 1.0$ , guaranteeing that every group (except those with zero matches) provides a meaningful positive signal for at least one behavior. This sacrifices some cross-question comparability for within-question discriminability, which is the right tradeoff for GRPO's group-relative advantage computation.

Decision 4: Giving correct answers a reward of 1 regardless of entity match rate. The alternative — scaling the reward of correct answers by entity match rate — would penalize correct answers that arrived at the solution through reasoning paths that didn't explicitly mention the tracked entities in the thoughts. This could discourage valid problem-solving strategies and create a perverse incentive for the agent to "name-check" entities even when unnecessary. The paper does not explicitly discuss this alternative, but the choice is well-motivated by the goal of using entity matching to differentiate failures, not to evaluate successes.

Decision 5: Removing the KL-divergence penalty from GRPO. Following DAPO, the paper argues that "the clipping mechanism alone provides sufficient regularization." This is a pragmatic choice for search agent training specifically: the trajectories are long and diverse, making the KL divergence between the policy and a fixed reference model potentially very large. A KL penalty term could dominate the objective and prevent the policy from exploring behaviors sufficiently different from the SFT checkpoint — behaviors that might be necessary for successful web search (e.g., aggressive query reformulation, multi-page browsing strategies).

Decision 6: Excluding overlength rollouts from the loss but including them in group statistics. This is a nuanced handling that reflects empirical observations. The paper states that including overlength rollouts in the loss "can lead to policy collapse" — presumably because the policy learns to produce very long, low-quality rollouts that consume the tool-call budget without making progress, a form of reward hacking. However, excluding them from the group statistics entirely would artificially inflate the advantages of properly-formatted rollouts by removing low-reward competitors from the normalization. The chosen strategy — include in statistics, exclude from loss — ensures that overlength rollouts still exert downward pressure on the group mean (making it harder for other rollouts to achieve positive advantages) while preventing the policy from directly optimizing for overlength behavior.

4. Key Insights and Innovations

Innovation 1: Reframing Discarded Data Byproducts as Supervision Signal

The paper's most intellectually distinctive move is not the algorithm itself but the inversion of perspective it represents: recognizing that structural byproducts of synthetic data generation — entities that are systematically tracked and then discarded — are not waste material but rather an untapped, computationally free source of fine-grained supervision. This is a reframing of the relationship between data synthesis and policy optimization that prior work had not considered.

Before E-GRPO, the dominant assumption across the search agent training literature was that synthetic data pipelines produce exactly one useful output: (question, answer) pairs. Methods like ASearcher (Gao et al., 2025), SailorFog-QA (Li et al., 2025b), WebWalker (Wu et al., 2025b), and WebDancer (Wu et al., 2025a) all invest substantial effort in constructing complex questions through entity manipulation — injection, fuzzing, knowledge graph walks, subgraph sampling — and then discard the very entity trails that gave the questions their structure. Prior post-training work (Dong et al., 2025; Wu et al., 2025a; Li et al., 2025b) exclusively used the final question-answer pairs. The field treated entity tracking as a means to an end (generating complex questions) rather than as a valuable output in its own right.

E-GRPO challenges this framing by asking: what information does the synthesis pipeline already compute that we are throwing away? The answer — ground-truth entity sets — turns out to be precisely the intermediate supervision that could differentiate near-misses from complete failures in RL training. The paper's empirical validation of this insight (Section 3.1, Figure 1 right) is itself a contribution: by quantifying the correlation between entity match rate and answer correctness across 2,426 questions, demonstrating a 4-to-1 ratio of questions where correct rollouts match more entities than incorrect ones, and revealing the bimodal distribution of match rates for incorrect rollouts, the paper establishes that entity matching is not just intuitively sensible but empirically diagnostic of reasoning quality.

This is a fundamental rather than incremental contribution because it opens up an entire category of supervision signals that had been invisible to prior work. It suggests a general design principle — mine the structural byproducts of your data synthesis pipeline for reward components — that extends beyond the specific entity-and-string-matching implementation in this paper. The same logic could apply to other synthesis methods: logical subgoals from compositional question generators, intermediate reasoning steps from chain-of-thought synthesis, or dependency relations from multi-hop question constructors. The paper does not explore these extensions, but the conceptual move of treating "waste" from data generation as supervision is what enables them.

The significance is reinforced by the method's cost profile. Process Reward Models (Fan et al., 2025; Anonymous, 2025) and tree-based search (Yang et al., 2025; Hou et al., 2025) also provide fine-grained supervision, but at the expense of substantial additional computation, annotation, or model training. E-GRPO's entity-aware reward requires none of these — it is computed by exact string matching against entity sets that the data pipeline already produced. The paper emphasizes this explicitly (Section 5): "requiring no additional annotation, model training, or complex sampling." This is not merely an implementation convenience; it is a category difference in how dense rewards are obtained. Rather than building new infrastructure to generate supervision (PRMs, search trees), E-GRPO repurposes supervision that already exists but was being discarded. This makes fine-grained rewards accessible in a domain — open-ended web search — where prior dense-reward approaches were economically infeasible.

Innovation 2: The "Near-Miss" as a Diagnosed and Addressed Failure Mode in GRPO Training

The paper introduces and operationalizes the concept of a near-miss in GRPO-based search agent training — a rollout that contains substantially correct intermediate reasoning but produces a wrong final answer — and demonstrates that standard GRPO's outcome-based reward is structurally incapable of distinguishing these from complete failures. This diagnostic contribution is separable from the specific solution (entity-aware rewards): even without E-GRPO, the paper's analysis in Section 3.1 provides the field with a language and an empirical methodology for identifying when reward sparsity is actively harming learning.

Prior work on reward sparsity in LLM training (Qian et al., 2025; Deng et al., 2025) had identified the problem in abstract terms — sparse rewards slow learning, complicate credit assignment — but had not empirically characterized its manifestation in search agent training or quantified the information loss. The GRPO variant literature (DAPO by Yu et al., 2025; ARPO by Dong et al., 2025; GMPO by Zhao et al., 2025; Reinforce++ by Hu, 2025) focused on improving how gradients are computed from the available rewards — better clipping, better exploration, better normalization — but left the information content of the rewards themselves unchanged. These methods assume that the only problem with outcome-based rewards is their sparsity, not their inability to capture intermediate quality.

E-GRPO's contribution is to show that sparsity is not the only problem — uniformity of penalty is equally damaging. The paper identifies a specific failure mode: in an all-wrong group under standard GRPO, every rollout receives reward 0, the group statistics collapse (mean = 0, std = 0), and no gradient is produced. Even when the group contains a mix of near-misses and complete failures, the gradient treats them identically. The entity match rate analysis (Figure 1, bottom right) provides direct evidence that this uniformity destroys useful signal: a substantial fraction of incorrect rollouts have mid-to-high entity match rates (0.4–0.9), representing trajectories where the agent executed most of the reasoning correctly but made a final error. Standard GRPO cannot capitalize on this.

This is an incremental contribution to the diagnosis of RL training failures — it does not propose a new optimization algorithm — but it is foundational because it changes what practitioners should look for when GRPO performance plateaus. The paper's framework suggests that the bottleneck may not be the optimizer or the policy architecture, but rather the expressiveness of the reward function — a dimension that the GRPO variant literature had largely neglected.

The practical significance is underscored by the training dynamics in Figure 3 (right panel). The entity match rate and training accuracy rise in tandem for both GRPO and E-GRPO, confirming that entity matching is a valid proxy objective. But in standard GRPO, the policy must discover this correlation on its own through the sparse outcome signal — a slow and noisy process. E-GRPO makes the correlation explicit in the reward, accelerating the discovery. The 2.8-point average improvement over GRPO on QA benchmarks (Table 1, Local environment) and the 6.8-point Pass@3 gain on GAIA (Table 2) are the downstream consequences of this improved credit assignment.

Innovation 3: Establishing That Process-Level Supervision Can Be Derived from Data Structure Rather Than External Verifiers

The paper makes a methodological contribution that sits between the two dominant paradigms for addressing reward sparsity. On one side, Process Reward Models (PRMs) provide dense, step-level feedback but require training a separate verifier — expensive and brittle for open-ended domains. On the other side, outcome-based rewards are cheap but sparse. E-GRPO carves out a third category: process-level supervision derived from structural properties of the training data itself, requiring no additional models, no human annotation, and no complex sampling.

This is significant because it challenges an implicit assumption in the RL-for-LLMs literature: that moving beyond outcome-based rewards necessarily means building more sophisticated evaluation infrastructure (PRMs, learned critics, tree search). The paper shows that for entity-centric synthetic data — which is the dominant paradigm for search agent training — the raw materials for process-level supervision already exist and simply need to be preserved and exploited.

The evidence for this category distinction is in what E-GRPO does not require, compared to alternatives:

No separate verifier model (vs. PRM approaches: Fan et al., 2025; Anonymous, 2025; Zhang et al., 2025b)
No Monte Carlo rollouts or tree construction at training time (vs. TreeRPO: Yang et al., 2025; TreeRL: Hou et al., 2025)
No modification to the GRPO objective or advantage estimation (vs. DAPO, ARPO, GMPO)
No additional training data annotation (entity sets are a byproduct of synthesis)

The only change is in the reward computation step, which adds trivial string-matching overhead. This makes the method unusually portable — it can be dropped into any existing GRPO training pipeline that uses entity-centric synthetic data, as the paper demonstrates by applying it to both the ASearcher and SailorFog-QA data pipelines with different model architectures (dense 7B, MoE 30B) and different training environments (local, web).

The ablation study on α (Figure 4) provides important nuance: the entity-aware reward is not a free lunch. Performance peaks at α = 0.3 and degrades at α = 0.5, confirming that the entity-matching signal must be balanced against the primary correctness objective. An excessively strong entity bonus distracts the model from answer accuracy — it learns to name-check entities without synthesizing them into correct conclusions. This finding reframes the design problem from "how do we add process supervision" to "how do we balance process and outcome supervision," which is a more general and productive framing for future work.

Innovation 4: Efficiency Gains as a Byproduct of Better Credit Assignment, Not an Explicit Objective

One of the paper's most surprising results is that E-GRPO not only improves accuracy but also produces agents that use fewer tool calls — a form of emergent efficiency that arises from better credit assignment rather than from any explicit efficiency objective. Section 4.3 (Figure 3, middle panel) shows that E-GRPO rollouts consistently use fewer tool calls per step throughout training compared to GRPO. The paper attributes this to the entity-aware reward "guiding the agent towards more direct and informative solution steps" — by rewarding the discovery of key entities, the policy learns to search more efficiently rather than exploring exhaustively.

This matters because search agent efficiency is a significant practical concern: each tool call (search engine query, web page visit) adds latency and computational cost. An agent that achieves the same accuracy with 20% fewer tool calls is substantially more deployable. The fact that this efficiency emerges from the reward design rather than from an explicit length penalty or budget constraint is methodologically interesting — it suggests that the entity-aware reward is providing a richer training signal that enables the policy to discover shorter solution paths on its own, without needing to be explicitly told to be efficient.

The paper does not overclaim this result — it is presented as an observed benefit in the training dynamics (Section 4.3) and mentioned in the abstract and conclusion — but it deserves recognition as a distinctive finding because it demonstrates that better credit assignment can have effects beyond accuracy. Standard GRPO's outcome-based reward provides no incentive for efficiency: a correct answer achieved in 30 tool calls receives the same reward as one achieved in 5. The entity-aware reward implicitly favors efficiency because matching entities early in the trajectory produces high match rates with fewer steps, which the partial-credit mechanism rewards. This is not an explicitly designed property of E-GRPO; it is an emergent consequence of the reward structure.

The case study in Appendix D provides qualitative evidence for this mechanism. The E-GRPO agent solves a complex multi-entity question (identifying explorer Karl Weyprecht) in 5 rounds, directly pursuing the key entities (Tegetthoff, International Polar Year, Founder's Medal). The GRPO agent takes 6 rounds, meanders through less relevant searches, and ultimately fails by misidentifying the explorer as Julius von Payer. The E-GRPO trajectory is both more accurate and more direct — the entity-matching incentive appears to have shaped a reasoning policy that pursues the factual backbone of the question rather than exploring tangentially.

Innovation 5: Robustness of Entity-Based Supervision Across Synthetic Data Generation Methods and Training Environments

The paper provides a robustness demonstration that entity-aware rewards are not tied to a specific data synthesis method or training environment. This is more than an engineering validation — it addresses a legitimate concern that a method exploiting structural properties of training data might be fragile to changes in how that data is generated.

The evidence spans two qualitatively different data synthesis paradigms:

ASearcher (Gao et al., 2025): iterative entity injection and fuzzing, where entities are explicitly tracked as they are manipulated. The entity set is the collection of all entities that were transformed during question construction.
SailorFog-QA (Li et al., 2025b): knowledge graph random walks and subgraph sampling, where entities are the nodes of the sampled subgraph. The entity set is derived from graph structure rather than from a transformation history.

These represent two different mechanisms for generating entity sets — one from an edit trail, one from a graph structure — yet both produce entity sets that work effectively with E-GRPO. The paper also spans two training environments (a closed-world local knowledge base using Wikipedia retrieval, and the live open web using Google Search and Jina) and two model architectures (dense Qwen2.5-7B, MoE Qwen3-30B). The consistent gains across all configurations (Tables 1 and 2) suggest that the entity-as-supervision principle is robust to implementation details of both the data pipeline and the training setup.

This is pragmatically important because the search agent field uses diverse data generation methods — ASearcher and SailorFog-QA are just two examples from a broader landscape that includes WebShaper (Tao et al., 2025), SimpleDeepSearcher (Sun et al., 2025), and WebSailor-V2 (Li et al., 2025a). The paper's demonstration that entity-aware rewards work across different paradigms lowers the barrier for adoption: practitioners can integrate E-GRPO with whatever entity-centric synthesis method they already use, as long as they preserve the ground-truth entity sets during generation.

The cross-environment result is particularly striking: the Local-7B-E-GRPO model, trained entirely in a simulated Wikipedia environment, achieves the highest average accuracy (67.8) when evaluated in the live Web environment on QA benchmarks (Table 1, second block), outperforming the Web-trained GRPO counterpart (66.2) and larger models like Search-R1-32B (62.5). This suggests that the entity-aware reward produces policies that transfer better — perhaps because entity matching is a generalizable skill that does not depend on the specific search engine or web corpus used during training. The paper does not explicitly analyze this transfer property, but the result is evident in the evaluation tables and merits recognition as a distinctive empirical finding.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation spans 11 benchmarks total, organized into two categories. For standard question-answering (QA), three single-hop datasets are used — Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (TQ) (Joshi et al., 2017), and PopQA (Mallen et al., 2022) — and four multi-hop datasets — 2WikiMultiHopQA (2Wiki.) (Ho et al., 2020), HotpotQA (HQA) (Yang et al., 2018), Bamboogle (Bamb.) (Press et al., 2022), and MuSiQue (Musi.) (Trivedi et al., 2022). For deep research, four challenging benchmarks are used: GAIA (Mialon et al., 2023), BrowseComp (Wei et al., 2025), BrowseComp-ZH (Zhou et al., 2025), and xbench-DeepSearch (xbench-DS) (Xbench-Team, 2025). Following ASearcher (Gao et al., 2025), 1000 sampled instances from the validation sets of HQA, 2Wiki., and Musi. are used; for GAIA, the 103-sample text-only validation subset is used; for all other benchmarks, the full test sets are used.
Base model(s). Experiments use Qwen2.5-7B-Instruct (Yang et al., 2024) and Qwen3-30B-A3B-Instruct-2507 (Team, 2025a), covering both a dense 7B model and a Mixture-of-Experts architecture with 30B total parameters (3B active). The choice of two architectures at different scales tests whether the algorithmic benefit generalizes beyond a single model configuration. The paper explicitly states that the goal is "to validate the effectiveness of E-GRPO at the algorithmic level, not merely to pursue state-of-the-art performance" (Section 4.1), so model scale is chosen to demonstrate the method works rather than to maximize absolute performance.
Metrics. The primary metric throughout is Pass@1 — the fraction of test instances for which a single sampled rollout from the trained policy produces the correct final answer, as determined by an LLM judge. For deep research benchmarks, Pass@3 is additionally reported, measuring whether at least one of three independently sampled rollouts is correct. Model answers are extracted from text enclosed in <answer> and </answer> tags and evaluated for correctness using Qwen2.5-72B-Instruct under an LLM-as-Judge setting (Section 4.1). The average number of tool calls per rollout is tracked during training to assess policy efficiency (Section 4.3, Figure 3 middle panel).
Baselines. The primary controlled baseline is the same model trained with standard GRPO (Shao et al., 2024) — identical architecture, data, and training pipeline, differing only in the reward function (binary outcome-based vs. entity-aware). An SFT-only baseline (cold-start supervised fine-tuning without any RL) provides a lower bound showing the benefit of RL itself. For external comparison on QA benchmarks, reference agents include R1-Searcher-7B (Song et al., 2025), DeepResearcher-7B (Zheng et al., 2025), Search-R1-32B (Jin et al., 2025), Simple-DS-QwQ (Sun et al., 2025), and ASearcher-14B (Gao et al., 2025). For deep research benchmarks, both advanced proprietary models (OpenAI-o3, Claude-4-Sonnet, Kimi-K2, DeepSeek-V3.1) and open-source agents with ≤32B parameters (R1-Searcher-7B, WebThinker-RL, WebDancer-QwQ, WebSailor-7B/32B) are included (Li et al., 2025d; Wu et al., 2025a; Li et al., 2025b). These reference agents provide context for E-GRPO's performance level but are not its primary comparison — the GRPO counterpart is the key baseline for isolating the algorithmic contribution.
Generation budget / compute accounting. Generation budget during RL training is 8 rollouts per question (group size G = 8), with each rollout constrained to a maximum of 40 tool calls and 32K context tokens (Appendix C). Rollouts exceeding these limits are assigned a reward of 0 and excluded from the loss computation (Section 3.2). At evaluation time, the same constraints apply, with inference using temperature 0.6 and top-p 0.95 (Appendix C). The paper does not report wall-clock training time or total FLOPs, focusing instead on final accuracy comparisons at fixed training budgets.
Cross-validation / statistical protocol. The paper does not employ cross-validation for evaluation — all metrics are reported on standard test/validation splits. However, the training procedure involves generating fresh on-policy rollouts for each RL update (PPO batch size equals training batch size, "exactly on policy optimization"), meaning the policy is evaluated on data it has not previously trained on at each step. No confidence intervals or statistical significance tests are reported.

Main Quantitative Results

The results are organized across three evaluation axes: QA benchmarks with a closed-world local knowledge base, QA benchmarks in the live web environment, and deep research benchmarks.

Standard QA Benchmarks: Local Environment

Results for models trained and evaluated within the controlled Local environment (Wikipedia 2024 corpus, simulated search/visit via dense passage retrieval) are presented in the top block of Table 1.

Headline numbers. Local-7B-E-GRPO achieves an average Pass@1 of 64.2 across the 7 QA benchmarks, representing a 2.8-point absolute improvement over Local-7B-GRPO (61.4) and a 4.0-point improvement over Local-7B-SFT (60.2). This establishes that the entity-aware reward provides benefit beyond both the SFT baseline and standard GRPO in the same training environment.

Per-benchmark consistency. E-GRPO outperforms GRPO on 6 of the 7 individual benchmarks, with particularly large margins on 2Wiki. (79.6 vs. 75.1, a 4.5-point gain), HQA (69.0 vs. 65.1, a 3.9-point gain), Bamb. (78.4 vs. 74.4, a 4.0-point gain), NQ (55.8 vs. 51.5, a 4.3-point gain), and TQ (83.9 vs. 82.0, a 1.9-point gain). The only benchmark where E-GRPO does not improve over GRPO is PopQA (50.2 vs. 50.4), where the difference is negligible and both outperform the SFT baseline (49.6). The broad consistency across diverse question types (single-hop factual, multi-hop compositional) supports the claim that entity-aware rewards provide general benefit rather than being specialized to particular question structures.

Magnitude of RL benefit. The GRPO baseline itself provides only a modest improvement over SFT (61.4 vs. 60.2, a 1.2-point gain), suggesting that outcome-based RL on 1K training samples yields limited returns — consistent with the paper's argument that reward sparsity constrains learning from small RL datasets. E-GRPO's larger gain (4.0 points over SFT) indicates that the entity-aware reward makes more effective use of the same limited RL training data.

Standard QA Benchmarks: Web Environment

The second block of Table 1 presents results for models evaluated in the live web environment (Google Search, Jina for page fetching). Notably, the Local-trained models are evaluated in this web environment without any web-specific RL training — this is a transfer setting that tests the robustness of learned policies.

Headline numbers. Local-7B-E-GRPO achieves an average Pass@1 of 67.8 in the web environment, outperforming Local-7B-GRPO (66.2) by 1.6 points and the SFT baseline (64.7) by 3.1 points. This is the highest average among all compared agents, including the larger Web-trained models and external baselines like Search-R1-32B (62.5) and ASearcher-14B (65.6).

Transfer robustness. The fact that Local-trained E-GRPO outperforms its Web-trained GRPO counterpart (66.2) — which received RL training directly in the web environment — is striking. It suggests that the entity-aware reward produces policies that transfer better from simulated to live environments, perhaps because entity-matching is a generalizable skill independent of the specific search infrastructure used during training. The paper does not explicitly analyze this transfer property, but the result is evident in comparing the local and web blocks of Table 1: the Local-7B models generally perform better in the Web setting than in the Local setting on most benchmarks (e.g., Local-7B-E-GRPO moves from 64.2 to 67.8 average), possibly because the live web provides richer information than the Wikipedia-only local corpus.

Comparison with larger models. Local-7B-E-GRPO (67.8 average) outperforms Search-R1-32B (62.5), a model with over 4× the parameters, and matches or exceeds ASearcher-14B (65.6), a model with twice the parameters trained specifically for search. While the paper does not do a controlled FLOPs-matched comparison as the reference example paper does, these results qualitatively demonstrate that better training algorithms can compensate for model scale on this benchmark suite.

Per-benchmark patterns. E-GRPO outperforms GRPO on 4 of the 7 benchmarks in the web setting, with notable gains on 2Wiki. (80.4 vs. 77.2, a 3.2-point gain), NQ (59.1 vs. 55.9, a 3.2-point gain), and TQ (90.4 vs. 89.3, a 1.1-point gain). On HQA, Bamb., and Musi., E-GRPO matches GRPO closely, while on PopQA the two are nearly identical (50.2 vs. 50.1). No benchmark shows a meaningful reversal where GRPO substantially outperforms E-GRPO.

Deep Research Benchmarks

Table 2 presents results on the four challenging deep research benchmarks: GAIA, BrowseComp, BrowseComp-ZH, and xbench-DeepSearch. These benchmarks test sophisticated information synthesis capabilities beyond standard QA, often requiring multi-source verification, temporal reasoning, and handling of ambiguous or conflicting information.

7B model results. Web-7B-E-GRPO achieves Pass@1 scores of 36.9 on GAIA (vs. 33.0 for GRPO, a 3.9-point gain), 9.3 on BrowseComp (vs. 6.3, a 3.0-point gain), 18.1 on BrowseComp-ZH (vs. 17.5, a 0.6-point gain), and 42.0 on xbench-DS (vs. 40.7, a 1.3-point gain). These improvements are consistent across all four benchmarks, though the margin varies.

Pass@3 results reveal the strongest algorithmic differentiation. The Pass@3 metric — whether any of three rollouts is correct — shows substantially larger gaps between E-GRPO and GRPO than Pass@1. On GAIA, Web-7B-E-GRPO achieves 51.5 Pass@3 vs. GRPO's 44.7 — a 6.8-point gain. On BrowseComp, the gap is 16.1 vs. 11.7 (4.4 points). BrowseComp-ZH shows 32.1 vs. 31.5 (0.6 points). xbench-DS shows 59.0 vs. 56.0 (3.0 points). The paper attributes this pattern to a "key algorithmic difference: GRPO's outcome-based reward tends to refine existing successful strategies, whereas E-GRPO's entity-aware reward explicitly encourages exploring promising but incomplete paths" (Section 4.2). In other words, E-GRPO produces a more diverse set of rollouts — the entity-aware reward incentivizes the policy to pursue different "near-miss" reasoning paths rather than converging narrowly on the single strategy that happened to work in the training group.

30B model results confirm scaling. Web-30B-E-GRPO achieves Pass@1 scores of 48.5 on GAIA (vs. 47.6 for GRPO, a 0.9-point gain), 12.9 on BrowseComp (vs. 12.3, a 0.6-point gain), 26.4 on BrowseComp-ZH (vs. 25.7, a 0.7-point gain), and 46.7 on xbench-DS (vs. 45.3, a 1.4-point gain). The absolute gains are smaller at 30B scale than at 7B scale, which is consistent with the larger model being more capable at baseline, leaving less room for the entity-aware reward to provide marginal benefit. However, the Pass@3 gains remain substantial: GAIA 65.1 vs. 62.1 (3.0 points), BrowseComp 21.0 vs. 18.9 (2.1 points), BrowseComp-ZH 41.2 vs. 38.8 (2.4 points), xbench-DS 66.0 vs. 65.0 (1.0 point).

Positioning against advanced models. Table 2 contextualizes these results against both proprietary giants and open-source competitors. Web-30B-E-GRPO's Pass@1 of 12.9 on BrowseComp surpasses Claude-4-Sonnet (12.2) despite the vast scale difference, and its 26.4 on BrowseComp-ZH outperforms Claude-4-Sonnet (29.1? — the paper reports 29.1; check: Table 2 shows 29.1 for Claude on BrowseComp-ZH, so 26.4 is slightly below, but it outperforms Kimi-K2's 28.8... actually, wait — re-reading Table 2 carefully: BrowseComp-ZH: Claude-4-Sonnet is 29.1, Kimi-K2 is 28.8, Web-30B-E-GRPO is 26.4, so it does not surpass these. The paper claims E-GRPO "narrows the gap with others" which is accurate). On GAIA, Web-30B-E-GRPO at 48.5 is below OpenAI-o3 (70.5) and Claude-4-Sonnet (68.3) but competitive with WebThinker-RL (48.5 identical) and ahead of other open-source agents. These comparisons are informative for positioning but are not the primary evaluation — the controlled comparison remains E-GRPO vs. GRPO at the same model scale.

SFT to GRPO to E-GRPO progression. Across both scales and all four benchmarks, the progression is consistent: SFT → GRPO provides modest gains (e.g., Web-7B-SFT 31.7 → GRPO 33.0 on GAIA, a 1.3-point gain), while GRPO → E-GRPO provides larger gains (33.0 → 36.9, a 3.9-point gain). This pattern reinforces the paper's core argument that standard GRPO's outcome-based reward leaves substantial learning signal on the table, and that entity-aware rewards recover it.

Training Dynamics Analysis

Figure 3 presents detailed training curves comparing E-GRPO and GRPO during the 30B model's RL training in the Web environment. These are not final evaluation metrics but rather intermediate training statistics plotted over 80 training steps.

Training accuracy (Figure 3, left panel). E-GRPO consistently achieves higher training accuracy than GRPO throughout training, with the gap widening in the middle steps (roughly steps 20–60) before narrowing slightly at the end. E-GRPO's curve shows a "steadier and more pronounced upward trend" than GRPO, which the paper attributes to the "dense, entity-aware reward [providing] a more effective and stable learning signal." This is consistent with the entity-aware reward providing meaningful gradients even when the group contains no correct answers — standard GRPO would produce zero-gradient steps in such cases, while E-GRPO can still update based on relative entity-matching performance.

Tool call efficiency (Figure 3, middle panel). E-GRPO rollouts consistently use fewer tool calls per step than GRPO rollouts. At step 80, E-GRPO averages roughly 11.0–11.5 tool calls while GRPO averages roughly 12.0–12.5 — a reduction of approximately 1 call per rollout, or roughly 8–10%. The paper attributes this to the entity-aware reward "guiding the agent towards more direct and informative solution steps" (Section 4.3). This is an emergent property — no explicit efficiency penalty was included in the reward — and demonstrates that better credit assignment can yield more resource-efficient policies as a byproduct.

Entity match rate vs. accuracy correlation (Figure 3, right panel). Both E-GRPO and GRPO show entity match rate curves (orange and green, respectively) that rise in tandem with training accuracy curves (purple and blue). For GRPO, the entity match rate improves even though the reward function does not explicitly incentivize it — the policy discovers entity matching as a useful intermediate behavior through the sparse outcome signal alone. However, E-GRPO's entity match rate (orange) is consistently above GRPO's (green), confirming that the explicit entity-matching bonus accelerates this discovery. The paper argues this "directly translates into superior final answer accuracy" because entity matching is causally related to correctness (as established in Section 3.1).

Extended training (Appendix G, Figure 6). When training is extended to 120 steps, both methods' performance gradually converges, but E-GRPO "maintains a consistent lead over GRPO" and "continues to use fewer steps on average." This suggests the entity-aware reward provides sustained benefit rather than just accelerating early learning, though the diminishing gap at later steps indicates that standard GRPO eventually catches up to some degree.

Ablation Studies and Robustness Checks

Entity matching weight α (Figure 4): The hyperparameter α controls the balance between outcome-based reward and entity-matching bonus. Sweeping α ∈ {0.0, 0.1, 0.3, 0.5} across all four deep research benchmarks (30B model, Web environment) reveals a consistent inverted-U pattern. At α = 0.0 (equivalent to standard GRPO), performance is lowest on all benchmarks: GAIA 33.0, BrowseComp 6.3, BrowseComp-ZH 17.5, xbench-DS 40.7. Performance improves at α = 0.1, peaks at α = 0.3 (GAIA 36.9, BrowseComp 9.3, BrowseComp-ZH 18.1, xbench-DS 42.0), and then degrades at α = 0.5 (GAIA 34.0, BrowseComp 6.3, BrowseComp-ZH 15.1, xbench-DS 38.3). The α = 0.5 degradation is substantial on BrowseComp-ZH (falling below the GRPO baseline) and xbench-DS (also below baseline), while GAIA and BrowseComp at α = 0.5 are still above baseline but well below the α = 0.3 peak. This ablation is critical because it validates that entity matching is beneficial only when properly balanced — an excessively strong entity-matching signal can distract the model from the primary objective of answer correctness.

Decaying α vs. fixed α (Appendix F, Table 3): Training with α linearly decaying from 0.3 to 0.0 over the first 60 steps (of 80 total steps) produces Pass@1 scores of GAIA 48.2, BrowseComp 12.8, BrowseComp-ZH 26.2, xbench-DS 47.3. Compared to fixed α = 0.3 (48.5, 12.9, 26.4, 46.7), there is "no clear or consistent advantage for the decaying alpha strategy." This suggests that the entity-matching bonus remains useful throughout training rather than only being beneficial early — had the bonus been useful only for initial exploration, the decaying schedule would have matched or outperformed the fixed schedule. The paper speculates that "applying the decaying schedule over longer training horizons could be more impactful" but leaves this to future work.

Exact string matching vs. robust matching (Appendix H, Figure 7 and Table 4): The robust matching strategy — prompting an LLM to generate 5–10 variations per entity before exact string matching — produces a stronger correlation between match rate and correctness (Figure 7, compare left and right panels: the correct-sample density peak at 1.0 is higher, the incorrect-sample peak at 0.0 is reduced, and the incorrect distribution is more spread out). Training with robust matching yields Pass@1 scores of GAIA 49.2, BrowseComp 12.9, BrowseComp-ZH 26.8, xbench-DS 47.0, compared to exact matching's 48.5, 12.9, 26.4, 46.7 — small improvements on three of four benchmarks (Table 4). The paper views this as promising but uses simple exact matching for the main experiments due to its computational efficiency.

Model scale ablation (7B vs. 30B): The experiments at two model scales (Tables 1 and 2) serve as an implicit ablation on whether entity-aware rewards benefit different model capacities. The consistent gains at both 7B and 30B scales, across both dense and MoE architectures, suggest the method is robust to model architecture and capacity. The larger absolute gains at 7B than 30B (e.g., +3.9 vs. +0.9 Pass@1 on GAIA) are consistent with the intuition that smaller models benefit more from improved credit assignment, since they have less capacity to discover effective intermediate behaviors through sparse outcome signals alone.

Training environment ablation (Local vs. Web): Training models in both the Local and Web environments, and evaluating Local-trained models in both environments, provides implicit ablation on sensitivity to training environment fidelity. The fact that Local-7B-E-GRPO outperforms Web-7B-GRPO when both are evaluated in the web (67.8 vs. 66.2, Table 1) suggests the entity-aware reward provides benefit that partially compensates for training environment mismatch. This is a non-trivial robustness check: the entity-aware reward works even when entities are derived from a Wikipedia-based synthesis pipeline (ASearcher) but evaluation uses the live web.

Training data budget (implicit): All RL experiments use only 1K training samples. The consistent improvements over both SFT and GRPO baselines with this small RL dataset suggest that entity-aware rewards are particularly valuable in low-data regimes, where every rollout's gradient signal matters more. The paper does not run ablations varying the RL dataset size — this would be informative for understanding whether the benefit diminishes with more data — but the choice to use only 1K samples is deliberate and the results speak to that regime.

Critical Assessment

The experiments provide convincing evidence for the paper's core claim — that augmenting GRPO's outcome-based reward with an entity-match-rate bonus consistently improves search agent performance — but the scope of the evidence has several important boundary conditions that should be carefully delineated.

Does E-GRPO genuinely solve the "near-miss" problem, or does it simply improve training efficiency? The paper frames E-GRPO as addressing a specific failure mode: standard GRPO's inability to distinguish near-misses from complete failures. The Pass@3 results (Table 2) provide the strongest evidence for this mechanism — E-GRPO's larger Pass@3 gains relative to Pass@1 gains are consistent with the entity-aware reward encouraging exploration of diverse partially-correct strategies, which increases the probability that at least one of multiple rollouts succeeds. The training dynamics (Figure 3, right panel) showing E-GRPO's higher entity match rate confirm that the reward is indeed incentivizing the claimed behavior. However, the experiments do not directly measure whether E-GRPO actually shifts probability mass from complete failures to near-misses (as opposed to simply making both types less likely by improving overall policy quality). A more targeted analysis — measuring the rate of near-miss rollouts (e.g., entity match rate > 0.5 but answer incorrect) in E-GRPO vs. GRPO policies — would strengthen this claim.

The single-family limitation is genuine and unaddressed. All experiments use Qwen models (2.5-7B-Instruct and Qwen3-30B-A3B-Instruct-2507). While the paper spans two architectures (dense and MoE), it does not test with models from other families (e.g., Llama, DeepSeek, Mistral). The entity-matching behavior — how readily the model mentions entities in its reasoning, how it responds to the entity-aware reward — may depend on the base model's pretraining characteristics that are not universal. The paper's claim that Qwen2.5-7B-Instruct is "representative of the capabilities of many contemporary LLMs" (the paper doesn't actually say this explicitly, but Section 4.1 implies it by choosing commonly-used models) is plausible but unverified. Testing with at least one non-Qwen model would substantially strengthen the generality claim.

The entity matching assumption is validated only for entity-centric synthetic data. The paper's method depends on having ground-truth entity sets from the data generation pipeline. The two synthesis methods tested (ASearcher and SailorFog-QA) are representative of the entity-centric paradigm, but the method would not directly transfer to training on naturally-occurring questions (e.g., NQ, TQ) where no entity trail exists. This is not a weakness of the method per se — the paper explicitly scopes its contribution to synthetic data training — but it limits the method's applicability to scenarios where entity-centric synthesis is used. The paper does not discuss whether entity sets could be retroactively constructed for existing QA datasets (e.g., via entity linking), which would broaden the method's reach.

The 1K RL sample budget is both a strength and a limitation. Using only 1K samples demonstrates sample efficiency and positions E-GRPO as particularly valuable in low-data regimes. However, it leaves open the question of whether the benefit persists or diminishes with larger RL datasets. It is possible that with 10K or 100K RL samples, standard GRPO's outcome-based reward provides sufficient signal that the entity-aware bonus becomes redundant. The paper's extended training analysis (Appendix G, 120 steps) shows the E-GRPO advantage persisting but the gap narrowing, which is suggestive but not conclusive — larger training datasets would provide a stronger test of asymptotic behavior.

No confidence intervals or statistical testing are reported. The paper reports point estimates (e.g., average accuracy) without standard errors, confidence intervals, or significance tests. This makes it difficult to assess whether differences of 1-2 percentage points on individual benchmarks are reliable or within sampling noise. The 500-instance MATH test set in the reference example paper is small enough to warrant such caution; the test sets used here vary in size (e.g., 103 for GAIA validation, 1000 sampled for HQA/2Wiki./Musi.), and reporting uncertainties would clarify which gains are statistically meaningful. For BrowseComp and BrowseComp-ZH in particular, where absolute Pass@1 scores are in the single digits and low teens, small absolute differences represent large relative changes, but the statistical reliability of these differences on small test sets is unclear.

Missing ablation: does the entity match rate actually need to be normalized within groups? The paper normalizes entity match rates by the group maximum. An ablation comparing max-normalized vs. raw-rate vs. z-score-normalized entity bonuses would clarify whether the specific normalization scheme matters or whether any 0-to-1 transformation of the match rate would work. The paper's justification for max-normalization (Section 3.4, now covered in prior sections) is reasoned but not empirically validated for this specific method.

Missing ablation: is thought-only matching genuinely better than trajectory-level matching for training? Appendix B.2 shows that trajectory-level matching produces noisier entity match rate distributions (Figure 5), but this is an analysis of the signal quality, not a training ablation. Training with trajectory-level entity matching and comparing the resulting policy quality would directly test whether the thought-only constraint matters for downstream performance or only for analytical cleanliness.

The efficiency claim (fewer tool calls) is observational but causal direction is ambiguous. E-GRPO policies use fewer tool calls (Figure 3, middle panel), but the paper does not establish whether this is because the entity-aware reward directly incentivizes efficiency or because E-GRPO policies are simply better at the task (higher accuracy) and efficient strategies are a byproduct of competence. An ablation comparing E-GRPO to a version with an explicit length penalty added to the reward — or measuring efficiency conditional on correctness — would disentangle these explanations. Without this, the efficiency claim is an interesting observation but not firmly causally attributed to the entity-aware mechanism.

The α ablation suggests a genuine tradeoff that the paper handles well. The inverted-U performance as a function of α (Figure 4) with consistent peaks at α = 0.3 across all four benchmarks is strong evidence that the entity-matching bonus is not a free parameter that can be cranked up arbitrarily. The degradation at α = 0.5 (falling below GRPO baseline on two benchmarks) establishes that the entity-matching objective can distract from the primary objective if over-weighted. This is an honest presentation of the method's sensitivity and provides practical guidance for practitioners. The decaying-α result (Appendix F) showing no advantage over fixed α suggests the entity bonus remains useful throughout training, which is an interesting and non-obvious finding.

Overall assessment: The experiments support the paper's central claim that entity-aware rewards improve GRPO training for search agents. The improvement is consistent across two model scales, two architectures, two training environments (local and web), two data synthesis paradigms, and eleven evaluation benchmarks — a breadth of evidence that makes the core finding robust. The magnitude of improvement is meaningful but not transformative (roughly 1-4 percentage points absolute on most benchmarks, with larger gains on Pass@3 for deep research). The paper's explicit positioning as "algorithmic validation" rather than "state-of-the-art chasing" is appropriate: the experiments demonstrate that the mechanism works and characterize its behavior (α sensitivity, Pass@3 benefits, efficiency improvements) without overclaiming about absolute performance.

The most significant limitations are the single model family (Qwen only), the absence of confidence intervals or significance testing, and the somewhat narrow scope of the robustness checks (no ablation on group size G, no ablation on normalization scheme, no ablation on larger RL datasets). Addressing these would strengthen the paper but does not undermine its core contribution — the entity-as-supervision principle is well-supported by the available evidence, and the paper provides sufficient detail for others to replicate and extend the method.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Makes the Headline Gains an Upper Bound

The assumption or constraint. The entity-aware reward relies on ground-truth entity sets from the synthetic data generation pipeline. While the paper emphasizes that these entities require "zero additional annotation cost" (Section 5) because they are a byproduct of question synthesis, the method's applicability is gated on access to an entity-centric synthesis pipeline. For training on naturally-occurring questions (e.g., NQ, TQ, or any dataset not generated through entity manipulation), no entity trail exists and E-GRPO cannot be directly applied. The paper does not discuss whether entity sets could be retroactively constructed via entity linking or similar techniques for existing datasets. As the authors acknowledge (Appendix B.1), the method works because "the question generation process ensures question quality" — this is an assumption about the training data provenance that does not hold for most QA datasets.

The consequence. Any deployment that does not use entity-centric synthetic data for RL training cannot benefit from E-GRPO without first developing an entity extraction pipeline. This limits the method's reach to a specific — albeit important — paradigm in search agent training. Furthermore, the quality of the entity set depends on the synthesis method's fidelity: if the generation pipeline produces noisy or incomplete entity sets (e.g., missing key intermediate entities, including spuriously relevant ones), the entity-aware reward signal degrades. The paper's robustness argument — that irrelevant entities are "likely to be missed by all rollouts within a group" and therefore "does not change their relative performance" (Appendix B.1) — only holds for entities that are universally missed, not for entities that are randomly matched by some rollouts due to chance co-occurrence in reasoning text.

What evidence exists in the paper. The paper tests two entity-centric synthesis methods (ASearcher and SailorFog-QA, Appendix B.1) and demonstrates E-GRPO works with both, but does not evaluate on non-entity-centric data or experiment with degraded entity sets. The claim that entity noise is robust to the normalization scheme is reasoned but not empirically tested — there is no ablation where random noise entities are injected into the ground-truth sets to measure how reward signal quality degrades with entity set precision.

Mitigation status. Not addressed. The paper treats entity set quality as given by the synthesis pipeline and does not propose methods for constructing entity sets for arbitrary QA data. The robust matching extension (Appendix H) addresses entity matching brittleness (synonyms, abbreviations) but not entity set availability — it assumes ground-truth entities exist and focuses on making matching more flexible. Extending E-GRPO to non-synthetic data would require either automatic entity extraction (e.g., via named entity recognition on the question and answer, or via entity linking to a knowledge base) or a different class of process-level supervision, neither of which is explored.

The Method Has Only Been Validated on a Single Model Family, Leaving Generality Unproven

The assumption or constraint. All experiments use models from the Qwen family: Qwen2.5-7B-Instruct and Qwen3-30B-A3B-Instruct-2507 (Section 4.1). While the paper spans two architectures (dense 7B and Mixture-of-Experts 30B) and two scales, it does not test with models from other families such as Llama, DeepSeek, Mistral, or Gemma. The paper states its goal is "to validate the effectiveness of E-GRPO at the algorithmic level" (Section 4.1), implicitly claiming the method should generalize, but provides no cross-family evidence.

The consequence. The entity-matching behavior that E-GRPO incentivizes — explicitly mentioning ground-truth entities in thinking blocks — may depend on model-specific characteristics that are not universal. Different model families may exhibit different tendencies in how they verbalize intermediate reasoning: some may mention entities frequently in their chain-of-thought, while others may reason more implicitly or use different naming conventions. A model that rarely mentions entities by name in its reasoning would receive near-zero entity match rates regardless of reasoning quality, rendering the entity-aware reward signal uninformative. Conversely, a model that verbosely name-checks entities without actually using them in reasoning could exploit the entity bonus, a form of reward hacking that the paper observes with LLM-based matching (Appendix B.2) but does not test for with exact matching across model families.

What evidence exists in the paper. The paper provides no cross-family experiments. The analysis in Section 3.1 — establishing the correlation between entity match rate and accuracy — is conducted on a Qwen-based agent (WebSailor-7B, which is built on Qwen). It is unknown whether a Llama-based or DeepSeek-based agent would exhibit the same strong positive correlation (the 4-to-1 ratio of questions where correct rollouts match more entities, Figure 1 right). Without this correlation, the entity match rate would not be a valid proxy for reasoning quality, and E-GRPO would provide no benefit or might actually harm training by introducing a misleading reward component.

Mitigation status. Not addressed. The paper does not acknowledge model-family generality as a limitation or suggest future cross-family validation. Given that the Qwen family is among the most widely used for Chinese and English language tasks, and that the paper's WebSailor baselines are Qwen-based, the choice is practically reasonable, but the algorithmic validity claim requires testing beyond a single family to be fully substantiated.

The Pass@3 Story is Compelling but Statistical Reliability is Unquantified Throughout

The assumption or constraint. The paper reports all results as point estimates — average Pass@1 or Pass@3 accuracy — without standard errors, confidence intervals, or statistical significance tests. Test set sizes vary substantially: GAIA uses 103 text-only validation samples, BrowseComp and BrowseComp-ZH use their full test sets (sizes not explicitly stated in the paper, but BrowseComp is known to contain 1,266 instances and BrowseComp-ZH is presumably comparable), HQA/2Wiki./Musi. use 1,000 sampled instances each, and other benchmarks vary. On the smallest test set (GAIA, 103 samples), a 3.9-point Pass@1 difference between E-GRPO and GRPO (36.9 vs. 33.0 for 7B models, Table 2) represents approximately 4 additional correct answers — a difference that could arise from sampling variance alone.

The consequence. The paper's headline Pass@3 gains — which form a key part of the algorithmic argument (Section 4.2: "GRPO's outcome-based reward tends to refine existing successful strategies, whereas E-GRPO's entity-aware reward explicitly encourages exploring promising but incomplete paths") — are impossible to assess for statistical reliability. On GAIA with 103 test samples and Pass@3 (each instance gets 3 rollouts, scored as correct if any succeeds), the 6.8-point gap between Web-7B-E-GRPO (51.5) and Web-7B-GRPO (44.7) represents roughly 7 additional instances where at least one rollout succeeded. Without confidence intervals, it is unclear whether this difference would replicate on a different test split or is within the expected variance of the Pass@3 estimator given the small sample size. Similarly, on individual QA benchmarks where E-GRPO's margin over GRPO is 1-3 points (e.g., Bamboogle 85.6 vs. 82.4, a 3.2-point gap in Table 1, Web environment), the practical significance is ambiguous without error bounds.

What evidence exists in the paper. None. The paper provides no uncertainty quantification. The two-fold cross-validation protocol from the reference example paper (used for compute-optimal strategy selection) is not employed here. The training dynamics curves (Figure 3) show clear separation between E-GRPO and GRPO during training, which provides some qualitative confidence that the difference is systematic, but the final evaluation numbers are single-point estimates.

Mitigation status. Not addressed. The paper does not mention the lack of statistical testing or propose future work to address it. For a paper positioned as "algorithmic validation," reporting confidence intervals — particularly on the smaller test sets — would substantially strengthen the evidence that observed differences are reliable and not artifacts of test-set idiosyncrasies.

The Efficiency Benefit is Correlational, Not Causal, and May Not Transfer to Deployment

The assumption or constraint. The paper observes that E-GRPO policies use fewer tool calls than GRPO policies during training (Figure 3, middle panel: approximately 11.0–11.5 vs. 12.0–12.5 tool calls at step 80) and attributes this to the entity-aware reward "rewarding the discovery of key entities, which guides the agent towards more direct and informative solution steps" (Section 4.3). However, no experiment establishes the causal direction: E-GRPO policies are also more accurate, and the efficiency improvement could simply be a byproduct of higher competence — a policy that solves tasks correctly may naturally use fewer tool calls because it converges faster, not because the entity-aware reward directly incentivizes conciseness.

The consequence. If the efficiency improvement is solely a byproduct of higher accuracy, it may not generalize to deployment scenarios where accuracy gains are smaller (e.g., on out-of-distribution questions, or when the model has saturated). Furthermore, efficiency during training (where tool calls are simulated or involve real API costs) may not reflect efficiency at deployment scale: a 1-tool-call average reduction (~8–10%) is meaningful for high-volume applications, but if it only manifests when the policy is already performing well, it may not reduce costs on the hardest queries where most tool calls are spent. The paper does not measure whether efficiency gains are uniform across difficulty levels or concentrated on easy questions where the policy already succeeds.

What evidence exists in the paper. The tool call curves in Figure 3 (middle panel) show a gap throughout training, including early steps where accuracy is still low (step 0–20, accuracy around 45% for both methods). This suggests the efficiency difference is not purely an artifact of higher terminal accuracy. However, the curves also show that both methods' tool call counts decrease over training (from ~12.5–13.5 to ~11.0–12.5), which is consistent with efficiency emerging from general policy improvement rather than from the entity reward specifically. No ablation measures efficiency conditional on correctness (e.g., average tool calls for correct rollouts only vs. incorrect rollouts only) or tests whether adding an explicit efficiency penalty to standard GRPO would achieve similar results. The case study in Appendix D shows the E-GRPO agent solving a problem in 5 rounds vs. the GRPO agent's 6 rounds (and failing), which is illustrative but anecdotal.

Mitigation status. Not addressed. The paper presents efficiency as an observed benefit but does not isolate its cause. A targeted analysis — measuring tool call counts stratified by rollout correctness, or ablating with an explicit length penalty in GRPO — would clarify whether entity-aware rewards provide efficiency benefits beyond what general competence improvement would produce.

Exact String Matching Creates a Tension Between Signal Quality and Robustness

The assumption or constraint. E-GRPO's entity matching uses exact string matching against ground-truth entity strings (Section 3.2). The paper explicitly justifies this choice: it is computationally trivial compared to LLM-based matching, and it prevents reward hacking — "the rigidity of exact matching acts as a safeguard against exploitation" (Appendix B.2). However, this rigidity creates a systematic failure mode: the agent may identify and reason about an entity without using its exact string form (e.g., referring to "the Austrian admiral" instead of "Wilhelm von Tegetthoff," or "the polar initiative" instead of "International Polar Year"), producing a zero match rate for factually correct reasoning.

The consequence. E-GRPO systematically undercounts entity matching when the agent uses paraphrases, abbreviations, or coreference rather than the exact entity string. This introduces noise into the reward signal: some factually correct reasoning trajectories receive lower entity bonuses than they deserve, while others that happen to use the exact string receive credit. The paper's analysis in Section 3.1 — which establishes the correlation between entity match rate and accuracy — was conducted using exact matching, meaning the correlation already accounts for this undercounting. The fact that the correlation is strong despite this noise suggests that exact matching is "good enough," but it leaves open the question of how much stronger the signal could be with more flexible matching.

The robust matching extension (Appendix H) partially addresses this by expanding the entity set to include variations generated by an LLM. However, this introduces a new dependency — an external LLM call for variation generation — that the paper does not account for in its cost claims. Furthermore, robust matching was tested as a post-hoc analysis, not used in the main experiments, and its performance gains were modest (Table 4: +0.7 Pass@1 on GAIA, +0.4 on BrowseComp-ZH, +0.3 on xbench-DS, 0.0 on BrowseComp). The paper states that "the continued refinement of this strategy [is] a promising direction for future work" but does not resolve the core tension.

What evidence exists in the paper. The robust matching analysis in Appendix H provides a direct comparison: robust matching produces a stronger correlation between match rate and correctness (Figure 7), with the correct-sample density peak at 1.0 being higher and the incorrect-sample peak at 0.0 being reduced. This demonstrates that exact matching is missing some genuine entity matches — the correlation improves when matching is more flexible. The training results with robust matching (Table 4) show small improvements, confirming that better matching yields slightly better policies, but the paper does not explore why the downstream gains are modest despite the improved signal quality.

Mitigation status. Partially addressed by the robust matching extension, but this extension is presented as preliminary future work rather than a resolved solution. The computational cost of robust matching (LLM calls for variation generation per entity) is not characterized, and the paper does not recommend it as the default approach, instead using exact matching for all main experiments. The fundamental tension — between cost/simplicity (exact matching) and signal quality/coverage (flexible matching) — remains unresolved, and practitioners must choose based on their tolerance for matching noise vs. matching cost.

7. Implications and Future Directions

How This Work Changes the Landscape

E-GRPO introduces a new category of supervision signal for RL-based search agent training — one derived from structural byproducts of the data synthesis pipeline rather than from external verifiers, human annotation, or complex sampling. This is not a paradigm shift in the sense of overturning established theory (the GRPO machinery remains unchanged), but it represents a methodological reframing that changes what practitioners should attend to when designing reward functions: the information needed for fine-grained credit assignment may already exist in the training pipeline and simply needs to be preserved rather than discarded.

The field's approach to reward sparsity in search agent training prior to this work fell into two camps: either accept sparse outcome-based rewards and refine the optimizer to cope with them (the GRPO variant literature — DAPO, ARPO, GMPO), or invest in building dense supervision infrastructure (Process Reward Models, tree-based search). E-GRPO demonstrates a third path: repurpose information that the data synthesis process already computes. This is conceptually distinct from both camps. Unlike the GRPO variants, E-GRPO changes what information is in the reward rather than how the gradient is computed from it. Unlike PRMs, E-GRPO requires no additional model training, no human annotation, and no per-step correctness judgments — the only computational cost is exact string matching against a pre-existing entity set.

This reframing matters because it reconciles a tension in the search agent training literature. Entity-centric synthetic data generation methods (ASearcher, SailorFog-QA, WebWalker, WebDancer, WebShaper) invest substantial effort in constructing complex question-answer pairs through entity manipulation — tracking entities during injection, fuzzing, and knowledge graph construction — and then systematically discard those entities, using only the final (question, answer) pairs for training. Prior post-training work (Dong et al., 2025; Wu et al., 2025a; Li et al., 2025b) operated entirely within this discard paradigm. E-GRPO shows that this is not just wasteful but actively harmful — the discarded entities are precisely the signal that could distinguish near-misses from complete failures in RL training. The paper's core finding that entity match rate strongly correlates with answer correctness (Figure 1, right: correct rollouts match more entities in ~80% of questions) makes this diagnosis concrete: prior work was throwing away a validated proxy for reasoning quality.

The practical consequence is that research attention should shift toward mining structural byproducts of data synthesis for reward components, rather than exclusively toward building more sophisticated verifiers or optimizers. The paper's α ablation (Figure 4) provides important calibration: entity-based supervision is beneficial but must be balanced against the primary correctness objective (peak at α = 0.3, degradation at α = 0.5). This establishes that the design problem is not "replace outcome rewards with process rewards" but rather "augment outcome rewards with the right weight on process signals" — a more nuanced and productive framing.

Several research directions become more attractive as a consequence of this work:

Lightweight dense reward design: using entity trails, sub-goal structures, dependency graphs, or other synthesis byproducts as reward components, rather than building full PRMs. The paper's demonstration that exact string matching suffices (with robust matching as an optional enhancement) lowers the barrier to entry — practitioners can experiment with process-level rewards without the infrastructure investment that PRMs require.
Data synthesis with RL in mind: designing synthesis pipelines to produce richer structural byproducts (beyond entity sets) that are explicitly intended to serve as reward components, rather than treating them as incidental artifacts. The paper's observation that "both methods are anchored in entities from Wikipedia" (Section 4.1) suggests that entity trails are a natural byproduct of fact-centered synthesis, but synthesis methods could be designed to track additional structures (logical subgoals, information-seeking sub-queries, verification checkpoints) specifically for reward use.
Sample-efficient RL for search agents: the paper's results with only 1K RL training samples (Section 4.1) — a 4.0-point improvement over SFT for E-GRPO vs. 1.2 points for standard GRPO (Table 1, Local) — suggest that entity-aware rewards are particularly valuable in low-data regimes. This opens the door to RL training with much smaller datasets than previously assumed necessary, which is practically important given the cost of generating on-policy search agent rollouts.

Conversely, some directions become less attractive:

GRPO variants that only modify the optimizer: the paper's evidence that the fundamental bottleneck is reward information content, not gradient computation, suggests diminishing returns from further refinements to clipping, normalization, or exploration schedules within the outcome-based reward paradigm. Methods like DAPO, ARPO, and GMPO provide value, but the paper's consistent finding that E-GRPO outperforms standard GRPO across all configurations (Tables 1 and 2) indicates that enriching the reward signal yields larger gains than optimizing how a sparse reward is processed.
PRMs for open-ended web search: while the paper does not directly compare against PRMs, its argument that PRMs are "ill-suited for the open-ended nature of web search" due to annotation cost and distributional complexity (Section 1) is strengthened by the demonstration that a much cheaper alternative — entity matching — provides meaningful process-level supervision. This does not make PRMs obsolete (they remain valuable in mathematics and code where step-level correctness is well-defined), but it suggests that the search agent community should prioritize data-derived supervision over verifier training.

The paper also provides a diagnostic framework that changes how training failures should be analyzed. The near-miss concept — an incorrect rollout with high entity match rate — operationalizes a previously vague intuition ("the model was almost right") into a measurable quantity. Practitioners evaluating their own GRPO training can compute entity match rate distributions for incorrect rollouts to determine whether reward sparsity is harming their specific setup, and if so, whether entity-aware rewards would help. The bimodal distribution in Figure 1 (bottom right) — a peak at 0.0 for complete failures and a spread across 0.2–0.9 for near-misses — provides a reference pattern against which other training runs can be compared.

Follow-Up Research This Work Enables

Cross-model-family validation of the entity-match-rate/correctness correlation. The paper establishes the correlation between entity match rate and accuracy using only Qwen-based models (Section 3.1, WebSailor-7B). A strong follow-up would replicate the analysis in Section 3.1 (specifically the per-question comparison of average entity match rates for correct vs. incorrect rollouts, and the distribution of normalized match rates) on at least three model families — Llama-3, DeepSeek, and Mistral — using the same SailorFog-QA subset and the same 8-rollout-per-question protocol. The key question is whether the 4-to-1 ratio (1,939 questions where correct rollouts match more entities vs. 487 where they do not) holds across families, or whether some models exhibit fundamentally different entity-verbalization behavior in their chain-of-thought. If a model family shows a weak or absent correlation, E-GRPO would be ineffective or harmful for that family, defining a critical boundary condition. If the correlation is universal, it strengthens the paper's generality claim and suggests entity-aware rewards can be adopted without per-family validation.

Retroactive entity set construction for non-synthetic QA datasets. E-GRPO's current applicability is gated on access to entity-centric synthetic data — the method cannot be used with naturally-occurring QA datasets like NQ, TQ, or HotpotQA. A follow-up study would test whether entity sets can be retroactively constructed via automated entity linking. The experiment: take the training sets of NQ, TQ, and HotpotQA; run a standard entity linker (e.g., a Wikipedia-based linker) on each question to extract mentioned entities; use the linked entities as a proxy ground-truth entity set; train with E-GRPO and compare against standard GRPO. The key metric is whether the retroactively-constructed entity sets produce a meaningful entity-match-rate/correctness correlation (measured via the same analysis as Figure 1, right) and whether E-GRPO training with these sets yields accuracy improvements comparable to those observed with synthetic data (2-4 points on QA benchmarks, Table 1). A negative result — no correlation or no training improvement — would clarify that the entity trail must be generated during synthesis (where it is guaranteed to be causally related to the answer) rather than extracted post-hoc (where the relationship is correlational and potentially noisy). A positive result would dramatically broaden E-GRPO's applicability.

Asymptotic behavior of E-GRPO with larger RL datasets. The paper uses only 1K RL training samples and observes E-GRPO outperforming GRPO, with the gap narrowing but persisting through extended training (Appendix G, 120 steps). A systematic study would train both GRPO and E-GRPO with RL dataset sizes of {1K, 5K, 10K, 50K, 100K} samples on the same base model and data synthesis pipeline (ASearcher over Wikipedia 2024), measuring final Pass@1 and Pass@3 on the 7-benchmark QA suite (Table 1, Local) and the 4 deep research benchmarks (Table 2). The key question is whether E-GRPO's advantage is primarily a sample-efficiency benefit (large gains at small data sizes, diminishing to zero at large sizes) or a fundamental improvement (persistent gap even as data scales). The extended training curves in Appendix G (Figure 6) hint at convergence, but training steps ≠ dataset size — the paper uses 1K samples for 80–120 steps, meaning each sample is seen multiple times. A proper scaling study would fix the number of training epochs and vary dataset size. If the gap closes at large data sizes, E-GRPO is best understood as a technique for making small RL datasets viable. If it persists, the entity-aware reward provides information that outcome-based rewards fundamentally cannot recover, regardless of data volume.

Stratified analysis of E-GRPO's efficiency benefit. The paper observes that E-GRPO policies use fewer tool calls (Figure 3, middle panel) but does not establish whether this is a direct effect of the entity-aware reward or a byproduct of higher accuracy. A follow-up would measure average tool calls stratified by rollout correctness: for both E-GRPO and GRPO policies, compute the average number of tool calls for correct rollouts only, for incorrect rollouts only, and overall, across the full test suite. If E-GRPO's efficiency advantage persists even when conditioning on correctness (i.e., E-GRPO correct rollouts use fewer tool calls than GRPO correct rollouts), that would establish a direct efficiency benefit — the entity-aware reward is incentivizing shorter reasoning paths, not just better outcomes. If the advantage disappears when conditioning on correctness, efficiency is purely a byproduct of higher accuracy and may not transfer to out-of-distribution settings where accuracy is lower. Additionally, measuring efficiency stratified by difficulty bin (using the paper's test benchmarks, which span single-hop to complex multi-hop to deep research) would reveal whether the efficiency benefit is uniform or concentrated on easier questions where the entity-matching incentive is most effective at guiding direct reasoning.

Combining entity-aware rewards with explicit negative signals for failure modes. E-GRPO assigns partial credit to near-misses but does not explicitly penalize specific failure modes identified in the paper's failure analysis (Appendix E): distracted querying, information overload from concurrent queries, and broken reasoning coherence. A follow-up would augment the E-GRPO reward with explicit penalty components for these behaviors — for example, a small negative reward for redundant search queries (detected via embedding similarity between queries in the same step), or a penalty for reasoning steps that ignore the stated goal of a previous visit call (detected via semantic similarity between the goal and the subsequent thought). The experiment would compare E-GRPO + failure penalties against vanilla E-GRPO on the deep research benchmarks, measuring both accuracy and the rate of the specific failure modes (via manual annotation of a sample of incorrect trajectories). A positive result would demonstrate that entity-based rewards and failure-mode penalties are complementary — entities provide positive guidance, while failure penalties suppress specific bad behaviors. A negative result (no improvement or degradation) would suggest that the entity reward already implicitly reduces these failure modes by incentivizing focused, entity-driven reasoning.

Entity-aware rewards for non-search agent tasks with structured synthesis. The paper's principle — repurpose structural byproducts of data synthesis as reward components — may generalize beyond search agents. A follow-up would apply the same approach to code generation (where synthesis pipelines often track intermediate sub-goals or test case coverage) or to multi-step planning (where synthesis tracks sub-goal completion). The experiment: for code generation, use an existing synthesis pipeline that generates problems with associated unit tests; treat individual test cases as "entities" and compute a test-pass rate as the entity match analogue; train with E-GRPO-style partial credit (α · test-pass rate for programs that fail some tests) vs. standard GRPO (binary pass/fail); evaluate on HumanEval and MBPP. The key question is whether the entity-as-supervision principle transfers to domains where the "entities" represent different types of intermediate correctness (execution results, constraint satisfaction) rather than factual recall. This would test the generality of the paper's core insight beyond the specific entity-matching implementation.

Practical Applications and Downstream Use Cases

Cost-efficient training of custom search agents from small RL datasets. Organizations that want to train domain-specific search agents — for internal knowledge bases, technical documentation, legal research, or medical literature — typically cannot afford the massive synthetic data generation and RL training pipelines of large AI labs. E-GRPO's strong performance with only 1K RL training samples (Section 4.1: +4.0 points over SFT vs. +1.2 points for GRPO in the Local environment, Table 1) makes it directly applicable to this setting. A team with a modest budget could: (1) generate ~1K domain-specific questions using an entity-centric synthesis method (ASearcher or SailorFog-QA adapted to their corpus), preserving entity trails; (2) train their base model (Qwen, Llama, or a fine-tuned variant) with SFT on 10-20K examples; (3) run E-GRPO on the 1K RL samples with a group size of 8. The expected outcome, based on the paper's results, is a 2-4 percentage point accuracy improvement over standard GRPO on their domain-specific evaluation set, achieved with a training budget that makes RL viable where it previously was not. The key practical enabler is that entity preservation during synthesis adds zero marginal cost — the pipeline already computes entity sets; the team simply needs to not discard them.

Improving the diversity of agent outputs in self-improvement and data generation pipelines. When using search agents to generate training data for themselves (as in STaR, ReST^EM, or rejection sampling fine-tuning), output diversity matters: the generated dataset should cover a range of reasoning paths to avoid training the next iteration on a narrow distribution. E-GRPO's Pass@3 advantage over GRPO (Table 2: 6.8-point gain on GAIA for 7B models vs. only 3.9-point Pass@1 gain) indicates that the entity-aware reward encourages the policy to explore diverse partially-correct strategies rather than converging on a single successful pattern. For a self-improvement pipeline, this means using E-GRPO instead of GRPO during the RL phase of each iteration should produce a more diverse set of correct and near-correct rollouts for distillation into the next SFT phase. Concretely: a team running a search agent self-improvement loop could replace their GRPO training step with E-GRPO, use the same entity sets they already generate during question synthesis, and expect that the collected rollouts for distillation will contain a wider variety of successful reasoning strategies, reducing the risk of policy collapse in later iterations. The paper's failure analysis (Appendix E) identifies specific failure modes (distracted querying, information overload) that E-GRPO partially mitigates, further improving the quality of generated training data.

Deploying search agents with stricter latency or cost budgets. The paper's finding that E-GRPO policies use fewer tool calls than GRPO policies (Figure 3, middle: ~1 call reduction, or ~8-10% at step 80) has direct operational implications. Each tool call in a deployed search agent incurs latency (search engine API round-trip, web page fetch) and cost (API fees for search and browsing services). For a high-volume deployment processing millions of queries per day, an 8-10% reduction in tool calls translates to proportional savings in both latency and infrastructure cost. A deployment team could adopt E-GRPO with α = 0.3 (the paper's recommended setting) and expect that the trained agent will, on average, use fewer search queries and page visits to achieve the same or better accuracy than a GRPO-trained agent. The paper does not report inference-time latency, but the tool call reduction is measured during training and is consistent across training steps, suggesting it is a stable property of the learned policy rather than a transient training artifact. The practical caveat is that the paper does not establish whether the efficiency benefit holds on the hardest queries (where most tool calls are spent); deployment teams should benchmark E-GRPO vs. GRPO on their specific query distribution before committing.

When to Prefer This Method

The paper does not articulate an explicit tradeoff against named alternatives in the form of a decision rule. It positions E-GRPO as a strict improvement over standard GRPO when entity-centric synthetic data is available — the α = 0.0 ablation (equivalent to standard GRPO) is the worst-performing configuration on all four deep research benchmarks (Figure 4), and E-GRPO outperforms GRPO on 22 of 24 benchmark-model combinations across Tables 1 and 2 (with the two exceptions being PopQA where the difference is negligible). The paper's argument is that E-GRPO should be preferred whenever the training pipeline already uses entity-centric synthetic data generation, since the entities are available at zero additional annotation cost and the computational overhead of string matching is trivial. There is no identified scenario where standard GRPO outperforms E-GRPO given the same entity sets.

The paper does not compare E-GRPO against Process Reward Models or tree-based search methods experimentally, so no direct tradeoff can be drawn from the evidence. The paper's argument is that those alternatives are infeasible for web search (Section 1, Section 5), but this is a qualitative claim rather than an empirical finding. A practitioner choosing between E-GRPO and a PRM-based approach would need to weigh the paper's claim of PRM infeasibility against their own annotation budget and domain characteristics — the paper provides no head-to-head data to inform this decision.