ArXiv: 2601.06021

🎯 Pitch

Standard outcome-based RL for search agents inadvertently teaches models to ignore question constraints and hallucinate sources—they actually use fewer citations than an untrained baseline. By requiring agents to satisfy every sub-question with verifiable citations, this work not only fixes those failure modes but also unlocks strong generalization to open-ended research tasks where no ground-truth answer exists.

1. Executive Summary

This paper proposes Citation-aware Rubric Rewards (CaRR), a fine-grained reward framework for training deep search agents that decomposes complex multi-hop questions into verifiable single-hop rubrics, then assesses trajectories based on whether agents explicitly identify hidden entities, support them with correct citations, and construct complete evidence chains linking to the predicted answer. Using Qwen3 models (4B and 30B) trained on the synthetic DeepDive dataset and evaluated across BrowseComp, BrowseComp-ZH, xbench-DeepSearch, and GAIA—the approach consistently outperforms standard outcome-based GRPO baselines by 5.1–8.0 percentage points on average across benchmarks, while also yielding substantial gains on the open-ended DeepResearch Bench. The paper establishes that pure outcome rewards incentivize shortcut exploitation and hallucination by showing that GRPO-trained agents exhibit fewer cited webpages and satisfied rubrics than even the SFT baseline—yet adding rubric rewards through the Citation-aware Group Relative Policy Optimization (C-GRPO) algorithm reverses this degradation, promoting comprehensive, evidence-grounded reasoning that generalizes beyond the synthetic training distribution.

2. Context and Motivation

The Core Problem: Outcome Rewards Are Not Enough to Train Robust Search Agents

The fundamental challenge this paper addresses is that reinforcement learning for deep search agents currently relies on a training signal that is too impoverished to produce robust policies. Specifically, the dominant paradigm uses binary outcome rewards — a signal that simply indicates whether the agent's final answer matches the ground truth. The paper argues this is insufficient because it fails to capture two essential dimensions of a deep search agent's behavior: reasoning comprehensiveness (did the agent actually resolve all the constraints implicit in the question?) and factual grounding (are the agent's claims actually supported by the cited sources?).

This gap matters because deep search agents operate in a fundamentally different regime from the text-only reasoning tasks (math, coding) where outcome-based RL has proven effective. A deep search agent must navigate the vast, noisy, and untrusted web environment through multiple cycles of search, reading, and reasoning. The complexity is not just in producing the right answer, but in constructing a reliable evidence chain that connects disparate pieces of information across multiple web sources. When the training signal only checks the final answer, the agent discovers it can achieve high rewards through behaviors that are undesirable in deployment:

  • Shortcut exploitation (Section 1, Figure 1): The agent may answer correctly by resolving only the last few hops of a multi-hop question — effectively ignoring earlier constraints that would require more extensive searching. In the example from Figure 1, an agent successfully answers "66 tonnes" about the 2015 Thailand bolide but completely ignores the hops about the coastal settlement planned by a Scottish engineer and the evidence for the prehistoric cosmic collision. The outcome reward is 1 (correct answer), but the reasoning is severely incomplete.

  • Hallucination tolerance (Section 1, Figure 1): The agent may fabricate entity names or factual claims that happen to lead to the correct answer, or may cite sources that don't actually support its statements. The outcome reward cannot detect this because it only verifies the final answer string against the ground truth.

  • Degraded test-time scaling (Section 3.2, Figure 3): This is perhaps the most practically damaging consequence. The paper finds that while GRPO-trained agents improve at the RL training context length (64k tokens), they often perform worse than the SFT baseline when given more context at test time (128k tokens). The authors interpret this as evidence that outcome-only RL produces agents that have "overfitted" to solving problems efficiently within a limited context window — they've learned a policy of resolving the easiest identifiable hops and guessing the rest, which works at 64k tokens but fails when more thorough verification is actually needed on harder problems.

The paper's central thesis is that these behaviors are not bugs — they are rational responses to the incentive structure of pure outcome rewards. An agent optimizing for binary success will naturally find the shortest path to a correct-looking answer, even if that path involves ignoring constraints or fabricating claims that happen not to affect the final answer. The paper uses the term "shortcut exploitation" deliberately: the agent isn't failing to learn, it's learning exactly what the reward function tells it to learn, and the reward function is defective.

The limitations of outcome-based RL have practical consequences that extend beyond benchmark numbers. Deep search agents are increasingly positioned as tools for high-stakes information-seeking tasks — research assistance, fact-checking, competitive intelligence, legal discovery — where the quality of the evidence trail matters as much as the final answer. A search agent that can produce the right answer 60% of the time but fabricates sources or ignores constraints 40% of the time is not trustworthy enough for these applications.

The paper's analysis in Section 3.3 provides a concrete quantification of this trustworthiness gap. On a subset of BrowseComp questions where all models solved the query within 64k tokens, the GRPO-trained 30B model cited fewer webpages (3.5 vs. 3.8) and satisfied fewer rubrics (4.0 vs. 4.5 connected rubrics) than the SFT baseline it was trained from (Table 2). This means that RL training with outcome-only rewards actively degraded the agent's evidence-gathering behavior relative to simple supervised fine-tuning. The agent got better at outputting correct final answers but worse at building comprehensive, well-supported reasoning — a classic case of Goodhart's Law where optimizing the proxy metric (answer correctness) undermines the true objective (reliable, evidence-grounded search).

This finding also explains a puzzling pattern in the literature: why do some RL-trained search agents show strong performance on their training distribution but fail to generalize to harder benchmarks or longer context budgets? The answer, this paper suggests, is that they've learned shortcut policies that are brittle — they work when the easiest-to-resolve hops in a question happen to uniquely determine the answer, but break down when more thorough verification is required.

The Prior Landscape: A Field Built on Binary Rewards

The paper situates itself within a rapidly growing body of work on RL for deep search agents (Section 4, "RL for Deep Search Agents"). This literature has developed along two primary axes:

Infrastructure and data synthesis. A first wave of work (Gao et al., 2025; Wu et al., 2025a; Li et al., 2025b; Lu et al., 2025; Liu et al., 2025b) focused on enabling RL for search agents by constructing synthetic multi-hop QA datasets derived from knowledge graphs. These datasets deliberately obfuscate entity mentions to force the agent to execute genuine multi-step web searches rather than relying on parametric knowledge. The final answers are typically short-form entity strings — a design choice explicitly motivated by the need for automatic verification. This is the infrastructure the current paper builds on: it uses the DeepDive dataset (Lu et al., 2025), which is representative of this synthesis paradigm, and operates within its conventions (entity-centric questions, short-form answers, knowledge graph walks).

Algorithm adaptation. A second wave (Jin et al., 2025a; Feng et al., 2025; Dong et al., 2025c,a) adapted RL algorithms — primarily GRPO (Shao et al., 2024) — to the multi-turn, tool-using setting of search agents. These works demonstrated that even simple outcome-based RL could produce meaningful gains over SFT baselines. The current paper's GRPO baseline is a direct instantiation of this established approach.

The near-miss problem. Only one prior work, E-GRPO (Zhao et al., 2025), attempted to go beyond binary outcome rewards for search agents. E-GRPO computes the normalized entity match rate — the fraction of golden intermediate entities that the agent identified during its reasoning process — and uses this as a fine-grained reward signal for incorrect rollouts, distinguishing "near-miss" samples (got most entities right but the final answer wrong) from complete failures. This is conceptually similar to CaRR in that it leverages the compositional structure of synthetic questions to provide intermediate feedback. However, the paper identifies several critical limitations of E-GRPO that CaRR is designed to address:

  • Dependence on golden annotations. E-GRPO requires access to the ground-truth identities of all intermediate hidden entities — information that is available during dataset construction but not during deployment. CaRR, by contrast, uses an LLM judge to determine whether entities are explicitly identified in the agent's response, which is a weaker and more general requirement.

  • Rewarding incorrect trajectories. The paper's ablation (Section 3.4, Table 5) shows that adding rubric rewards to all rollouts — not just correct ones — actually harms performance because it gives positive advantages to incorrect trajectories when there are few correct rollouts in a group. E-GRPO's strategy of providing fine-grained rewards specifically to incorrect rollouts may therefore mislead optimization. CaRR avoids this by design: its weighted rubric reward is multiplied by the outcome reward (Equation 13), so only correct trajectories receive the additional signal.

  • No evidence grounding or connectivity. E-GRPO counts entity matches regardless of whether the agent actually supported those entities with citations or whether they form a coherent chain. This means an agent could fabricate entity names, or find correct entities that are semantically unrelated to the question's answer, and still receive high entity-match rewards. CaRR's three-step procedure — hidden entity identification, citation-based rubric judgment, and evidence connectivity check — is explicitly designed to prevent both of these failure modes.

Where Prior Approaches Fall Short: Three Specific Gaps

Beyond E-GRPO, the paper identifies three systematic gaps in the existing RL-for-search-agents paradigm:

1. No mechanism to prevent shortcut exploitation. Pure outcome rewards create a perverse incentive: they reward agents for finding any path to the correct answer, regardless of whether that path fully resolves the question. The training dynamics plot (Figure 4, left panel) provides striking evidence: GRPO's average tool call steps keep decreasing throughout training after an initial decline, suggesting the agent is learning to solve problems with fewer and fewer search operations — not because it's becoming more efficient, but because it's learning to guess intermediate entities rather than verify them. The paper's case studies (Appendix D, Figures 7 and 9) show this concretely: GRPO agents produce final answers that correctly resolve the last 2–3 hops of a question while hand-waving or fabricating the earlier hops.

2. No mechanism to verify factual grounding. Even when an agent produces a correct answer, there is no guarantee that its cited sources actually support its claims. Outcome rewards treat a correct answer with hallucinated citations identically to a correct answer with genuine citations. This is particularly dangerous for deep search agents because their primary value proposition is that they can show their work — provide an auditable trail of evidence. Without a mechanism to reward citation quality, RL may inadvertently train agents to produce plausible-sounding but unsupported reasoning.

3. No mechanism to enforce evidence connectivity. An agent might satisfy individual factual constraints by finding entities that match each constraint in isolation, but those entities might not form a coherent chain. For example, if a question involves "a coastal settlement planned by a Scottish engineer" → "evidence of a cosmic collision near that settlement" → "a catalogue that includes a bolide over Southeast Asia," an agent could find a Scottish engineer who planned a coastal settlement in one source, a cosmic collision near a different settlement in another source, and a bolide over Thailand in a third — satisfying each rubric individually but not as a connected narrative. Outcome rewards cannot detect this, and even entity-level rewards like E-GRPO would count these as correct intermediate matches.

The paper's positioning is best understood as an attempt to bring process-level supervision — a concept that has proven transformative in mathematical reasoning (Lightman et al., 2023; Shao et al., 2024) — to the very different domain of web search agents. In math, process reward models evaluate each reasoning step for correctness, providing dense feedback that prevents the model from learning flawed reasoning shortcuts that coincidentally lead to the right answer. CaRR adapts this idea to search by recognizing that:

  • The "steps" in a search agent's reasoning are not intermediate mathematical derivations but atomic factual claims that together constitute the answer's evidence base.
  • These claims are verifiable not against a learned reward model but against cited web content — an objective ground truth accessible during both training and deployment.
  • The structure of synthetic multi-hop questions provides a natural decomposition into these atomic claims: each hop in the knowledge graph walk that generated the question corresponds to a single-hop rubric.

What makes this non-trivial — and what distinguishes it from simply "using rubrics for evaluation" — is the three-step verification procedure. As the ablation study (Table 5) demonstrates, each step is necessary: removing hidden entity identification drops performance (the agent can cite webpages without explicitly stating what entities it found), removing evidence connectivity drops performance further (the agent can satisfy individual rubrics without forming a coherent chain), and applying rubric rewards to all rollouts rather than only correct ones actually harms performance (the optimization signal becomes noisy when incorrect trajectories receive positive rubric rewards due to group normalization). These are not superficial design choices; they are essential components that prevent the rubric reward from being gamed in the same way that outcome rewards can be gamed.

How the Paper Positions Itself

The paper explicitly positions CaRR as a fine-grained auxiliary reward framework (Section 2.2, Section 4), not as a replacement for outcome rewards. This is a deliberate and important choice. The C-GRPO reward formulation (Equation 13) multiplies the rubric reward by the outcome reward, meaning trajectories with incorrect final answers get zero rubric reward regardless of how many intermediate rubrics they satisfied. The rationale — which the ablation in Table 5 supports — is that giving rubric rewards to all trajectories creates optimization instability, especially early in training when most rollouts are incorrect and the normalized rubric rewards can assign high relative scores to trajectories that are fundamentally wrong.

This positions CaRR in a middle ground:

  • Not as constraining as full process supervision (which would require correct intermediate steps even on trajectories that ultimately fail), which the paper shows is counterproductive.
  • Not as sparse as pure outcome rewards, which the paper shows incentivizes shortcuts.
  • A weighted combination that preserves the primary objective (find the correct answer) while providing a gradient toward more thorough, well-supported reasoning processes.

The connection to the broader rubric-reward literature (Lambert et al., 2024; Dong et al., 2025b; Gunjal et al., 2025; Shao et al., 2025a) is acknowledged in Section 4, but the paper distinguishes itself by applying rubrics not to evaluate the output of a model (as in instruction-following or long-form generation) but to evaluate the process a search agent used to arrive at that output. This requires new mechanisms — hidden entity identification, citation checking, and evidence connectivity — that are specific to the search context and have no direct analog in prior rubric-based alignment work.

3. Technical Approach

3.1 Reader Orientation

This paper presents a training algorithm—not a new model architecture or dataset—that augments reinforcement learning for deep search agents with fine-grained rewards that evaluate how an agent arrived at its answer, not just whether the answer is correct. The core problem it solves is that standard RL with binary outcome rewards produces agents that learn shortcut behaviors (resolving only the easiest parts of questions, fabricating support) rather than conducting thorough, evidence-grounded search; the solution is a three-step rubric verification pipeline that decomposes questions into atomic factual claims and then scores agent trajectories based on whether they explicitly identify intermediate entities, back every claim with citations, and form connected evidence chains—then folds this score into the RL objective as a weighted auxiliary reward that applies only to trajectories that already got the answer right.

3.2 Big-Picture Architecture (Diagram in Words)

The system has six major components arranged in a pipeline that spans both data preparation (offline, before RL begins) and reward computation (online, during each RL training step):

  1. Training Data (DeepDive dataset) — 2,234 synthetic multi-hop QA pairs derived from knowledge graph random walks with deliberate entity obfuscation. Each question has a short-form ground-truth answer. This is the environment: on each RL rollout, the agent receives a question from this set, executes web search and browsing actions, and produces a final answer with citations.

  2. Rubric Initializer (LLM M_rubric) — An LLM (DeepSeek-v3.2 in all experiments) that decomposes each training question into a list of single-hop factual statements (rubrics) and identifies which entities are "hidden" (must be discovered during search). This runs once before RL training and the rubrics remain fixed throughout.

  3. Agent Policy (Qwen3 base model + LoRA or full fine-tuning) — The LLM being trained, which follows the ReAct paradigm: it generates thoughts, issues tool calls (search, open, find), receives observations, and eventually produces a final response with an explanation with citations and an exact answer. This is the policy \pi_\theta being optimized.

  4. Citation-Aware Rubric Reward Engine (LLM M_judge + graph algorithm) — A three-step verification procedure that (a) identifies which hidden entities the agent explicitly named in its response, (b) checks whether each fully-identified rubric is supported by the cited web content, and (c) constructs a bipartite entity-rubric graph and runs BFS from the predicted answer entity to determine which rubrics form a connected evidence chain. The rubric reward R^H_r is the fraction of all rubrics that pass all three checks.

  5. Outcome Reward Verifier (LLM M_judge, same model) — Checks whether the agent's extracted final answer matches the ground-truth answer. Produces a binary reward R^H_o \in \{0, 1\}.

  6. C-GRPO Optimizer — Computes a mixed reward (Equation 13) that combines outcome and rubric rewards, normalizes across the rollout group, and updates the policy via the GRPO token-level objective (Equation 15). The key design choice: rubric rewards are multiplied by the outcome reward, so they only apply to trajectories that already got the right answer.

Information flows as follows: before training, the rubric initializer processes all 2,234 training questions into fixed rubrics and entity sets. During each RL step: a batch of questions is sampled → the agent policy generates G rollouts per question (actual implementations use 8 samples per prompt across 16 prompts for a global batch of 128) → each rollout produces a trajectory with tool calls, observations, and a final response → the outcome verifier assigns binary rewards → the citation-aware rubric engine computes rubric rewards for each trajectory → the C-GRPO optimizer combines these into mixed rewards, normalizes within groups, and updates the policy parameters \theta.

3.3 Roadmap for the Deep Dive

  • First, the training data and agent framing (ReAct paradigm, tool definitions, trajectory format) — because everything downstream depends on understanding what a "trajectory" contains and what kind of questions the agent faces.

  • Second, rubric initialization — how questions are decomposed into atomic statements and hidden entities, because this is the static reference against which all trajectories are judged.

  • Third, the three-step reward computation procedure — hidden entity identification, citation-based rubric judgment, and evidence connectivity check — because this is the core technical contribution and the most complex component.

  • Fourth, the C-GRPO algorithm — how rubric rewards are combined with outcome rewards, the normalization scheme, and the parameter \alpha that controls the balance — because this is the mechanism that connects fine-grained feedback to policy optimization.

  • Fifth, the training pipeline — SFT cold-start, RL hyperparameters, context budgets, and tool configurations — because these practical choices determine whether the algorithmic ideas translate to measurable improvements.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an algorithm design and empirical validation paper whose core idea is that reinforcing comprehensive, citation-grounded reasoning during RL training—rather than only reinforcing final-answer correctness—produces search agents that are both more accurate and more robust, and that this can be achieved through a rubric-based reward signal derived automatically from the compositional structure of synthetic multi-hop questions.

Training Data and Agent Framing

The paper operates within the established paradigm of training deep search agents through RL on synthetic multi-hop QA datasets. Understanding the data and agent architecture is prerequisite to understanding the reward mechanism, because the reward mechanism exploits specific structural properties of the data.

The DeepDive dataset. The training data comes from DeepDive (Lu et al., 2025), an open-source deep search dataset constructed through the following procedure: (1) random walks over knowledge graphs generate multi-hop reasoning paths connecting entity-centric facts; (2) entity mentions in the resulting questions are deliberately obfuscated—replaced with descriptive phrases rather than proper names. For example, rather than asking "What was the estimated initial mass of the 2015 Thailand bolide?", the question becomes "The evidence for a prehistoric cosmic collision was found near a coastal settlement planned by a Scottish engineer. Such collisions produce very bright atmospheric phenomena, and a catalogue of these includes a notable mid-2010s occurrence over a Southeast Asian nation. What was the estimated initial mass of the object causing that Southeast Asian event?" The obfuscation forces the agent to execute genuine web searches to resolve entity identities rather than relying on parametric knowledge. The dataset contains 1,016 SFT samples and 2,234 RL samples. Each question's final answer is a short-form entity string (e.g., "66 tonnes", "St. Lawrence Parks Commission"), enabling automatic correctness verification through string matching or LLM judging.

The ReAct agent formulation. The agent follows the ReAct paradigm (Yao et al., 2023), formalized in Equation 1:

H=(τ1,a1,o1,…,τt,at,ot,…,τT,aT)H = (\tau_1, a_1, o_1, \ldots, \tau_t, a_t, o_t, \ldots, \tau_T, a_T)

where \tau_t is the thought at step t (the LLM's internal reasoning about what to do next), a_t is the action at step t (a tool call for steps 1 through T-1, or the final response for step T), and o_t is the observation returned by the environment at step t (search results, opened webpage content, or keyword match results).

What this equation represents: a complete agent trajectory is a sequence of alternating reasoning-tool-observation cycles. For steps 1 through T-1, the action is one of three browsing tools; for step T, the action is the final response containing an explanation with citations and an exact answer. The trajectory captures the full history—every thought, every tool call, every piece of web content the agent saw.

Tool specifications. The agent has access to three browsing tools, each generating observations that become part of the trajectory context:

  • browser.search: Takes a query string and an optional num parameter (default 10). Returns the top-n relevant webpages from the Serper API, each represented as a title, URL, and text snippet.

  • browser.open: Takes a URL (or ID referencing a previously returned URL) and fetches the full webpage content via the Jina API. Returns the first 10,000 characters of the page to fit within context limits.

  • browser.find: Takes a pattern string and searches for it within the currently opened webpage using vanilla string matching. Returns the surrounding content of each match.

The trajectory format is specified in precise detail in Appendix A (Figures 5 and 6 of the paper), using a structured token scheme with <|im_start|> and <|im_end|> delimiters that separate user messages, assistant thoughts with tool calls, and tool responses. This format is mechanically enforced during training: trajectories with format errors (malformed tool calls, missing delimiters) receive a reward of 0 (Section 2.3).

Why this framing matters. The ReAct formulation with explicit tool calls and observations means that every piece of information the agent uses must come from the web environment (in the observations) or from the LLM's parametric knowledge. The rubric reward mechanism will later exploit this by checking whether the agent's stated facts are supported by the cited web content—not by the agent's parametric knowledge. This is a crucial distinction: the reward framework treats parametric knowledge as insufficient, requiring explicit citation support, which pushes the agent toward verifiable evidence-gathering behavior.

Rubric Initialization: Decomposing Questions into Verifiable Single-Hop Statements

Before RL training begins, the system processes every question in the training set through a rubric initialization procedure that produces a static set of rubrics and hidden entities. This procedure is critical because it determines the granularity and coverage of the reward signal.

The rubric initializer prompt. The system uses a prompt (Appendix E, Figure 12) that instructs an LLM M_rubric (DeepSeek-v3.2) to decompose a complex multi-hop question into atomic single-hop factual statements. The prompt includes:

  • A clear specification that the final answer entity is <E0>
  • Instructions that each constraint must be a single-hop factual statement with intermediate entities denoted as <E1>, <E2>, <E3>, etc.
  • Requirements that each constraint contains at least one entity, is grammatically correct, and does not attempt to infer actual entity identities
  • A detailed example showing the decomposition of a multi-hop question about a political movement's central institution into 14 single-hop constraints

The formal rubric initialization. Equation 2 captures this process:

Eq,Rq=Mrubric(q)E_q, R_q = M_{rubric}(q)

where q is a training question, E_q = \{e_0, e_1, \ldots, e_{n_q}\} is the set of hidden entities (placeholders) that the agent should discover when solving q, and R_q = \{r_1, \ldots, r_{m_q}\} is the set of rubrics. Each hidden entity e_i is represented by a placeholder <E_i>, with e_0 being the final answer entity. Each rubric r_j = (s_j, E_{q,j}) consists of an atomic factual statement s_j (a natural language sentence with entity placeholders) and an entity set E_{q,j} \subseteq E_q (the subset of hidden entities that appear in that rubric).

What this produces concretely. For the Thailand bolide example from Figure 1, the rubric initializer produces something like the rubric set shown in Figure 2:

  • R1: <E1> is a coastal settlement planned by a Scottish engineer <E2>.
  • R2: Evidence for a prehistoric cosmic collision <E3> was found near <E1>.
  • R3: Collisions like <E3> produce very bright atmospheric phenomena <E4>.
  • R4: <E5> is a catalogue of <E4>.
  • R5: <E5> includes a mid-2010s occurrence <E6>.
  • R6: <E6> is over a Southeast Asian nation <E7>.
  • R7: <E6> was caused by an object <E8>.
  • R8: <E0> is the estimated initial mass of <E8>.

The hidden entity set would be E_q = {e_0, e_1, e_2, e_3, e_4, e_5, e_6, e_7, e_8}, with each e_i corresponding to the placeholder <E_i>.

Why this decomposition form is chosen. The key property that makes this decomposition useful for RL is that each rubric is independently verifiable given the agent's response and cited web content. There is no need to check whether R3 "correctly follows from" R2 or whether the reasoning chain is logically valid—only whether (a) the agent identified the entities that fill the placeholders, (b) the resulting factual statement is supported by citations, and (c) the supported rubrics can be connected to the predicted answer. This makes the reward computation modular and reduces the burden on the judge LLM, which only needs to verify atomic facts against source text rather than evaluate multi-step logical derivations.

Pre-generation and freezing. The paper explicitly states that rubrics are "pre-generated before training and remain unchanged throughout the RL process" (Section 2.2.1). This is a deliberate design choice that avoids several failure modes: if rubrics were generated dynamically during training, the reward signal would be non-stationary (the same trajectory could receive different rewards at different training steps due to variations in rubric generation), and the cost of rubric generation would scale with the number of RL steps. By fixing rubrics upfront, the reward computation is deterministic for a given trajectory, and the only per-step cost is the three-step verification procedure (which uses the judge LLM but not the rubric initializer).

Relationship to synthetic data construction. The rubric decomposition exploits a property specific to synthetic multi-hop QA: because the questions are constructed from knowledge graph walks, the intermediate entities and hops are well-defined even if they are obfuscated in the question text. A purely natural question like "Why did the Roman Empire fall?" does not decompose into well-defined atomic factoids in this way. The paper acknowledges this limitation in Section 6: "rubric generation relies on the compositional structure of synthetic multi-hop questions" and may not directly adapt to open-ended QA. However, since the RL training itself uses synthetic data, this limitation does not affect the training procedure—only the potential extension to other training data paradigms.

Step 1 of Reward Computation: Hidden Entity Identification

Once rubrics are initialized and an agent generates a trajectory during RL, the first step of CaRR evaluates reasoning comprehensiveness: did the agent actually uncover and explicitly state the identities of the hidden entities?

The hidden entity identification procedure. Equation 4 formalizes this step:

{e0H,…,enH}=Mjudge(q,Rq,Eq,aT)\{e^H_0, \ldots, e^H_n\} = M_{judge}(q, R_q, E_q, a_T)

where M_judge is the judge LLM (again DeepSeek-v3.2), q is the question, R_q is the rubric set, E_q is the hidden entity set, a_T is the agent's final response (the text containing the explanation with citations and the exact answer), and each e^H_i is either the entity name explicitly stated in a_T for placeholder <E_i>, or null if the agent did not clearly identify that entity.

What "explicitly identified" means. The judge LLM is prompted to extract entity identities from the agent's response (Appendix E, Figure 13). The critical instruction is: "Only use information that is explicitly and unambiguously stated in the assistant's response. Do not infer, guess, or deduce entity identities beyond what is explicitly provided." This means the agent must say the entity name in its final answer—having it implicitly in the cited web content or in the agent's earlier thoughts is insufficient. The paper provides a human verification study (Appendix C): across 128 hidden entities from 10 trajectories, the judge LLM achieved 97.7% accuracy compared to human assessments, indicating high reliability for this extraction task.

The fully-identified rubric filter. Not all rubrics are evaluated further. Equations 5-7 define the filter:

fH(ei)=eiH,∀ei∈Eqf_H(e_i) = e^H_i, \quad \forall e_i \in E_q

This is the entity identification mapping from placeholders to actual names (or null). Each rubric r_j = (s_j, E_{q,j}) is instantiated by replacing placeholders with their identified names:

Eq,jH={fH(ei)∣ei∈Eq,j}E^H_{q,j} = \{f_H(e_i) \mid e_i \in E_{q,j}\} rjH=(sj,Eq,jH)r^H_j = (s_j, E^H_{q,j})

The set of fully-identified rubrics is:

Rqidentify={rjH∈RqH∣eiH≠null,∀eiH∈Eq,jH}R^{identify}_q = \{r^H_j \in R^H_q \mid e^H_i \neq \text{null}, \forall e^H_i \in E^H_{q,j}\}

In operational terms: for each rubric, check whether every hidden entity in that rubric has been explicitly identified by the agent. If any entity in a rubric is still null (the agent didn't name it), that rubric is excluded from further judgment. Only rubrics where all entities are identified proceed to Step 2.

What this computes in practice. For the Thailand bolide example, if the agent's response mentions "2015 Thailand bolide," "Thailand," "meteoroid," and "66 tonnes" but never explicitly identifies the coastal settlement, the Scottish engineer, the prehistoric collision, or the bolide catalogue, then only rubrics R6, R7, and R8 would pass the fully-identified filter (because those rubrics involve entities E6, E7, E8, and E0, which the agent did identify). Rubrics R1 through R5 would be excluded because they involve entities E1 through E5, which are null.

Why this step matters. This is the mechanism that penalizes shortcut exploitation. If an agent resolves only the last few hops of a question—the ones directly leading to the answer—it will have a low entity identification rate for the earlier hops. Even if the outcome reward is 1 (the final answer is correct), the rubric reward will be low because most rubrics were never fully identified. This creates a gradient toward resolving all constraints in the question, not just the ones that determine the answer.

The ablation in Table 5 ("w/o Hidden entity identification") confirms this: removing this step and letting the judge LLM directly select supported rubrics without checking entity identification leads to a clear performance drop (e.g., on BrowseComp, accuracy drops from 17.5 to 16.5 for the 4B model). Without entity identification enforcement, the agent can cite webpages without explicitly stating what it found in them, making the reward signal less informative about reasoning comprehensiveness.

Step 2 of Reward Computation: Citation-Based Rubric Judgment

The second step evaluates factual grounding: are the agent's claims actually supported by the web content it cited?

The context extraction procedure. Before judgment can occur, the system must collect the relevant web content. Equation 8 defines this:

{url1,…,urlk}=ExtractCitation(aT)\{url_1, \ldots, url_k\} = \text{ExtractCitation}(a_T) CH=CollectContent(H,url1,…,urlk)C_H = \text{CollectContent}(H, url_1, \ldots, url_k)

The ExtractCitation function uses regex to extract cited URLs from the agent's final response. The paper imposes a limit of at most 20 cited URLs "to prevent the agent from hacking the reward by citing a large amount of webpages" (Section 2.2.2, footnote). The CollectContent function then gathers, from the full trajectory H, all web content associated with those URLs: deduplicated search snippets, opened webpage content, and keyword match results from browser.find calls.

Why extract only cited URLs rather than all observed content. This is a deliberate choice that prevents a subtle form of reward hacking. If the system used all web content the agent observed during the trajectory, the agent could cite one set of URLs in its response (making it look well-supported to a human reader) while actually relying on different URLs that were never explicitly cited. By restricting the supporting context C_H to only the content of cited URLs, the system ensures that the agent's explicit claims of support must be genuine—the cited sources must actually contain the evidence.

The citation-based judgment procedure. Equation 9 defines this:

{sp1,…,spmq}=Mjudge(Rqidentify,CH)\{sp_1, \ldots, sp_{m_q}\} = M_{judge}(R^{identify}_q, C_H) Rqsupport={rjH∈Rqidentify∣spj=1}R^{support}_q = \{r^H_j \in R^{identify}_q \mid sp_j = 1\}

For each fully-identified rubric r^H_j, the judge LLM M_{judge} examines the supporting context C_H and determines whether the factual statement r^H_j (with all entity placeholders now filled with identified names) is fully supported by the cited web content. The judgment is binary: sp_j = 1 if supported, sp_j = 0 otherwise. The supported rubric set R^{support}_q contains all fully-identified rubrics that pass this citation check.

The judge LLM prompt for this step (Appendix E, Figure 14) instructs the model to:

  1. Find exact evidence from the webpage contents that supports or contradicts each statement
  2. Explain clearly why the statement is or is not fully supported, citing relevant parts of the provided text
  3. List the URLs where supporting evidence was found
  4. Output a JSON mapping each statement label to a boolean

This is more demanding than a simple yes/no judgment—the model must provide reasoning and evidence locations, which improves reliability (as confirmed by the 95.1% human agreement rate reported in Appendix C) and also provides interpretability for debugging.

What "fully supported" means. The judge checks whether the cited web content contains sufficient information to verify the specific factual claim in the rubric, with the identified entities substituted in. For example, for rubric R8 instantiated as "66 tonnes is the estimated initial mass of meteoroid," the judge would check whether the cited Wikipedia article about the 2015 Thailand bolide states that the meteoroid had an estimated mass of 66 tonnes. If the web content says something different (e.g., "3.5 metres in diameter" without mentioning mass, or "estimated mass of 60 tonnes"), the rubric is not supported.

Why this step matters for factual grounding. This is the mechanism that punishes hallucination. If an agent identifies an entity but cites a webpage that does not actually support the associated factual claim, that rubric gets filtered out at this stage. The agent cannot earn rubric reward by citing irrelevant or non-existent sources—the content must genuinely support the claim.

Note that citation-based judgment is applied only to R^{identify}_q (rubrics whose entities were all identified), not to all rubrics. Rubrics with unidentified entities are already excluded, so no further judgment is needed.

Step 3 of Reward Computation: Evidence Connectivity Check

The third step evaluates evidence connectivity: do the supported rubrics form a coherent chain that actually connects to the predicted answer? This prevents the agent from satisfying rubrics in isolation without demonstrating that those satisfied rubrics are relevant to answering the question.

The bipartite graph construction. Equation 10 defines:

GH={EqH∪Rqsupport,E}G_H = \{E^H_q \cup R^{support}_q, E\}

This is a bipartite graph with two types of nodes: (a) all identified entities E^H_q (including those that were not part of any supported rubric), and (b) all supported rubrics R^{support}_q. An edge (e^H_i, r^H_j) \in E exists if entity e^H_i appears in rubric r^H_j, i.e., if e^H_i \in E^H_{q,j} (the entity set of that rubric after instantiation).

The breadth-first search from the predicted answer. Equation 11 defines:

Rqconnect={rjH∣rjH is connected to e0H in GH}R^{connect}_q = \{r^H_j \mid r^H_j \text{ is connected to } e^H_0 \text{ in } G_H\}

Starting from e^H_0 (the predicted answer entity identified from the agent's response), the system runs BFS on the bipartite graph. A supported rubric r^H_j is "connected" to the answer if there exists a path in the graph from e^H_0 to r^H_j alternating between entity nodes and rubric nodes. This path represents an evidence chain: the predicted answer connects to a rubric that involves it, which connects to other entities, which connect to other rubrics, and so on.

The final rubric reward. Equation 12 defines:

RrH=∣Rqconnect∣∣Rq∣R^H_r = \frac{|R^{connect}_q|}{|R_q|}

This is the fraction of all rubrics in the question that are (a) fully identified, (b) citation-supported, and (c) connected to the predicted answer via the evidence chain graph.

What this computes concretely. For the Thailand bolide example with 8 rubrics total, if the agent identified entities for rubrics R4 through R8, and all five of those are citation-supported, and R6, R7, R8 form a connected chain to the answer entity E0 (66 tonnes), but R4 and R5 are isolated from that chain (perhaps because the entity E5, "Bolide Catalogue," is not connected through supported rubrics to E0), then R^{connect}_q would contain R6, R7, and R8—3 rubrics—and the rubric reward would be 3/8 = 0.375.

Why connectivity is necessary. The ablation in Table 5 ("w/o Evidence connectivity check") demonstrates the importance of this step. Without it, the rubric reward would simply be |R^{support}_q| / |R_q|, counting any supported rubric regardless of whether it forms a coherent chain to the answer. The ablation shows this leads to "a substantial decline in performance" because "the agents learn to hack rubrics by finding entities that satisfy isolated factual statements but are unrelated to the final answer." For example, an agent could find a Scottish engineer who planned a coastal settlement (satisfying R1), and separately find the 2015 Thailand bolide's mass (satisfying R8), without demonstrating that the coastal settlement and the bolide are actually connected through the intermediate constraints. The connectivity check enforces that the satisfied rubrics must form a logical bridge from the answer backward through the question's constraint structure.

The graph as a structure-aware filter. The bipartite graph and BFS encode an assumption about the question's structure: that the answer entity should be reachable through a chain of shared entities across rubrics. This assumption holds for synthetic multi-hop questions (which are constructed from entity-centric knowledge graph walks) but might not hold for arbitrary decompositions. The paper doesn't discuss this assumption explicitly, but it is a natural consequence of the synthetic data paradigm—each question is literally generated by walking a graph, so the rubrics derived from that question should form a connected subgraph if fully resolved.

The judge LLM does not perform this step. Notably, the connectivity check is algorithmic (BFS on a constructed graph), not LLM-based. This is an important design choice: graph connectivity is a deterministic, objective property that doesn't require the judge LLM to make subjective assessments about "whether the evidence forms a coherent argument." The judge LLM is only used for the entity identification (Step 1) and citation judgment (Step 2), which require natural language understanding. The connectivity check is purely structural and cannot be gamed by the agent through persuasive but incorrect response formatting.

The C-GRPO Algorithm: Combining Rubric and Outcome Rewards

The CaRR framework produces a scalar rubric reward R^H_r for each trajectory. The C-GRPO algorithm defines how this signal is combined with the traditional outcome reward to form the training objective.

The C-GRPO mixed reward. Equation 13 defines the core combination:

Ri=(1−α)⋅RoHi+α⋅RoHi⋅R^rHiR_i = (1 - \alpha) \cdot R^{H_i}_o + \alpha \cdot R^{H_i}_o \cdot \hat{R}^{H_i}_r

where R_i is the mixed reward for trajectory H_i in the rollout group, R^{H_i}_o is the binary outcome reward (1 if the extracted answer matches the ground truth, 0 otherwise), \hat{R}^{H_i}_r is the normalized rubric reward, and \alpha \in [0, 1] is a hyperparameter controlling the balance between outcome and rubric rewards.

What this form achieves. The multiplication R^{H_i}_o \cdot \hat{R}^{H_i}_r means that rubric rewards apply only to trajectories that already achieved the correct answer. Incorrect trajectories—regardless of how many rubrics they satisfied—receive R_i = 0. This preserves the primary objective of answer correctness while using the rubric signal to differentiate among correct trajectories: among trajectories that got the right answer, those that did so with more comprehensive, better-supported reasoning receive higher rewards.

Normalization of rubric rewards. Equation 14 defines:

R^rHi=RrHimax⁡j∈{1,…,G}RrHj\hat{R}^{H_i}_r = \frac{R^{H_i}_r}{\max_{j \in \{1,\ldots,G\}} R^{H_j}_r}

The rubric reward for each trajectory is divided by the maximum rubric reward in its rollout group. This normalization serves two purposes: (1) it ensures that the rubric reward component is always in [0, 1], keeping the scale consistent with the outcome reward and preventing the relative weighting \alpha from being confounded by varying absolute rubric reward magnitudes across different questions; (2) it creates a relative advantage signal: among correct trajectories, the one with the best reasoning (highest rubric satisfaction) gets a rubric reward of 1.0, while others get proportionally less.

Why normalize by group maximum rather than a fixed scale. The absolute rubric reward R^{H_i}_r depends on the number of rubrics |R_q|, which varies across questions. Some questions decompose into 5 rubrics; others into 15. Without normalization, the effective weight of the rubric component would vary with question complexity, making \alpha harder to tune. Group-relative normalization makes the rubric signal comparable across questions of different lengths.

Why only apply to correct trajectories. The ablation in Table 5 ("w/ Rubric rewards for all rollouts") is the key justification. Adding rubric rewards to all rollouts (i.e., R_i = (1-\alpha) \cdot R^{H_i}_o + \alpha \cdot \hat{R}^{H_i}_r without the outcome multiplier) causes substantial performance degradation (e.g., BrowseComp accuracy drops from 17.5 to 13.3). The paper explains: "the advantage of some incorrect rollouts will receive positive advantages when there are few correct rollouts or many overlength rollouts in a group, which frequently happens at the beginning of RL." In the GRPO advantage formulation, the advantage of a trajectory depends on its reward relative to the group mean and standard deviation. If most trajectories in a group are incorrect (outcome reward 0), but some have non-trivial rubric rewards (because they partially resolved the question), those incorrect-but-partially-correct trajectories would receive positive advantages, encouraging the model to reproduce behavior that doesn't actually lead to the right answer. Multiplying by the outcome reward zeros out rubric contributions for incorrect trajectories, preventing this pathological case.

The effect of \alpha. The hyperparameter \alpha was swept from 0 to 0.5 (Table 4). At \alpha = 0, C-GRPO reduces to standard GRPO (only outcome rewards). Performance improves as \alpha increases from 0 to 0.3, then declines at \alpha = 0.5. The paper interprets this as evidence that a moderate rubric weight encourages comprehensive reasoning without distracting from the primary goal of answer correctness. At \alpha = 0.5, the rubric signal dominates enough that the model may over-optimize for rubric satisfaction at the expense of finding the right answer—the classic tension between process quality and outcome quality.

The GRPO Optimization Objective

C-GRPO uses the same underlying optimization as GRPO but with the mixed reward R_i replacing the pure outcome reward.

The GRPO objective with token-level loss. Equation 15 defines:

J(θ)=E(q,gt)∼D,{Hi}i=1G∼πθold(⋅∣q)[1∑i=1G∑j=1∣Hi∣I(Hi,j)∑i=1G∑j=1∣Hi∣I(Hi,j)min⁡(ρi,jA^i,j,clip(ρi,j)1−ϵlow1+ϵhighA^i,j)]J(\theta) = \mathbb{E}_{(q, gt) \sim D, \{H_i\}^G_{i=1} \sim \pi_{\theta_{old}}(\cdot|q)} \left[ \frac{1}{\sum_{i=1}^G \sum_{j=1}^{|H_i|} I(H_{i,j})} \sum_{i=1}^G \sum_{j=1}^{|H_i|} I(H_{i,j}) \min \left( \rho_{i,j} \hat{A}_{i,j}, \text{clip}(\rho_{i,j})^{1+\epsilon_{high}}_{1-\epsilon_{low}} \hat{A}_{i,j} \right) \right]

where (q, gt) is a question-ground-truth pair from the training distribution D; \{H_i\}^G_{i=1} are G rollouts sampled from the old policy \pi_{\theta_{old}} for question q; H_{i,j} is the j-th token of trajectory H_i; I(H_{i,j}) \in \{0, 1\} is an indicator that is 1 if token H_{i,j} was generated by the LLM itself (i.e., not from observed web content—observations are environment tokens, not model-generated tokens); \rho_{i,j} = \pi_\theta(H_{i,j}|q, H_{i,1:j-1}) / \pi_{\theta_{old}}(H_{i,j}|q, H_{i,1:j-1}) is the importance sampling ratio (how much the new policy's probability for this token differs from the old policy's); and \hat{A}_{i,j} = (R_i - \text{mean}(\{R_k\}^G_{k=1})) / \text{std}(\{R_k\}^G_{k=1}) is the token-level advantage (the group-normalized reward, shared across all tokens in the trajectory).

What this objective computes. For each token generated by the model (not the environment), the algorithm computes: (1) the importance sampling ratio \rho_{i,j}—how much more or less likely the new policy is to generate this token compared to the old policy; (2) the advantage \hat{A}_{i,j}—how much better this trajectory's reward is compared to the average reward in the rollout group, expressed in standard deviation units; (3) a clipped surrogate objective \min(\rho \hat{A}, \text{clip}(\rho) \hat{A}) that prevents the policy update from being too large for any single token (the clip prevents \rho from going below 1 - \epsilon_{low} or above 1 + \epsilon_{high}, where \epsilon_{low} and \epsilon_{high} are hyperparameters). The objective is averaged across all model-generated tokens across all rollouts.

Why this form. GRPO's token-level objective with group-relative advantages and PPO-style clipping is designed for stable training in multi-turn settings where trajectory lengths vary and rewards are sparse. The group normalization (subtracting mean, dividing by standard deviation) means the absolute magnitude of the reward doesn't matter—only relative differences within the group. The clipping prevents the policy from changing too rapidly, which is especially important when the reward signal is noisy (as it is with LLM-based judges). The indicator I(H_{i,j}) ensures that the model is only trained to optimize its own generation choices, not to "predict" environment observations (which are fixed and not under the model's control).

The mixed reward's role in advantage computation. The advantage \hat{A}_{i,j} is computed from R_i, not from R^{H_i}_o alone. This means that in groups where multiple trajectories get the right answer (outcome reward 1), the trajectory with the higher rubric reward will have a higher advantage, receiving stronger positive updates. In groups where only one trajectory is correct and others are wrong, the rubric reward component doesn't matter (incorrect trajectories get mixed reward 0 regardless of rubrics, so the correct trajectory gets a large positive advantage and all others get negative advantages). The rubric signal is therefore most informative when the model has already learned to produce correct answers with non-trivial frequency—it then shapes which kind of correct reasoning is preferred.

Training Pipeline: SFT Cold-Start and RL Configuration

The training follows a two-stage process: cold-start supervised fine-tuning followed by RL.

Cold-start SFT. The paper uses a dataset of 832 high-quality trajectories generated through reject sampling with GLM-4.6 on the SFT split of DeepDive (1,016 questions). Reject sampling means generating multiple trajectories per question and keeping only those that both arrive at the correct answer and exhibit high-quality reasoning. This produces a clean starting policy that already knows the basic format and tool-using behaviors. Each model (4B and 30B) is fine-tuned on these 832 traces for 3 epochs with batch size 16, learning rate 4e-5, and maximum context length 128k tokens.

Why reject sampling rather than using all SFT data. The choice to use only 832 high-quality traces (out of 1,016 available questions) reflects a quality-over-quantity decision. Including incorrect or low-quality trajectories in SFT would teach the model undesirable behaviors (incomplete reasoning, format errors) that RL would then have to unlearn. Starting from a policy that already produces mostly correct, well-formatted trajectories makes the RL phase more stable because the initial rollout groups will have a higher proportion of successful trajectories, providing a stronger learning signal.

RL configuration. The RL phase uses all 2,234 QA pairs from the DeepDive RL split. The specific hyperparameters are:

  • Rollout size: 16 prompts per batch, 8 samples per prompt, for a global batch size of 128 trajectories
  • Temperature: 1.0 (high temperature encourages exploration)
  • Learning rate: 2e-6 (an order of magnitude lower than SFT, typical for RL fine-tuning)
  • Maximum context length: 64k tokens (shorter than the SFT context of 128k)
  • Training duration: 3 epochs over the RL dataset
  • Rubric reward weight \alpha: 0.3 (chosen based on the sweep in Table 4)
  • Judge LLM: DeepSeek-v3.2 for both outcome and rubric rewards

The 64k vs. 128k context distinction. The choice to train RL at 64k context while evaluating at both 64k and 128k is significant. Training at 64k forces the agent to learn efficient search within a constrained budget—trajectories that exceed 64k tokens receive a reward of 0. This constraint interacts with the reward design: with pure outcome rewards, the agent learns to be efficient by taking shortcuts (resolve the easiest hops, guess the rest). With C-GRPO, the agent learns to be efficient while still satisfying rubrics—it must achieve comprehensive, citation-supported reasoning within the budget. Evaluating at 128k then tests whether the learned policy generalizes to longer contexts, or whether it has overfit to the 64k efficiency constraint. The GRPO agent's degradation at 128k (Table 1) suggests overfitting; C-GRPO's continued improvement at 128k suggests the rubric reward teaches a policy that naturally scales with more context because it has learned to keep searching until rubrics are satisfied rather than stopping when an answer is found.

Format and overlength penalties. Trajectories with format errors (malformed tool calls, missing delimiters) or overlength problems (exceeding token or tool-call limits) receive a reward of 0. This is a hard constraint—the agent must learn to operate within the tool and context budget.

Tool API configurations. The search tool uses the Serper API with a default of 10 results per query. The open tool fetches pages via the Jina API and returns the first 10,000 characters. The find tool uses vanilla string matching. These are fixed throughout training—the agent cannot adjust them.

Summary of Key Design Choices and Their Justifications

  • Three-step verification (entity identification → citation judgment → connectivity check) rather than end-to-end LLM judging: Each step addresses a specific failure mode (incomplete reasoning, hallucinated support, isolated fact satisfaction) and can be verified independently. The ablation (Table 5) shows removing any step degrades performance, confirming each is necessary.

  • Rubric reward multiplied by outcome reward rather than added or applied to all trajectories: The ablation shows applying rubric rewards to incorrect trajectories harms performance due to optimization instability. The multiplicative form preserves the primary objective (answer correctness) while shaping how correct answers are produced.

  • Group-relative normalization of rubric rewards rather than absolute rubric scores: Different questions have different numbers of rubrics, and the absolute rubric reward scale varies. Group normalization makes the signal comparable across questions and ensures the rubric component doesn't dominate when a question has many rubrics.

  • Bipartite graph + BFS for connectivity rather than LLM-based coherence judgment: Graph connectivity is deterministic, objective, and computationally cheap. An LLM-based coherence assessment would introduce subjectivity, be more expensive, and potentially be gamed through persuasive but incorrect response formatting.

  • Pre-generated, frozen rubrics rather than dynamically generated rubrics during training: Ensures reward stationarity (the same trajectory always gets the same rubric reward), avoids per-step rubric generation cost, and prevents the rubric initializer from adapting to exploit patterns in the agent's behavior.

  • Training at 64k with evaluation at 128k rather than training at maximum context: Creates a test of generalization: policies that learn shortcut behaviors at 64k will fail to improve when given more context; policies that learn thorough evidence-gathering will improve further with more space.

  • Reject-sampled SFT rather than using all available SFT data: Starting from a policy that already produces correct, well-formatted trajectories makes RL more stable and prevents the model from needing to unlearn format errors or incorrect reasoning patterns learned during SFT.

4. Key Insights and Innovations

Innovation 1: Reframing Search Agent RL as a Process Supervision Problem — Not Just an Outcome Optimization Problem

The paper's most fundamental conceptual move is recognizing that training a deep search agent is structurally analogous to training a model for multi-step reasoning (like mathematical problem-solving), and therefore suffers from the same pathology: sparse outcome rewards incentivize correct answers reached through incorrect or incomplete reasoning. This is not a new observation in the abstract—process reward models are well-established in mathematical reasoning (Lightman et al., 2023; Shao et al., 2024)—but the paper is the first to argue, with empirical evidence, that the same dynamic governs web search agents, and to propose a process supervision mechanism adapted to the unique properties of that domain.

What makes this reframing non-obvious is the difference between mathematical reasoning steps and search agent steps. In math, a "step" is a logical derivation whose correctness can be evaluated by a learned reward model trained on human-labeled step-level data. In search, a "step" is a tool call, a web page observation, or a factual claim—entities that don't have an obvious correctness metric independent of the external web environment. The paper's insight is that the compositional structure of synthetic multi-hop questions provides a natural decomposition into verifiable atomic units—single-hop factual claims—that can serve as process checkpoints without requiring human annotation or learned reward models. The verification is grounded not in a model's judgment of reasoning quality but in the cited web content itself, which serves as an objective reference.

This reframing matters because it changes the optimization target. Prior work treated search agent RL as a problem of optimizing for answer accuracy, with methods like E-GRPO (Zhao et al., 2025) providing auxiliary signals only to distinguish near-misses from complete failures among incorrect trajectories. The paper's training dynamics (Figure 4) reveal why this framing is insufficient: outcome-only GRPO produces agents whose tool call steps keep decreasing throughout training—not because they're becoming more efficient searchers, but because they're learning to skip verification. The shift to process supervision reframes the goal as optimizing for comprehensive, evidence-grounded search processes that happen to produce correct answers, not answers that happen to be correct.

This is a fundamental reframing, not an incremental improvement. It recategorizes the problem from "how do we get the right answer more often?" to "how do we get the right answer for the right reasons?", and provides the empirical evidence (Table 2, Figure 4) that these two objectives diverge under outcome-only optimization—the former can improve while the latter degrades.

Innovation 2: The Three-Step Rubric Verification as a Defense-in-Depth Against Reward Hacking

The paper's second distinctive contribution is the architecture of the rubric verification pipeline itself: not just that rubrics are used, but that they are verified through three sequential filters—entity identification, citation support, and evidence connectivity—each designed to prevent a specific failure mode that a simpler rubric-based reward would invite. This is best understood as a security mindset applied to reward design: assume the agent will try to maximize the rubric reward without actually doing thorough search, and design the verification to be robust against those strategies.

Layer 1: Entity identification prevents implicit knowledge exploitation. Without this step, an agent could satisfy rubrics by citing webpages that contain the relevant information without ever explicitly stating what entities it found—the judge LLM could infer that the agent "must have seen" the entity. The requirement that entities be explicitly named in the agent's response (with the judge instructed not to infer or deduce) forces the agent to demonstrate that it actually processed the information, not just that the information existed somewhere in its context. The ablation (Table 5, removing this step costs 1.0–3.3 points across benchmarks) shows this matters empirically.

Layer 2: Citation-based judgment prevents hallucination. Even with explicit entity names, an agent could fabricate facts or cite sources that don't actually support its claims. The citation check—which restricts verification to only the content of URLs the agent itself cited—closes this loophole. The agent cannot earn rubric credit by mentioning correct-sounding facts; it must cite sources that genuinely contain those facts. This is a stronger constraint than prior rubric-based approaches (e.g., Gunjal et al., 2025; Shao et al., 2025a) that evaluate outputs without checking supporting evidence.

Layer 3: Evidence connectivity prevents rubric isolation. The graph-based connectivity check addresses the most subtle failure mode: an agent could satisfy individual rubrics by finding entities that match each constraint independently, without demonstrating that those entities actually form a coherent chain linking to the answer. For the Thailand bolide example, this prevents the agent from scoring rubrics about a Scottish engineer's settlement (found in one source) and rubrics about the bolide's mass (found in another source) without showing how those facts connect. The ablation (Table 5, removing this step costs 2.4–6.3 points—the largest degradation of any component) confirms this is the most important of the three filters.

What makes this design intellectually distinctive is that each layer addresses a threat model that simpler rubric approaches are vulnerable to. Prior work on rubric-based rewards (Lambert et al., 2024; Dong et al., 2025b) typically applies a single-stage evaluation: does the output satisfy the rubric? The paper shows that for search agents, this single-stage approach is insufficient because it doesn't distinguish between genuine satisfaction and various forms of gaming. The connectivity check, in particular, has no analog in prior rubric-reward literature—it is a novel mechanism that exploits the graph structure of synthetic multi-hop questions to enforce a property (coherence) that is otherwise difficult to verify automatically.

This is a fundamental methodological contribution, not just an application of existing techniques to a new domain. The three-layer architecture provides a template for designing robust process rewards in any setting where intermediate outputs can be cross-referenced against external ground truth—code generation with test cases, fact-checking with source documents, scientific reasoning with literature citations.

Innovation 3: The Outcome-Multiplied Reward Formulation as a Principled Solution to the Auxiliary Reward Instability Problem

The paper's third contribution is a specific design choice in the reward combination—multiplying the rubric reward by the outcome reward so that only correct trajectories receive the auxiliary signal—that addresses a known instability in multi-objective RL, supported by a clean ablation demonstrating that the alternative (adding rubric rewards to all trajectories) is actively harmful.

This is not a theoretically novel idea in reinforcement learning—it echoes the concept of reward shaping (Ng et al., 1999), where potential-based auxiliary rewards preserve optimal policies, and the broader idea that auxiliary objectives should not override the primary objective. But the paper's contribution is empirical and diagnostic: it demonstrates that for search agent RL specifically, the instability from applying process rewards to incorrect trajectories is severe enough to completely reverse the benefits of rubric-based training, and it provides a concrete mechanism explanation (incorrect trajectories receiving positive advantages when correct trajectories are scarce in early RL groups).

The ablation in Table 5 ("w/ Rubric rewards for all rollouts") is the key evidence: applying rubric rewards to all trajectories causes a catastrophic performance drop—BrowseComp accuracy falls from 17.5 to 13.3, BrowseComp-ZH from 24.7 to 14.0, xbench-DS from 54.0 to 40.3, and GAIA from 50.2 to 40.8. These are not marginal degradations; they represent a complete collapse of the benefits of rubric-based training. The paper's explanation—that early in RL, when most rollouts are incorrect, normalizing rubric rewards within groups can give high relative scores to trajectories that are fundamentally wrong—is a concrete diagnosis of why auxiliary rewards fail in this setting, not just an observation that they do.

This matters practically because it resolves a tension in the prior literature. E-GRPO (Zhao et al., 2025) proposed providing fine-grained rewards specifically to incorrect trajectories to distinguish near-misses from complete failures—the opposite strategy from C-GRPO. The paper does not directly compare to E-GRPO's mixed-reward strategy (E-GRPO is evaluated as a separate baseline, not as an ablation of C-GRPO's reward formulation), but the ablation strongly suggests that any approach giving process rewards to incorrect trajectories is vulnerable to the group-normalization instability problem. The paper's contribution is not a new theorem or algorithm but a critical empirical finding with direct practical implications: if you're adding process rewards to GRPO for search agents, the outcome multiplier is not a stylistic choice—it's essential for stability.

This is an incremental insight in terms of technical depth (it's one equation, not a new architecture), but a highly impactful one for practitioners. It provides a simple, portable rule (multiply auxiliary rewards by the outcome reward during GRPO training) that is supported by clean ablation evidence and a mechanistic explanation.

Innovation 4: Training Dynamics as a Diagnostic Tool for Revealing Policy Quality Degradation Under Outcome-Only RL

The paper makes a methodological contribution in how it analyzes training dynamics—not just reporting final benchmark scores, but tracking behavioral metrics (tool call steps, outcome rewards, rubric rewards, cited webpage counts) throughout RL training to diagnose when and how policies diverge from desirable behavior. This goes beyond standard evaluation and provides a window into the optimization process itself.

The key finding from the training dynamics analysis (Figure 4, Table 2) is that GRPO and C-GRPO produce trajectories of the same policy that move in opposite directions during training. GRPO's average tool call steps decline after an initial recovery period, eventually settling below the SFT baseline—a signature of shortcut learning, confirmed by case studies (Appendix D, Figures 7 and 9) showing GRPO agents resolving only the last few hops of multi-hop questions. C-GRPO's tool call steps increase after the initial decline, eventually exceeding the SFT baseline—a signature of more thorough search, confirmed by case studies (Figures 8 and 10) showing agents systematically verifying each constraint. Similarly, Table 2 shows that GRPO reduces the number of cited webpages and satisfied rubrics relative to SFT, while C-GRPO increases both.

What makes this diagnostic approach distinctive is that it reveals policy quality degradation that is invisible to the outcome reward curve. Figure 4 (center panel) shows that GRPO and C-GRPO achieve similar outcome rewards during training—both improve over SFT. A practitioner monitoring only outcome reward would conclude both methods are working. But the tool call step trajectory (left panel) tells a different story: GRPO is achieving those outcome rewards through a different kind of policy—one that does less search, cites fewer sources, and satisfies fewer constraints. The consequences of this divergence only become visible at test time on more challenging benchmarks or longer context budgets (Table 1: GRPO degrades at 128k while C-GRPO improves).

This is a methodological contribution with implications beyond this paper. It suggests that the standard practice of monitoring only reward curves during RL training is insufficient for multi-step agent settings—behavioral metrics that capture how the agent achieves its rewards are essential diagnostic tools. It also provides a concrete framework for future work: track tool call counts, citation frequency, and constraint satisfaction rates during training to detect shortcut learning before it crystallizes into a final policy. The paper doesn't propose these as formal metrics or claim they should replace benchmark evaluation, but the analysis demonstrates their diagnostic value in a way that generalizes to any search agent RL setup.

5. Experimental Analysis

Evaluation Methodology

  • Datasets. The primary training dataset is DeepDive (Lu et al., 2025), an open-source deep search dataset synthesized through knowledge graph random walks with deliberate entity obfuscation. It contains 1,016 questions for SFT and 2,234 questions for RL. All questions have short-form entity-string answers enabling automatic correctness verification. Evaluation is conducted on four challenging deep search benchmarks:

    • BrowseComp (Wei et al., 2025): English-language complex web browsing questions.
    • BrowseComp-ZH (Zhou et al., 2025): Chinese-language analog of BrowseComp.
    • xbench-DeepSearch (Xbench-Team, 2025): A deep search benchmark testing multi-step web navigation and reasoning.
    • GAIA (Mialon et al., 2024): The text-only validation subset of the GAIA benchmark for general AI assistants.

    Additionally, open-ended generalization is evaluated on DeepResearch Bench (Jin et al., 2025b), where agents must write PhD-level research reports assessed by Gemini-2.5-Pro-preview across multiple dimensions (comprehensiveness, insight, instruction following, readability).

  • Base models. All experiments use Qwen3-4B-Thinking-2507 (dense architecture) and Qwen3-30B-A3B-Thinking-2507 (Mixture-of-Experts architecture) from the Qwen3 family (Team, 2025b). These models were chosen to cover different scales and architectures (dense vs. MoE), demonstrating that the approach generalizes across model families. The "Thinking" variants include built-in chain-of-thought capabilities that are relevant for the reasoning demands of deep search.

  • Metrics. The primary metric is accuracy — the fraction of evaluation questions for which the agent's final answer matches the ground truth, as judged by GPT-5-Chat under the official LLM-as-judge settings of each benchmark. For BrowseComp-ZH, xbench-DeepSearch, and GAIA, evaluation is repeated 3 times and average accuracy is reported to account for the relatively small dataset sizes and stochasticity in agent behavior. For DeepResearch Bench, the metric is a quality score (0–100) assigned by Gemini-2.5-Pro-preview across four dimensions, with an overall score computed from the dimension averages.

  • Baselines. The paper compares against two primary RL algorithms and several reference agents:

    • GRPO (Shao et al., 2024; Jin et al., 2025a): Standard Group Relative Policy Optimization using only binary outcome rewards (1 if the extracted answer matches the ground truth, 0 otherwise). This is the dominant paradigm in prior work on RL for search agents.
    • E-GRPO (Zhao et al., 2025): An extension of GRPO that provides fine-grained rewards to incorrect rollouts based on the normalized entity match rate — the fraction of golden hidden entities identified during the agent's reasoning process, designed to distinguish "near-miss" samples from complete failures.
    • SFT baselines: The cold-start supervised fine-tuned models (DeepDive-4B-SFT and DeepDive-30B-SFT) before any RL training.
    • Reference agents (Table 1): OpenAI o3, DeepSeek-v3.1, Tongyi-DeepResearch, GLM-4.5, GLM-4.6, Asearcher-Web-32B, WebSailor-7B/32B, WebExplorer-8B, and DeepDive-9B/32B. These use varying training data, model scales, and context lengths, and are included for contextual comparison rather than controlled ablation. The paper explicitly notes they "may adopt different training data and context lengths from us."

  • Generation budget / compute accounting. The paper uses two budgets to measure test-time compute:

    • Context budget: The maximum token length of the agent's trajectory, tested at 64k (the RL training context length) and 128k (to assess test-time scaling behavior).
    • Tool call budget: The number of tool-calling steps the agent takes before producing a final answer. Trajectories exceeding tool-call limits receive a reward of 0 during training. Both budgets are reported in the test-time scaling analysis (Figure 3, middle and right panels) to understand how performance varies with available computation.

    During RL training, the key compute metric is the generation budget for rollouts: 16 prompts per batch, 8 samples per prompt, resulting in a global batch size of 128 trajectories. The cost of rubric reward computation (which requires LLM judging for entity identification and citation checking) is not explicitly accounted for in the training budget, though the paper notes it uses a separate judge LLM (DeepSeek-v3.2) rather than the policy model itself.

  • Cross-validation / statistical protocol. For the three smaller benchmarks (BrowseComp-ZH, xbench-DeepSearch, GAIA), evaluation is repeated 3 times with different random seeds and average accuracy is reported. This accounts for variance from both agent stochasticity (temperature = 1.0 during evaluation) and LLM-as-judge noise. BrowseComp evaluation is conducted once, likely due to its larger size. The paper does not report confidence intervals or statistical significance tests. For the DeepResearch Bench evaluation, Gemini-2.5-Pro-preview provides a single score per dimension per model, with no reported repetition.

Main Quantitative Results

Overall Benchmark Performance

The headline result is that C-GRPO consistently and substantially outperforms both GRPO and E-GRPO baselines across all four deep search benchmarks at both model scales (Table 1). At the 4B scale with 128k context budget, C-GRPO achieves an average improvement over GRPO of 8.0 percentage points (averaged across all four benchmarks: 17.5 vs. 14.7 on BrowseComp, 24.7 vs. 17.5 on BrowseComp-ZH, 54.0 vs. 41.3 on xbench-DS, 50.2 vs. 41.1 on GAIA). At the 30B scale with 128k context, the average improvement is 6.0 percentage points (24.8 vs. 18.9, 33.3 vs. 26.1, 57.7 vs. 52.0, 56.3 vs. 51.1, respectively).

A more granular breakdown (Figure 3, left panel) shows the improvement of each RL method over the SFT baseline at the 64k context length (the RL training context). For the 30B model, GRPO provides improvements of 3.8, 8.3, 8.3, and 5.1 percentage points on BrowseComp, BrowseComp-ZH, xbench-DS, and GAIA respectively. C-GRPO provides larger improvements of 5.7, 10.2, 12.3, and 7.7 percentage points on the same benchmarks — roughly 1.5× the improvement of GRPO on average.

Comparison with E-GRPO. E-GRPO performs inconsistently relative to GRPO. At the 4B scale with 128k context, E-GRPO achieves 14.5 on BrowseComp (vs. 14.7 for GRPO and 17.5 for C-GRPO), 20.2 on BrowseComp-ZH (vs. 17.5 for GRPO), 45.0 on xbench-DS (vs. 41.3 for GRPO), and 42.4 on GAIA (vs. 41.1 for GRPO). E-GRPO sometimes exceeds GRPO (BrowseComp-ZH, xbench-DS, GAIA at 4B) but sometimes underperforms it (BrowseComp at 4B, BrowseComp and BrowseComp-ZH at 30B). Critically, E-GRPO never matches C-GRPO — the gap ranges from 2.9 to 9.7 points at 4B-128k and from 1.0 to 9.3 points at 30B-128k. This consistent underperformance relative to C-GRPO, despite E-GRPO also providing fine-grained rewards, supports the paper's claim that entity match rate alone (without citation verification or connectivity checks) is an insufficient auxiliary signal.

Comparison with reference agents. At the 30B scale with 128k context, C-GRPO achieves 24.8 on BrowseComp — competitive with GLM-4.5 (26.4) and exceeding Asearcher-Web-32B (5.2), WebSailor-32B (10.5), WebExplorer-8B (15.2), and DeepDive-32B (15.3). On xbench-DS, C-GRPO's 57.7 approaches DeepSeek-v3.1 (71.2) and Tongyi-DeepResearch (75.0), substantially outperforming all other open-source agents (WebExplorer-8B at 53.7, DeepDive-32B at 51.8). On GAIA, C-GRPO's 56.3 is competitive with Asearcher (52.8) and WebExplorer (50.0) but lags behind proprietary agents (OpenAI o3 at 70.5, Tongyi-DeepResearch at 70.9, DeepSeek-v3.1 at 63.1). The paper notes that reference agents use different training data and may have access to larger context budgets or different tool configurations, making these comparisons contextual rather than strictly controlled.

Test-Time Scaling Behavior

One of the paper's most striking findings concerns how RL methods affect test-time scaling — the ability to use additional context or tool-call budget to improve performance at evaluation time (Figure 3, middle and right panels).

Context budget scaling (Figure 3, middle). The x-axis shows context length from 16k to 128k, and the y-axis shows accuracy. Three curves are plotted for the 30B model: SFT, GRPO, and C-GRPO. The SFT model shows steady improvement as context increases from 16k to 128k — approximately 8% → 21% accuracy. The GRPO model shows improvement from 16k to 64k (the training context length), reaching approximately 19% at 64k (above the SFT baseline at the same budget), but then flattens or slightly declines when extended to 128k (approximately 18.5%). The C-GRPO model starts below GRPO at 16k (approximately 5%) but shows consistent improvement across the full range, reaching approximately 24% at 128k — substantially above both GRPO and SFT.

This pattern — GRPO degrading at test-time context lengths beyond its training context — is interpreted as evidence of brittle shortcut policies. At 64k, the GRPO agent has learned an efficient policy that resolves questions with minimal search; when given 128k, this policy doesn't naturally extend because it was optimized to succeed within the tighter budget, not to conduct thorough verification. C-GRPO, incentivized to satisfy rubrics regardless of budget, learns a more scalable policy: more context simply means more room to find and verify evidence.

The same pattern appears in the tool call budget scaling (Figure 3, right), though with less dramatic divergence. All three methods improve with more tool calls, but C-GRPO shows the steepest improvement curve.

At the 4B scale, the test-time scaling behavior is similar but with smaller absolute gaps (Table 1: 4B-GRPO at 64k = 12.9 vs. 128k = 14.7 on BrowseComp, compared to 4B-C-GRPO at 64k = 13.9 vs. 128k = 17.5). The GRPO 4B model does show some improvement from 64k to 128k (unlike the 30B model which degraded), but the improvement is modest (+1.8 points) compared to C-GRPO (+3.6 points) and SFT (+6.4 points).

Training Dynamics Analysis

Figure 4 provides a window into how the policies diverge during RL training, tracking three metrics across 400 training steps: average tool call steps, outcome rewards, and rubric rewards.

Tool call steps (Figure 4, left). Both GRPO and C-GRPO show an initial sharp decline in average tool calls — from approximately 35 steps to 20–25 steps — as agents learn to avoid overlength rollouts that receive zero reward. After this initial efficiency phase, the trajectories diverge:

  • GRPO: Tool call steps continue to slowly decrease, settling around 18–20 steps by step 400. This is below the SFT baseline and represents the shortest search behavior.
  • C-GRPO: Tool call steps begin to recover and increase, reaching approximately 24 (4B) and 27 (30B) steps by step 400 — substantially above the SFT baseline.

The paper interprets the GRPO curve as evidence of shortcut learning: agents discover that they can achieve high outcome rewards by searching just enough to resolve the last few hops of a question, without verifying earlier constraints. The C-GRPO curve, by contrast, suggests that rubric incentives push agents to gather more evidence — each additional tool call is an opportunity to identify another hidden entity or find citation support for another rubric.

Outcome rewards (Figure 4, center). Both GRPO and C-GRPO show increasing outcome rewards throughout training, reaching approximately 0.55–0.65 by step 400. The curves are similar — C-GRPO achieves slightly higher outcome rewards at the 4B scale, while GRPO is slightly higher at the 30B scale. The key observation is that C-GRPO does not sacrifice outcome performance to achieve better process quality — the two objectives are not in tension at the final-answer level. This is consistent with the interpretation that C-GRPO finds a different, equally successful path to the correct answer, but one backed by more thorough evidence gathering.

Rubric rewards (Figure 4, right). This metric is tracked only for C-GRPO (since GRPO doesn't use rubric rewards). Rubric rewards increase from approximately 0.15–0.20 at the start of training to 0.35 for 4B and 0.42 for 30B by step 400. The steady increase indicates that agents are satisfying progressively more rubrics over time — identifying more hidden entities, supporting more claims with citations, and forming more complete evidence chains. The 30B model achieves higher absolute rubric rewards than the 4B model, consistent with the larger model's greater capacity for thorough search.

Comprehensiveness and Factuality Analysis

Table 2 provides a direct quantification of behavioral differences between models trained with different RL algorithms, evaluated on a subset of BrowseComp questions where all agents solved the queries within a 64k context length. The metrics are:

  • |C_H|: Average number of cited webpages in the agent's final response.
  • |R^identify_q|: Average number of fully-identified rubrics (Step 1 of CaRR: all entities explicitly named).
  • |R^support_q|: Average number of citation-supported rubrics (Step 2: claims backed by cited content).
  • |R^connect_q|: Average number of connected rubrics (Step 3: supported rubrics forming a chain to the answer).
  • |R_q|: Average total number of rubrics per question (constant at 10.1 across methods, since rubrics are fixed per question).

The results reveal a stark pattern:

  • DeepDive-30B-SFT: Cites 3.8 webpages, satisfies 4.5 connected rubrics out of 10.1 total on average.
  • + GRPO: Cites 3.5 webpages (fewer than SFT), satisfies 4.0 connected rubrics (fewer than SFT). Despite achieving higher answer accuracy (as shown in Table 1), GRPO training has reduced the agent's evidence-gathering behavior relative to the SFT starting point — the agent is finding correct answers through less thorough search.
  • + C-GRPO: Cites 4.3 webpages (more than both SFT and GRPO), satisfies 5.2 connected rubrics (more than both SFT and GRPO). C-GRPO training has increased evidence gathering relative to SFT, while also improving answer accuracy.

The gap between |R^support_q| and |R^connect_q| is also informative. For the SFT model, 8.0 rubrics are fully identified, 6.2 are citation-supported, but only 4.5 survive the connectivity check. This means that roughly 1.7 supported rubrics per question are satisfied in isolation — the entities were found and cited, but they don't connect to the answer through the evidence chain. C-GRPO reduces this gap: 8.2 identified, 6.6 supported, 5.2 connected — a larger fraction of supported rubrics pass the connectivity check, suggesting more coherent evidence gathering.

Generalization to Open-Ended Deep Research

Table 3 presents results on DeepResearch Bench, an open-ended research task where agents must produce PhD-level research reports evaluated by Gemini-2.5-Pro-preview across four dimensions: comprehensiveness, insight, instruction following, and readability. This evaluation tests whether skills learned from synthetic short-answer QA transfer to the very different setting of long-form research writing.

Overall scores (averaged across four dimensions):

  • 4B scale: C-GRPO achieves 37.51, compared to 34.79 for GRPO (+2.72) and 36.59 for E-GRPO (+0.92). The SFT baseline is 33.81, meaning C-GRPO provides a +3.70 improvement over SFT.
  • 30B scale: C-GRPO achieves 41.99, compared to 39.30 for GRPO (+2.69) and 36.12 for E-GRPO (+5.87 — notably, E-GRPO underperforms the SFT baseline of 37.51 at this scale). The improvement over SFT for C-GRPO is +4.48.

The 30B C-GRPO model's overall score of 41.99 is particularly notable because it outperforms several agents trained with proprietary data: Tongyi-DeepResearch (40.46) and Grok-Deeper-Search (38.22). It approaches but doesn't exceed Kimi-Researcher (44.64) and OpenAI-DeepResearch (46.45). The paper emphasizes this as evidence that C-GRPO's benefits — improved comprehensiveness, better factual grounding — transfer to open-ended research tasks.

Dimension-level analysis:

  • Comprehensiveness: C-GRPO shows the largest gains. At 30B, C-GRPO achieves 39.75 vs. GRPO's 36.10 (+3.65) and SFT's 34.27 (+5.48). This is consistent with C-GRPO incentivizing agents to cover more aspects of a question.
  • Insight: C-GRPO achieves 35.87 vs. GRPO's 31.66 (+4.21) at 30B. This is the largest relative improvement across dimensions, suggesting that comprehensive evidence gathering leads to deeper analytical insights.
  • Instruction following: Gains are smaller — C-GRPO achieves 48.51 vs. GRPO's 47.65 (+0.86) at 30B. This is expected since instruction following is less related to the rubric reward's focus on evidence quality.
  • Readability: C-GRPO achieves 46.63 vs. GRPO's 44.92 (+1.71) at 30B. Moderate improvement, likely because thorough evidence gathering produces better-structured reports.

The E-GRPO anomaly at 30B — where it underperforms even the SFT baseline (36.12 vs. 37.51) — is not discussed in detail by the paper but is consistent with the instability hypothesis: providing fine-grained rewards to incorrect trajectories may mislead optimization, especially for larger models where the rollout groups may have different success rate dynamics.

Ablation Studies and Robustness Checks

All ablations in this section use the 4B model and are reported at 128k context length in Table 5, unless otherwise noted.

  • Rubric reward weight \alpha (Table 4): Sweeping \alpha from 0 to 0.5 on the 4B model reveals a clear optimal value at \alpha = 0.3. At \alpha = 0 (equivalent to standard GRPO), accuracy is 14.7 / 17.5 / 41.3 / 41.1 on BrowseComp / BrowseComp-ZH / xbench-DS / GAIA. At \alpha = 0.1, performance is 13.0 / 18.0 / 46.0 / 46.3 — mixed results, with BrowseComp decreasing. At \alpha = 0.3, performance peaks at 17.5 / 24.7 / 54.0 / 50.2 — consistent gains across all benchmarks. At \alpha = 0.5, performance drops to 17.0 / 20.8 / 49.3 / 42.4 — still above GRPO on most benchmarks but substantially below the \alpha = 0.3 peak on BrowseComp-ZH, xbench-DS, and GAIA. This inverted-U shape supports the interpretation that a moderate rubric weight enhances training (by encouraging comprehensive reasoning) while an excessive weight distracts from the primary objective of answer correctness.

  • Hidden entity identification removal (Table 5, "w/o Hidden entity identification"): Eliminating Step 1 of CaRR — allowing the judge LLM to directly select supported rubrics without checking whether entities were explicitly named in the response — causes accuracy drops across all benchmarks: BrowseComp from 17.5 to 16.5 (-1.0), BrowseComp-ZH from 24.7 to 23.2 (-1.5), xbench-DS from 54.0 to 50.7 (-3.3), GAIA from 50.2 to 46.6 (-3.6). The larger drops on xbench-DS and GAIA suggest these benchmarks' questions require more explicit entity identification to verify reasoning completeness. This ablation validates that the entity identification step is not merely cosmetic — it provides a meaningful signal that shapes agent behavior toward more explicit, verifiable reasoning.

  • Evidence connectivity check removal (Table 5, "w/o Evidence connectivity check"): Eliminating Step 3 of CaRR — computing rubric reward as simply the fraction of supported rubrics (|R^support_q| / |R_q|) without requiring connection to the predicted answer — causes the largest performance drops of any ablation: BrowseComp from 17.5 to 15.1 (-2.4), BrowseComp-ZH from 24.7 to 20.8 (-3.9), xbench-DS from 54.0 to 47.7 (-6.3), GAIA from 50.2 to 44.0 (-6.2). These substantial degradations (especially on xbench-DS and GAIA, where drops exceed 6 points) confirm that the connectivity check is the most critical component of the verification pipeline. Without it, agents learn to "hack rubrics by finding entities that satisfy isolated factual statements but are unrelated to the final answer" — satisfying individual constraints without forming a coherent evidence trail.

  • Applying rubric rewards to all rollouts (Table 5, "w/ Rubric rewards for all rollouts"): Changing the C-GRPO reward formulation from R_i = (1-\alpha) \cdot R^H_o + \alpha \cdot R^H_o \cdot \hat{R}^H_r (rubric rewards only for correct trajectories) to R_i = (1-\alpha) \cdot R^H_o + \alpha \cdot \hat{R}^H_r (rubric rewards for all trajectories, regardless of correctness) causes catastrophic performance collapse: BrowseComp drops from 17.5 to 13.3 (-4.2), BrowseComp-ZH from 24.7 to 14.0 (-10.7), xbench-DS from 54.0 to 40.3 (-13.7), GAIA from 50.2 to 40.8 (-9.4). Performance falls below the GRPO baseline (\alpha = 0) on BrowseComp and BrowseComp-ZH, and roughly matches GRPO on xbench-DS and GAIA — meaning the rubric reward signal has become actively harmful rather than helpful. The paper attributes this to optimization instability: early in RL, when most rollouts are incorrect, normalizing rubric rewards within groups can assign positive advantages to incorrect trajectories that happen to satisfy a few rubrics, steering the policy toward partially-correct but ultimately wrong behavior.

  • Judge LLM reliability (Appendix C): To validate that the judge LLM's assessments are trustworthy, the authors conducted a manual review of 128 hidden entity identifications and 164 citation-based rubric judgments across 10 DeepDive-30B-SFT trajectories. Using human assessments as ground truth, the judge LLM (DeepSeek-v3.2) achieved 97.7% accuracy for entity identification and 95.1% accuracy for citation-based rubric judgment. These high agreement rates suggest that the rubric reward signal, while derived from an LLM rather than ground truth, is reliable enough to serve as a training signal. However, the evaluation was conducted on SFT trajectories rather than RL trajectories; as the policy changes during RL, the judge LLM's accuracy could theoretically degrade if the agent's outputs drift out of distribution relative to what the judge was validated on. The paper does not address this potential distribution shift.

Critical Assessment

The paper makes three central claims that the experiments must support: (1) C-GRPO with CaRR outperforms outcome-only GRPO across deep search benchmarks, (2) this improvement is attributable to the rubric reward discouraging shortcut exploitation and promoting comprehensive, evidence-grounded reasoning, and (3) the benefits generalize to open-ended deep research tasks beyond the synthetic QA training distribution.

Claim 1 (C-GRPO outperforms GRPO): The evidence for superior benchmark performance is strong and consistent across scales and benchmarks. Table 1 shows C-GRPO exceeding GRPO on all four benchmarks at both 4B and 30B scales, at both 64k and 128k context budgets — 16 out of 16 comparisons. The margins are substantial (averaging 2.6–8.0 percentage points depending on scale and budget), and the pattern holds against E-GRPO as well. The \alpha sweep (Table 4) shows that the improvement is specifically attributable to the rubric reward component, not to some other aspect of the training setup — at \alpha = 0, C-GRPO is identical to GRPO, and performance increases with \alpha up to 0.3.

However, several aspects of the experimental design limit the strength of this conclusion:

  • Single model family. All experiments use Qwen3 models. The paper argues these are "representative," but without replication on other architectures (e.g., LLaMA, DeepSeek, Gemini), it's unknown whether C-GRPO's benefits depend on Qwen3-specific properties like the built-in "Thinking" mode or the particular training data distribution.
  • Single training dataset. All RL uses DeepDive. While this is a reasonable choice for controlled experimentation, it means the rubric reward's effectiveness is demonstrated only for synthetic knowledge-graph-derived questions. Real-world multi-hop questions may have different structural properties.
  • No confidence intervals. The paper reports point estimates without statistical significance testing. For the smaller benchmarks (BrowseComp-ZH, xbench-DS, GAIA), 3 evaluation runs are averaged, but no standard deviations or confidence intervals are reported, making it impossible to assess whether the observed differences (e.g., 54.0 vs. 41.3 on xbench-DS at 4B-128k) are statistically reliable or within the range of evaluation noise.
  • Judge LLM is DeepSeek-v3.2 for both training and evaluation. The same model family (DeepSeek) serves as the judge for outcome rewards during training and for rubric reward computation. While the evaluation judge is GPT-5-Chat (a different model), the training signal comes from DeepSeek-v3.2. If DeepSeek-v3.2 has systematic biases in how it evaluates answer correctness or citation support, those biases are baked into the training objective and could produce policies that perform well against the training judge but not necessarily against a different evaluator.

Claim 2 (Rubric rewards discourage shortcuts and promote evidence-grounded reasoning): The evidence for the mechanism — that the performance improvement comes specifically from discouraging shortcut behaviors — is moderately strong but relies on indirect behavioral metrics.

The strongest evidence is Table 2, which directly measures evidence-gathering behavior: C-GRPO substantially increases cited webpages (4.3 vs. 3.5 for GRPO) and connected rubrics (5.2 vs. 4.0 for GRPO). This is not just a correlation — it's a direct measurement of the behavior the rubric reward is designed to incentivize, and it moves in the expected direction.

The training dynamics (Figure 4) provide converging evidence: C-GRPO's tool call steps increase during training while GRPO's decrease, consistent with the claim that C-GRPO learns to search more thoroughly. However, tool call count is a noisy proxy for reasoning quality — an agent could make many tool calls without actually finding useful information, or could make few tool calls but be highly effective. The paper's case studies (Appendix D, Figures 7–10) provide qualitative validation: the GRPO agent produces a response that correctly answers the question but ignores the first several hops (Case 1), while the C-GRPO agent systematically verifies each constraint with citations (Case 2). These are compelling illustrations but are selected examples, not systematic analysis.

The test-time scaling results (Figure 3) provide indirect evidence: GRPO's degradation at 128k while C-GRPO continues improving is consistent with GRPO having learned brittle shortcut policies that don't scale with additional compute. However, this could also be explained by other factors — for example, C-GRPO might simply be trained with a more diverse reward signal that leads to better generalization, without specifically being about "shortcut exploitation."

Weaknesses in the evidence for Claim 2:

  • No direct measurement of shortcut frequency. The paper doesn't report how often GRPO vs. C-GRPO trajectories exhibit shortcut behaviors (e.g., what fraction of constraints are resolved). Table 2 reports averages but not distributions — it's possible that C-GRPO's higher mean is driven by a few very thorough trajectories while the median behavior is similar to GRPO.
  • The connectivity check metric in Table 2 is computed post-hoc using CaRR itself. The rubric reward used during training is the same metric used to evaluate evidence quality at test time. This creates a form of metric circularity: C-GRPO is optimized to maximize this metric, and we then use this metric to show that C-GRPO produces better evidence. The metric is meaningful (it captures genuine aspects of reasoning quality), but the magnitude of improvement on this metric is partially an artifact of the training objective.
  • Case studies are illustrative but could be cherry-picked. The paper presents two positive examples for C-GRPO and two negative examples for GRPO. A more systematic analysis — e.g., randomly sampling 50 trajectories from each model and evaluating them against human judgments of evidence quality — would provide stronger evidence that the behavioral improvements are systematic rather than anecdotal.

Claim 3 (Benefits generalize to open-ended deep research): The DeepResearch Bench results (Table 3) provide supportive but incomplete evidence for generalization.

The positive evidence: C-GRPO outperforms GRPO across all four dimensions at both 4B and 30B scales, with overall improvements of +2.72 (4B) and +2.69 (30B). The 30B C-GRPO model even surpasses some proprietary-data agents (Tongyi-DeepResearch, Grok-Deeper-Search). This suggests that the skills incentivized by CaRR — comprehensive evidence gathering, factual grounding, coherent evidence chains — transfer to the very different format of research report writing.

Weaknesses in the evidence for Claim 3:

  • No direct connection between rubric satisfaction and DeepResearch Bench scores demonstrated. The paper doesn't analyze whether C-GRPO agents actually exhibit more rubric-like behaviors (e.g., citing more sources, covering more aspects of the research question) when writing DeepResearch reports. The improvement could be due to a general improvement in the agent's language capabilities from additional RL training rather than specifically from the rubric reward mechanism.
  • DeepResearch Bench evaluation uses a single judge (Gemini-2.5-Pro-preview). LLM-as-judge for long-form research reports is known to be noisy and potentially biased. Without human evaluation or multi-judge agreement, the reliability of the reported scores is uncertain. The paper doesn't report inter-judge agreement or compare Gemini's judgments to human assessments.
  • The SFT baseline on DeepResearch Bench is already competitive (37.51 at 30B), and C-GRPO's improvement (+4.48) is proportionally smaller than on the deep search benchmarks. This could indicate that the benefits of process supervision are larger for tasks closely matching the training distribution (synthetic QA) and diminish for more distant transfer tasks.
  • E-GRPO underperforms SFT at 30B on DeepResearch Bench (36.12 vs. 37.51), which is a concerning result that the paper doesn't explain or discuss. It suggests that some forms of fine-grained reward can actually hurt open-ended performance, even while improving QA accuracy.

Missing experiments that would strengthen the paper:

  • Ablation on the rubric initializer quality. All rubrics are generated by DeepSeek-v3.2. How sensitive is C-GRPO to the quality of rubric decomposition? Would weaker rubric generation (e.g., using a smaller model, or introducing noise into the rubrics) still provide benefits? This is critical for practical deployment, since rubric generation is a non-trivial step.

  • Human evaluation of evidence quality. The paper uses LLM judges for both training and evaluation. A human study comparing the evidence quality of GRPO vs. C-GRPO trajectories (e.g., having humans rate whether the agent's claims are genuinely supported by citations, whether the reasoning is complete) would provide stronger validation that the improvements measured by the rubric metric correspond to improvements in human-judged quality.

  • Scaling curve for rubric reward weight vs. model size. The optimal \alpha = 0.3 was found on the 4B model and presumably used for the 30B model as well. Does the optimal \alpha change with model scale? Larger models might benefit from different tradeoffs between outcome and rubric rewards.

  • Analysis of failure modes. The paper presents cases where C-GRPO succeeds and GRPO fails, but doesn't systematically analyze cases where C-GRPO fails. Understanding whether C-GRPO has its own characteristic failure modes (e.g., over-searching, spending too many tool calls on irrelevant constraints) would provide a more balanced picture of the method's limitations.

  • Ablation on the judge LLM identity. All rubric computations use DeepSeek-v3.2. What happens if a different LLM (e.g., GPT-4, Claude) is used for rubric judgment during training? This would test whether the benefits are tied to specific judge LLM properties or are robust to judge variation.

  • Direct comparison to a length-penalized GRPO baseline. C-GRPO's tool call steps increase during training while GRPO's decrease. Could simply adding a length penalty to GRPO (rewarding trajectories that use more tool calls, or penalizing very short trajectories) achieve similar benefits without the complexity of rubric rewards? This ablation would distinguish the value of rubrics specifically from the value of simply encouraging more search.

Conditional nature of the claims: The paper's claims hold within the specific experimental paradigm of synthetic multi-hop QA training data, Qwen3 models, DeepSeek-v3.2 as judge, and the four evaluation benchmarks. The claims about discouraging shortcut exploitation are supported by behavioral metrics and case studies but rely on the specific decomposition of questions into rubrics that is possible for synthetic knowledge-graph-derived data. The generalization to open-ended research is demonstrated on one benchmark (DeepResearch Bench) with one judge (Gemini-2.5-Pro-preview) and shows meaningful but proportionally smaller gains than on the deep search benchmarks. The strongest evidence is for the narrowest claim: C-GRPO improves performance on deep search benchmarks when training on synthetic multi-hop QA data, and this improvement is accompanied by behavioral changes consistent with more thorough evidence gathering. The broader claim — that C-GRPO produces "robust deep search agents" in a general sense — is supported directionally but would require testing across more diverse training distributions, model families, and evaluation settings to be fully substantiated.

6. Limitations and Trade-offs

The Rubric Decomposition Assumes Synthetic Multi-Hop Questions with Well-Defined Intermediate Entities

The assumption or constraint. The CaRR framework relies fundamentally on the ability to decompose each training question into a list of atomic single-hop factual statements with explicitly labeled hidden entities. This decomposition is feasible because the training data—DeepDive—is synthetically generated from knowledge graph random walks, meaning that every question has a known compositional structure: each hop corresponds to a specific edge in the knowledge graph, and the intermediate entities are well-defined even though they are obfuscated in the question text. The paper explicitly acknowledges this dependency in Section 6:

"rubric generation relies on the compositional structure of synthetic multi-hop questions, and may not be able to be directly adapted to open-ended QA training where some requirements are not explicitly stated in the question."

This is not merely a detail about one dataset—it reflects a structural coupling between the reward mechanism and the data generation process. The three-step verification procedure (entity identification → citation support → connectivity check) is designed around the properties of entity-centric, graph-derived questions: each rubric involves known hidden entities, the rubrics form a connected subgraph by construction, and the final answer is typically a single entity string that serves as the root of the connectivity BFS.

The consequence. The approach cannot be directly applied to training data that lacks this compositional structure, which includes virtually all natural QA datasets and most open-ended information-seeking tasks. Real-world complex questions—"What were the economic consequences of the 2008 financial crisis on emerging markets?" or "Compare the approaches to pandemic response in Sweden and South Korea"—do not decompose cleanly into a list of entity-linked single-hop factoids with well-defined connectivity graphs. An attempt to run the rubric initializer on such questions would produce rubrics but likely without the clean entity-placeholder structure and guaranteed connectivity that CaRR's verification pipeline assumes. The judge LLM's high accuracy (97.7% for entity identification, 95.1% for citation judgment; Appendix C) was measured on synthetic questions—it is unknown whether these rates would hold for more ambiguous, open-ended decompositions. If the rubric structure is unreliable, the training signal becomes noisy, and the C-GRPO advantage may diminish or reverse.

This limitation is practically consequential because the dominant trend in deep search agent deployment—as exemplified by the proprietary agents in Table 1 (OpenAI-DeepResearch, Kimi-Researcher, Tongyi-DeepResearch)—is toward handling genuinely open-ended research questions, not synthetic multi-hop puzzles. The paper demonstrates that C-GRPO-trained agents generalize to DeepResearch Bench (Table 3), but the training still requires synthetic data. A lab wanting to train a search agent on their own domain-specific question corpus (medical literature, legal research, financial analysis) cannot adopt CaRR unless their questions happen to have the same entity-centric compositional structure as DeepDive.

What evidence exists in the paper. Section 6 acknowledges the limitation explicitly. The DeepResearch Bench results (Table 3) provide indirect evidence that the limitation is not fatal—behaviors learned from synthetic rubric-based training do transfer to open-ended research—but do not address whether CaRR itself could be used for training on open-ended data. There is no ablation or experiment testing CaRR with non-synthetic training data. The judge LLM validation (Appendix C) was conducted on SFT trajectories from synthetic questions, providing no signal about rubric quality on natural questions.

Mitigation status. The paper does not attempt to extend CaRR to non-synthetic training data. It offers a partial defense: "both our work and previous works have shown that the synthetic, short-form question answering is an effective proxy for open-ended deep research tasks since they share the core requirement for long-horizon information-seeking capacity" (Section 6). This is an argument about transfer learning, not about the generality of the method itself. The paper suggests no concrete approach for adapting rubric generation to natural questions—for instance, using the LLM to infer which entities are "hidden" from unstructured text, or relaxing the connectivity requirement to a softer similarity-based metric. The limitation remains unresolved and represents a genuine barrier to adoption of CaRR as a general-purpose training framework.

The Cost of Rubric Reward Computation Is Not Included in Training Budget Analysis

The assumption or constraint. Every C-GRPO training step requires computing rubric rewards for all 128 trajectories in the global batch (8 samples × 16 prompts). This computation involves:

  1. Hidden entity identification (Step 1): The judge LLM must process each agent's final response against the question, all rubrics, and all hidden entity placeholders—generating structured JSON output with one entry per entity. For a question with 10 hidden entities, this is a non-trivial inference call.

  2. Citation-based rubric judgment (Step 2): For each fully-identified rubric (potentially 5–10 per trajectory), the judge LLM must evaluate whether the cited web content supports the factual claim. This requires feeding the judge LLM the collected web content from up to 20 cited URLs (Equation 8, truncated at 20 to prevent reward hacking), which can be thousands of tokens per trajectory.

  3. Evidence connectivity check (Step 3): This step is algorithmic (BFS on a bipartite graph) and likely cheap. But Steps 1 and 2 involve substantial LLM inference.

The paper uses DeepSeek-v3.2 as the judge LLM for both outcome and rubric rewards (Section 3.1). DeepSeek-v3.2 is a large, capable model—its inference cost per token is non-trivial. The training configuration specifies a global batch size of 128 trajectories and 3 epochs over 2,234 training questions, meaning the judge LLM is called at least 3 × (2,234/16) × 128 ≈ 53,616 times for rubric reward computation (assuming 16 prompts per batch, each batch requiring 128 rubric calls). Each call involves multiple complex prompts (Figures 13 and 14 in Appendix E), making the total judge LLM compute substantial.

The paper does not account for this cost in any budget calculation. The training section reports: rollout size 16, 8 samples per prompt, learning rate 2e-6, maximum context 64k—but the cost of the judge LLM is treated as external infrastructure, not part of the training compute budget.

The consequence. The headline comparison between C-GRPO and GRPO is cost-asymmetric. GRPO requires only outcome reward computation (a single judge LLM call per trajectory to check answer correctness), while C-GRPO requires outcome reward computation plus the full three-step rubric verification pipeline. The paper reports that C-GRPO achieves +5.1 to +8.0 percentage points over GRPO (Section 3.2), but does not report the additional compute cost incurred to achieve those gains. A practitioner deciding whether to adopt C-GRPO needs to know: is the performance improvement worth the additional judge LLM inference cost? The paper provides no data to answer this question.

This is analogous to the "difficulty estimation cost" problem in the test-time compute scaling literature: if the mechanism for improving performance itself consumes significant compute, the net efficiency gain may be smaller than the headline numbers suggest, or even negative in some regimes. For C-GRPO, the additional cost is incurred during training (not inference), so the training-to-inference amortization matters. If the trained agent will serve millions of queries, the one-time training cost of judge LLM inference is negligible. But if the goal is to rapidly iterate on training recipes (as is common in research settings), the per-step cost of C-GRPO may be prohibitive.

What evidence exists in the paper. There is no measurement of judge LLM cost in the paper. No token counts for rubric computation prompts, no latency measurements, no FLOPs or dollar-cost estimates. Section 3.1 mentions that DeepSeek-v3.2 is used "as the judge LLM for both outcome rewards and rubric rewards" but provides no cost analysis. The paper does not compare training wall-clock time between GRPO and C-GRPO. The judge LLM human verification study (Appendix C) measures accuracy (97.7%, 95.1%) but not cost.

Mitigation status. The paper does not acknowledge this as a limitation, nor does it suggest strategies for reducing judge LLM cost. Potential mitigations—using a smaller model for rubric judgment, caching rubric computations across similar questions, batching judge LLM calls, or using the policy model itself as the judge (self-verification)—are not discussed. The limitation remains entirely unaddressed.

The Method Has Been Validated on Only One Model Family and One Training Dataset

The assumption or constraint. All experiments use Qwen3 models (4B dense and 30B MoE, both "Thinking" variants) and the DeepDive training dataset. The paper states in Section 3.1 that these models are "covering different model sizes and architectures (dense and MoE)," implying that the results should generalize. But "different architectures" here means two variants within the same model family—same pretraining data, same tokenizer, same training recipe, same "Thinking" fine-tuning approach. This is not a test of cross-family generalization.

Specific properties of Qwen3 that could interact with C-GRPO's effectiveness include:

  • The "Thinking" variants include built-in chain-of-thought prompting that may affect how the model structures its search reasoning, potentially making it more receptive to rubric-based feedback that rewards explicit entity identification.
  • Qwen3's pretraining data mixture may affect its web search and information synthesis capabilities in ways that interact with the reward signal.
  • The tokenizer and context window behavior may affect how trajectories fit within the 64k training budget and 128k evaluation budget.

Similarly, DeepDive is one specific synthetic dataset generated through a particular knowledge graph (the paper doesn't specify which), with a particular obfuscation strategy, a particular question complexity distribution, and a particular answer format (short-form entity strings). The rubric decomposition quality, the difficulty distribution across questions, and the nature of hidden entity obfuscation are all dataset-specific parameters that could affect C-GRPO's relative advantage over GRPO.

The consequence. A practitioner using a different base model (e.g., LLaMA, DeepSeek, Gemma, Claude via API fine-tuning) or a different training dataset (e.g., WebSailor's data, BrowseComp-derived training questions, a custom domain-specific corpus) cannot assume the reported ~5–8 point improvement will replicate. The improvement could be larger (if Qwen3 is somehow less receptive to outcome-only RL than other models) or smaller (if the specific properties of Qwen3 or DeepDive are necessary for the rubric reward to be effective).

Specific failure modes that could arise with different model families:

  • A model with weaker in-context learning capabilities might struggle to follow the complex multi-turn trajectory format, reducing the effective training signal regardless of reward design.
  • A model with stronger parametric knowledge might rely more on memorized facts and less on web search, making the citation-based rubric judgment less meaningful (the model "knows" the answer and cites sources pro forma).
  • A model with different "Thinking" behavior might produce final responses that are less amenable to entity extraction (Step 1 of CaRR), degrading the rubric reward signal quality.

Similarly, with a different training dataset:

  • Questions with fewer hops (simpler structure) might show smaller C-GRPO benefits because there are fewer opportunities for shortcut exploitation—outcome-only RL might already produce thorough reasoning on simple questions.
  • Questions without the entity-placeholder structure might require a fundamentally different rubric initialization procedure.
  • Questions with longer-form answers might make outcome verification harder, increasing the relative importance of the rubric reward signal.

What evidence exists in the paper. The paper provides no cross-model or cross-dataset validation. All 16 main result comparisons (Table 1: 2 model scales × 4 benchmarks × 2 context budgets) are within the Qwen3 + DeepDive combination. The DeepResearch Bench evaluation (Table 3) tests generalization to a different task but still uses the same Qwen3 models trained on the same DeepDive data. The paper does not include experiments with, for example, Qwen3 trained on WebSailor data, or LLaMA trained on DeepDive data, or any other cross-factor combination.

Mitigation status. The paper does not explicitly claim generalization across model families or datasets. It presents the results as evidence for C-GRPO's effectiveness in the tested configuration and leaves broader validation to future work. The limitation is unacknowledged—the paper does not discuss the single-model-family and single-dataset scope as constraints on the conclusions. A cautious reading suggests the results should be understood as "C-GRPO works well for Qwen3 models trained on DeepDive-style synthetic data" rather than "C-GRPO is a general improvement over GRPO for deep search agent training."

Training at 64k Context with Evaluation at 128k Reveals a Brittleness in GRPO That C-GRPO Partially Mitigates—But the Origin of This Brittleness Is Not Fully Diagnosed

The assumption or constraint. The paper makes a deliberate and interesting methodological choice: train RL at 64k context length (a constrained budget) and evaluate at both 64k and 128k (an extended budget). The results (Table 1, Figure 3 middle) show that GRPO-trained agents degrade or stagnate at 128k while C-GRPO agents continue to improve. The paper interprets this as evidence that outcome-only RL produces shortcut policies that are brittle when given more compute, while rubric-based RL produces more scalable policies.

However, the paper does not establish which specific aspect of C-GRPO causes the improved test-time scaling. Several mechanisms could explain the pattern, and they have different practical implications:

  • Hypothesis A (the paper's interpretation): C-GRPO produces policies that have learned to keep searching until constraints are satisfied, so additional context naturally enables more thorough verification. GRPO produces policies that stop as soon as an answer is found, so additional context is unused.

  • Hypothesis B (length bias): C-GRPO's training dynamics show increasing tool call steps (Figure 4, left), meaning C-GRPO agents use more of the context budget during training. They are simply better adapted to operating at longer effective context lengths because they've practiced it more. GRPO agents, optimizing for efficiency, rarely approach the context limit during training and therefore don't learn to effectively use additional context at test time.

  • Hypothesis C (reward shaping): C-GRPO's mixed reward includes a dense signal (rubric satisfaction) that provides learning even when the outcome reward is saturated (the answer is correct). This denser signal may lead to more robust policy optimization that generalizes better under distribution shift (from 64k to 128k context). GRPO's sparse binary reward provides less signal per correct trajectory, potentially leading to overfitting to the 64k context regime.

  • Hypothesis D (exploration): The rubric reward encourages agents to explore more diverse search strategies (to satisfy different rubrics), leading to a broader learned policy distribution that generalizes better. GRPO, optimizing only for answer correctness, may converge to a narrow set of strategies that work at 64k but fail at 128k.

These hypotheses are not mutually exclusive, but they have different implications for practitioners. If Hypothesis B is the main driver, then the benefit of C-GRPO could be replicated by simply training GRPO at a longer context length or with a length bonus. If Hypothesis A is correct, then the rubric structure itself is necessary. The paper's experiments do not distinguish between these mechanisms.

The consequence. A practitioner trying to decide whether to adopt C-GRPO based on the reported test-time scaling benefits cannot assess what they would need to change in their own setup to achieve similar benefits. If the key factor is simply "train with longer trajectories" (Hypothesis B), they could achieve test-time scaling improvements with standard GRPO by increasing the training context budget—a much simpler change than implementing the full CaRR pipeline. If the key factor is "provide process-level feedback" (Hypothesis A), then the rubric mechanism is necessary and the investment is justified.

The paper's interpretation favors Hypothesis A based on the behavioral metrics (Table 2: C-GRPO agents cite more sources and satisfy more rubrics) and case studies. But these metrics are outcomes of the training process, not mechanisms. It remains possible that both the increased tool calls and the improved test-time scaling are downstream effects of some other aspect of C-GRPO training (e.g., the denser reward signal, the higher effective context usage during training), and that the rubric content itself is less important than the fact that a dense reward exists.

What evidence exists in the paper. The training dynamics (Figure 4) show that C-GRPO's tool calls increase while GRPO's decrease—consistent with C-GRPO operating at longer effective context lengths during training (Hypothesis B). But the paper does not report the distribution of trajectory lengths during training for each method, so the magnitude of this difference is unclear. The paper does not include an ablation where GRPO is trained with a length bonus or at 128k context to test whether this alone improves test-time scaling. The DeepResearch Bench results (Table 3) show C-GRPO generalizing to a different task, which is more consistent with Hypothesis A or C (a genuine improvement in reasoning quality) than with Hypothesis B (a context-length artifact), but this is suggestive rather than conclusive.

Mitigation status. The paper does not discuss alternative explanations for the test-time scaling results. The limitation is unacknowledged—the paper presents the scaling behavior as straightforward evidence of C-GRPO producing "more robust" policies without analyzing the mechanism. This is a gap in the experimental analysis, not a flaw in the method itself, but it limits the strength of the paper's causal claims about why C-GRPO works.

The Connectivity Check Enforces a Graph Structure That May Be Too Strict for Partially Correct Reasoning

The assumption or constraint. Step 3 of CaRR—the evidence connectivity check—requires that supported rubrics be connected to the predicted answer entity e^H_0 via a path in the bipartite entity-rubric graph (Equations 10–11). Rubrics that are fully identified and citation-supported but not reachable from the answer entity receive zero credit. The final rubric reward is |R^connect_q| / |R_q| (Equation 12), meaning that satisfying 5 out of 10 rubrics in the correct region of the graph counts for more than satisfying 8 out of 10 rubrics where 4 are disconnected.

This assumption encodes a specific view of what constitutes good reasoning: the evidence must form a single coherent chain from the answer backward through all supporting constraints. A trajectory that identifies entities and supports facts for 80% of rubrics—but does so across two disconnected components (e.g., it correctly resolves the first 5 hops and the last 3 hops of a 10-hop question, but misses the 2 connecting hops)—would receive a rubric reward of approximately 0.3 (only the 3-hop component connected to the answer), not 0.8.

The consequence. The connectivity check may penalize legitimate partial progress that doesn't happen to form a connected subgraph. In complex multi-hop questions, an agent might correctly resolve two distinct aspects of the question without establishing the connection between them—but this is still more thorough reasoning than an agent that resolved only the final hops. By assigning zero credit to disconnected-but-correct rubrics, C-GRPO may discard useful learning signal about partially successful search strategies.

This is particularly relevant early in RL training, when agents are unlikely to produce fully connected evidence chains. The paper reports (Figure 4, right) that C-GRPO's rubric reward starts around 0.15–0.20 and rises to 0.35–0.42 over 400 steps. But this average masks the distribution: if many early trajectories have 4–5 supported rubrics but none are connected (rubric reward = 0), the agent receives no positive reinforcement for those rubrics, potentially slowing learning. In contrast, if the connectivity check were softened—e.g., by giving partial credit based on the size of the largest connected component, or by treating connectivity as a continuous rather than binary property—the agent might receive more informative feedback about which search strategies are partially effective.

The ablation (Table 5, "w/o Evidence connectivity check") shows that removing the connectivity check entirely hurts performance—agents learn to game the reward by finding isolated rubrics. But this doesn't rule out the possibility that a softer connectivity check (e.g., weighted by component size, or applied only to a threshold fraction of rubrics) could capture the benefits of connectivity enforcement while avoiding the harshness of the all-or-nothing binary check. The paper tests only the extremes: full connectivity enforcement (default) vs. no connectivity enforcement (ablation).

What evidence exists in the paper. The connectivity check ablation (Table 5) demonstrates that the binary connectivity check is better than no connectivity check—removing it causes performance drops of 2.4–6.3 points across benchmarks. But this does not establish that the binary check is optimal among possible connectivity formulations. The training dynamics (Figure 4, right) show that rubric rewards increase slowly, consistent with the connectivity check being a challenging constraint that takes many steps to satisfy. The paper does not report the fraction of trajectories that receive zero rubric reward due to connectivity failure, nor does it analyze whether the connectivity requirement becomes easier to satisfy as training progresses (which would be expected if agents learn to form more coherent evidence chains).

Mitigation status. The paper does not discuss alternatives to the binary connectivity check. It does not acknowledge the potential tension between providing dense learning signals (which favors giving partial credit) and preventing reward hacking (which favors strict connectivity enforcement). The limitation is unacknowledged and represents a design choice whose optimality is assumed rather than validated empirically. A more thorough analysis would sweep over connectivity formulations (binary, component-size-weighted, threshold-based) and measure both final performance and learning speed to determine the best tradeoff between signal density and reward robustness.

The 38% Correct-to-Incorrect Reversion Rate from Prior Work Suggests a Deeper Issue with Sequential Revision That C-GRPO Inherits but Does Not Address

The assumption or constraint. This limitation concerns a subtle interaction between C-GRPO and the sequential nature of deep search agent trajectories. The ReAct paradigm (Section 2.1) requires agents to generate multiple tool-calling steps before producing a final answer. Each step involves a thought (\tau_t) and an action (a_t), and the trajectory accumulates all observations. The C-GRPO reward is computed from the final response a_T—the entity identification, citation support, and connectivity are all evaluated based on the agent's concluding output, not on intermediate reasoning steps.

This means that an agent could, during the search process, identify correct entities and find supporting evidence, but then fail to include them in the final response—perhaps because the final response synthesis step omits some findings, or because the agent changes its answer during the trajectory and discards earlier correct work. The CaRR framework provides no reward for intermediate reasoning quality; only what appears in the final response counts. Similarly, an agent could identify an entity early in the trajectory but later "revise" it to an incorrect entity in the final response, losing rubric credit for the earlier correct identification.

This is not directly demonstrated in the current paper's experiments—the paper does not analyze whether C-GRPO trajectories exhibit a gap between intermediate reasoning quality and final response quality. However, it is structurally analogous to a well-documented problem in the sequential revision literature that the paper should be aware of: the correct-to-incorrect reversion problem, where models that generate multi-step revisions sometimes "revise" a correct intermediate answer into an incorrect final answer.

The consequence. C-GRPO may under-reward trajectories where the agent conducted thorough search but produced a poorly synthesized final response, and over-reward trajectories where the agent searched poorly but produced a well-written final response that happened to mention the right entities. The rubric reward signal is only as good as the final response's reflection of the actual search process. If the agent's final response writing capability is a bottleneck—if it can find evidence but can't coherently synthesize it—C-GRPO's rubric reward will be suppressed regardless of search quality, and the RL optimization may waste capacity improving response writing rather than search strategy.

Conversely, an agent could learn to produce final responses that appear comprehensive (mentioning many entities, citing many sources) without having conducted thorough search—for instance, by fabricating entity names that sound plausible, or by citing sources without actually reading them (the citation check only verifies that the cited content supports the claim; it doesn't verify that the agent read that content during the trajectory). The citation check (Step 2) partially mitigates this by verifying against actual web content, but it cannot verify that the agent genuinely derived its knowledge from those sources rather than from parametric knowledge supplemented with post-hoc citations.

What evidence exists in the paper. The paper does not analyze the relationship between intermediate search quality and final response quality. The hidden entity identification step (Step 1) is applied only to the final response a_T, not to intermediate thoughts. There is no experiment comparing C-GRPO's performance when rubric rewards are computed from intermediate steps vs. from the final response. The judge LLM reliability study (Appendix C) validates entity identification and citation judgment accuracy but does not address whether the final response faithfully represents the search process. The paper includes trajectory examples in Appendix D (Figures 8 and 10) that show C-GRPO agents producing thorough final responses, but these are selected positive examples and don't reveal whether there exist trajectories where the agent searched well but wrote a poor final response.

Mitigation status. The paper does not acknowledge this as a limitation. The CaRR framework implicitly assumes that the final response is an adequate summary of the search process. In practice, this assumption may not hold—particularly for smaller models (4B) or early in training, where response synthesis capabilities may lag behind search capabilities. Potential mitigations—such as computing rubric rewards from intermediate tool call outputs, or adding an auxiliary objective that rewards consistency between search actions and final claims—are not discussed. The limitation remains unacknowledged and represents a potential source of noise in the rubric reward signal that could reduce training efficiency, particularly for smaller models or in domains where response synthesis is challenging.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a methodological reframing with practical consequences rather than a paradigm shift. It does not introduce a new model architecture, a new training objective, or a new class of algorithms. What it does is change how the field should think about the optimization target when training deep search agents with RL—from "correctness of the final answer" to "comprehensiveness and factual grounding of the evidence trail that leads to that answer." This reframing is supported by two empirical findings that together shift the conversation:

  1. Outcome-only RL actively degrades reasoning quality even as it improves answer accuracy. The evidence in Table 2 is stark: a GRPO-trained 30B model cites fewer webpages (3.5 vs. 3.8) and satisfies fewer connected rubrics (4.0 vs. 4.5) than the SFT model it was trained from. RL made the agent better at producing the right answer string but worse at building comprehensive, well-supported arguments. This is a concrete instance of Goodhart's Law in agent training—optimizing the proxy metric (answer match) undermined the true objective (reliable, evidence-grounded search)—and it establishes that process supervision is not merely "nice to have" but is necessary to prevent regressions in agent behavior during RL.

  2. Shortcut exploitation is the default outcome of outcome-only RL, not a rare failure mode. The training dynamics (Figure 4, left panel) show that GRPO's tool call steps decline steadily throughout training after an initial efficiency phase—agents are learning to do less search, not more. The case studies (Appendix D, Figures 7 and 9) reveal the mechanism: agents resolve the last few hops of a multi-hop question and guess the rest. This is rational behavior under a binary reward function, and it means that any outcome-only RL pipeline for search agents should be assumed to produce shortcut policies unless proven otherwise. C-GRPO provides a concrete, validated mechanism for proving otherwise.

These findings reconcile a latent tension in the literature. Prior work on RL for search agents (Jin et al., 2025a; Gao et al., 2025; Li et al., 2025b; Lu et al., 2025) has demonstrated performance gains from outcome-based training. This paper does not contradict those gains—indeed, its own GRPO baseline improves over SFT at 64k context (Table 1). But it reveals that those gains come at the cost of behavioral quality that is invisible to benchmark accuracy metrics but manifests as brittleness under distribution shift (the 128k context degradation in Figure 3, middle) and as weaker generalization to open-ended tasks (Table 3 shows GRPO trailing C-GRPO on DeepResearch Bench by 2.7–2.7 points). The implication is that benchmark accuracy alone is insufficient to evaluate search agent training—behavioral metrics like citation frequency, constraint satisfaction, and test-time scaling behavior must be monitored.

This reframing makes several research directions newly salient:

  • Verifier quality as the primary bottleneck for search agent RL. The paper demonstrates that process rewards work when the verification pipeline is carefully designed (three-step with connectivity), but the verification itself depends on LLM judge quality (97.7% entity identification accuracy, 95.1% citation judgment accuracy; Appendix C). Improving judge reliability—through better prompting, multi-judge ensembles, or fine-tuned verifier models—is now directly on the critical path to better search agent training. The paper's success with DeepSeek-v3.2 as judge provides a starting point, but the field needs systematic studies of how judge quality affects downstream policy quality.

  • The death of "just use outcome rewards" as a default. Prior to this paper, the dominant approach was to treat outcome rewards as the obvious choice for simplicity and scalability, with process rewards seen as a potential future enhancement. After this paper, the burden of proof shifts: a practitioner proposing outcome-only RL for search agents should be expected to demonstrate that their specific setup does not suffer from the shortcut exploitation and behavioral degradation that this paper documents. The default should be some form of process supervision, with outcome-only RL requiring explicit justification.

  • Training dynamics monitoring as a standard practice. The paper's diagnostic approach—tracking tool call steps, outcome rewards, and rubric rewards simultaneously during training (Figure 4)—reveals policy divergence that is invisible to the outcome reward curve alone. This methodology should become standard in search agent RL research. It is inexpensive (logging metrics during training), immediately informative, and can catch pathological behaviors (like declining tool call counts) before they crystallize into final policies.

A direction that becomes less attractive after this work is the development of increasingly sophisticated search algorithms or tool configurations for search agents without corresponding investment in reward design. The paper shows that even with a fixed tool set (search, open, find) and a fixed search paradigm (ReAct), the reward function is the dominant factor determining whether agents learn robust or brittle policies. This suggests that research investment in better verifiers and reward mechanisms may have higher marginal returns than investment in better search architectures or more complex tool environments—at least until verifier quality saturates.

Follow-Up Research This Work Enables

Self-verifying agents that compute their own rubric rewards during training to eliminate judge LLM cost. The paper's largest unaddressed practical barrier is the cost of running a large judge LLM (DeepSeek-v3.2) for rubric computation at every RL step. Section 6 discusses this gap explicitly: the paper provides no measurement of judge LLM cost, and the 53,000+ judge calls required for a full training run could easily dominate the total training budget. A critical follow-up would test whether the policy model itself can serve as its own judge for rubric computation—computing R^H_r from its own outputs rather than from an external LLM. This is a self-verification approach analogous to what Shao et al. (2025b) explored for mathematical reasoning. The specific experiment would train a Qwen3-30B agent with C-GRPO where the judge LLM is also Qwen3-30B (or a fine-tuned variant), measure the correlation between self-judged and external-judge rubric rewards, and compare final agent performance against the DeepSeek-v3.2 judge baseline. The paper's human-validated judge accuracy numbers (97.7% and 95.1%) provide a concrete target: self-verification would need to achieve comparable reliability to be viable. A negative result (self-verification introduces noise that degrades training) would be equally informative, establishing that external verification is genuinely necessary rather than merely convenient.

Adaptive difficulty-dependent reward weighting that allocates more rubric reward weight to harder training questions. The paper's \alpha parameter (rubric reward weight) is fixed at 0.3 across all training questions. But the benefits of process supervision likely vary with question complexity: easy questions (few hops, obvious entity connections) may need little rubric guidance because outcome rewards are already sufficient to produce thorough reasoning, while hard questions (many hops, subtle constraints) may benefit disproportionately from the dense feedback of rubric rewards. A natural extension would make \alpha question-dependent, allocated based on an estimate of question difficulty—for instance, scaling \alpha with the number of rubrics |R_q|, or with the SFT model's pass@1 rate on that question (analogous to the difficulty estimation in the inference-time compute scaling literature). A concrete experiment: bin DeepDive questions into difficulty quintiles based on GRPO's training accuracy, sweep \alpha separately per bin, and measure whether difficulty-conditioned \alpha yields better overall performance than the uniform \alpha = 0.3 used in the paper. This would also test whether the optimal \alpha = 0.3 is an artifact of DeepDive's specific difficulty distribution or a more general property.

Combining CaRR with iterative on-policy data generation for self-improvement loops. The paper trains C-GRPO on a fixed synthetic dataset (DeepDive), but the rubric framework is naturally suited to an iterative self-improvement pipeline: (1) train an agent with C-GRPO, (2) use the trained agent to generate new trajectories on new questions, (3) compute rubric rewards on those trajectories to filter for high-quality examples, (4) fine-tune on the filtered trajectories. This is directly analogous to the STaR/ReSTEM^{EM} paradigm (Zelikman et al., 2022; Singh et al., 2024) but with the rubric reward providing a principled quality filter that doesn't require ground-truth answers. Importantly, the CaRR rubric reward can be computed for any question that can be decomposed into rubrics—not just those with known answers—because the verification is against cited web content, not against golden labels. This means the self-improvement loop could operate on unlabeled web-derived questions, dramatically scaling the amount of training data. A concrete experiment: start with C-GRPO trained on DeepDive, use it to generate trajectories on BrowseComp training questions (without using ground-truth labels), filter trajectories with rubric reward above some threshold (say, 0.7), fine-tune on the filtered set, and evaluate on BrowseComp test. Compare against a baseline that filters using only outcome rewards (which requires labels, limiting scalability). The key metric is whether rubric-filtered self-training on unlabeled data can approach the performance of training on labeled data.

Stress-testing C-GRPO on naturally occurring multi-hop questions to determine the limits of synthetic-only training. The paper's central limitation (Section 6) is that CaRR assumes synthetic, entity-centric questions with clean compositional structure. An essential follow-up would test how C-GRPO-trained agents perform on natural multi-hop questions—questions written by humans that may have ambiguous entity references, implicit constraints, or answer formats that don't match the entity-string convention. The MUSIQUE dataset (Trivedi et al., 2022) or the MultiHop-RAG benchmark (Tang and Yang, 2024) would be appropriate test beds. The experiment would: (1) generate rubrics for natural questions using the same M_rubric prompt (Appendix E, Figure 12), (2) measure rubric quality through human evaluation (are the generated rubrics accurate and complete?), (3) train C-GRPO on a mix of DeepDive and natural questions, and (4) evaluate on held-out natural questions. If C-GRPO's benefits transfer seamlessly, the synthetic data limitation is less binding than Section 6 suggests. If C-GRPO underperforms GRPO on natural questions (because the rubrics are noisy or the connectivity check fails on ambiguous entity references), that would establish a clear boundary condition: CaRR is effective specifically for synthetic training data, and extending to natural data requires fundamentally new rubric generation or verification approaches. Either result would refine the field's understanding of when process supervision via rubrics is applicable.

Ablating the role of the "Thinking" variants in C-GRPO's effectiveness. All experiments use Qwen3-"Thinking" models, which include built-in chain-of-thought capabilities. These models may be particularly well-suited to CaRR because the explicit reasoning structure (thoughts before actions) naturally surfaces entity identities and factual claims, making entity identification (Step 1) more reliable. A clean ablation would train C-GRPO on the non-Thinking Qwen3 variants (same parameter counts, same pretraining, no built-in CoT) and compare performance. If C-GRPO's advantage over GRPO is substantially smaller or absent on non-Thinking models, the paper's claims would need to be scoped to models with explicit reasoning capabilities. This is practically important because many production deployment scenarios use non-reasoning models for latency reasons, and the community needs to know whether CaRR is applicable in those settings.

C-GRPO with open-weight judges to enable fully open-source replication. The paper uses DeepSeek-v3.2 (a proprietary model accessible via API) as the judge LLM. For the open-source community to replicate and extend this work without API dependencies, a follow-up should evaluate C-GRPO using open-weight judges—for example, using Qwen3-30B itself as the judge, or using Llama-3-70B, or using a fine-tuned smaller model specifically trained for rubric verification on synthetic QA data. The experiment would compare: (1) judge agreement rates against the DeepSeek-v3.2 "reference" judgments, (2) final agent performance when trained with open-weight judges vs. proprietary judges, and (3) total training cost (including judge inference) in FLOPs or GPU-hours. If a smaller open-weight judge can achieve comparable training outcomes (even with lower individual judgment accuracy, if the noise averages out over training), the practical barrier to CaRR adoption drops substantially.

Practical Applications and Downstream Use Cases

Cost-efficient training of custom search agents for domain-specific research. Organizations that need search agents for specialized domains—medical literature review, legal precedent search, financial due diligence—face a build-vs-buy decision. Proprietary agents (OpenAI DeepResearch, Kimi Researcher) are effective but expensive per query and cannot be customized to domain-specific knowledge or security requirements. Training a custom agent on domain-specific synthetic data using C-GRPO offers a concrete alternative. The paper's results on the 30B model show that C-GRPO-trained agents achieve 41.99 on DeepResearch Bench, within striking distance of Kimi-Researcher (44.64) and OpenAI-DeepResearch (46.45)—all proprietary agents trained with proprietary data. A legal tech company could: (1) generate synthetic multi-hop questions from a legal knowledge graph (e.g., case law → precedents → statutes → judicial opinions), (2) decompose them into rubrics using the same M_rubric prompt, (3) train a 30B open-weight model with C-GRPO on their hardware, and (4) deploy a private, domain-specialized research agent with no per-query API costs. The 4× improvement in evidence grounding (Table 2: C-GRPO cites 4.3 webpages vs. 3.5 for GRPO) and the 50% improvement in connected rubrics (5.2 vs. 4.0) mean the agent's outputs would be substantially more auditable—a critical requirement in legal and regulatory contexts where every claim must be traceable to source documents.

Training data generation for self-improving search systems in low-resource languages. BrowseComp-ZH (Chinese) results in Table 1 show C-GRPO's largest relative gains: 4B accuracy improves from 16.6 (GRPO) to 24.7 at 128k—a +49% relative improvement. This suggests C-GRPO is particularly effective when the base model's search capabilities are weaker, which is often the case for non-English languages where web corpora are smaller and training data is scarcer. A multilingual search platform could: (1) use a small multilingual model (e.g., Qwen3-4B) with C-GRPO to generate high-quality search trajectories in a target language, (2) use the rubric reward as an automatic quality filter to select trajectories with comprehensive, citation-grounded reasoning (rubric reward > 0.5, say), (3) fine-tune a larger model on these filtered trajectories, and (4) iterate. Because the rubric reward requires no ground-truth answers (only cited web content, which is available in any language with an internet presence), this pipeline works for languages where labeled QA data is unavailable. The 4B model's 24.7 on BrowseComp-ZH—exceeding WebSailor-32B (25.5) which was trained specifically for web search—provides a concrete performance target for what a small-model bootstrapping pipeline could achieve.

Quality assurance and auditing for deployed search agents through rubric-based evaluation. The CaRR framework, even without the RL component, serves as an evaluation tool that measures reasoning quality along dimensions invisible to answer accuracy. A company deploying search agents in production could run CaRR evaluation (rubric decomposition + three-step verification) on a sample of production trajectories to monitor for behavioral regressions that wouldn't appear in accuracy metrics. If the average rubric reward drops from 0.42 to 0.35 after a model update—even though answer accuracy remains stable—that signals the new policy is achieving the same accuracy through less thorough search (analogous to the GRPO vs. SFT difference in Table 2). This kind of monitoring is currently absent from deployed search agent systems, which typically track only task success rates. The judge LLM reliability numbers (97.7% entity identification, 95.1% citation judgment; Appendix C) provide confidence that automated rubric evaluation is trustworthy enough for production monitoring, at least for the synthetic question types validated in the study. The cost of periodic CaRR evaluation on a sample of trajectories (say, 1% of production traffic) would be negligible compared to the cost of serving those trajectories, making this a low-cost addition to existing MLOps pipelines for search agent deployments.