Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

ArXiv: 2503.09516

🎯 Pitch

LLMs can be taught via reinforcement learning to interleave reasoning with autonomous web search, boosting QA accuracy by over 20% without any supervised search trajectories. The key insight is that simply masking retrieved tokens during policy optimization stabilizes training, allowing models to learn when and what to query on their own.

1. Executive Summary

This paper introduces SEARCH-R1, a reinforcement learning framework that trains LLMs to interleave their own step-by-step reasoning with multi-turn search engine calls, learning to autonomously generate search queries and incorporate retrieved results during the reasoning process. Experiments on seven question-answering datasets—including NQ, TriviaQA, HotpotQA, and four multi-hop benchmarks—using Qwen2.5-3B/7B models with E5 retrieval over Wikipedia demonstrate that SEARCH-R1 improves performance by 24% (Qwen2.5-7B) and 20% (Qwen2.5-3B) over RAG baselines under identical experimental conditions. The framework uses retrieved token masking to stabilize RL training (masking out loss contributions from externally retrieved tokens during PPO/GRPO optimization) and a simple outcome-based reward function (exact-match string comparison against ground-truth answers), establishing that RL-trained interleaved search-and-reasoning generalizes across both in-distribution and out-of-distribution evaluation datasets—but only when the model learns to dynamically adjust its retrieval strategy through the RL process, with GRPO converging faster while PPO provides greater training stability.

2. Context and Motivation

The Core Problem: LLMs Cannot Effectively Interact with Search Engines During Reasoning

The fundamental question this paper tackles is deceptively simple: how do we train an LLM to autonomously decide when, what, and how to search during complex reasoning tasks? This matters because LLMs face two well-documented limitations that directly conflict with each other. First, even the largest models lack access to up-to-date information and domain-specific knowledge not captured during pretraining — they are, in effect, reasoning from a frozen snapshot of the world. Second, many real-world reasoning tasks (multi-hop QA, fact verification, complex inference) require iterative information gathering — you discover what you need to know through a process of reasoning, searching, re-reasoning, and searching again. The LLM must dynamically adjust its retrieval strategy based on what it learns at intermediate steps.

The gap this paper identifies is that existing approaches fail to teach LLMs this dynamic interaction skill. The two dominant paradigms — retrieval-augmented generation (RAG) and tool-use methods — each attack part of the problem but leave the core capability unaddressed.

Why This Problem Matters

The practical significance is immediate and growing. LLMs are increasingly deployed in settings where answer correctness depends on accessing information that is either temporally unavailable (events after the training cutoff), domain-specific (medical literature, legal statutes, financial data), or compositionally complex (requiring multiple pieces of evidence from different sources to be combined). Simply prompting an LLM to "reason step by step" when it lacks the necessary facts produces hallucinations or incomplete answers. Simply retrieving documents and stuffing them into context produces irrelevant noise that degrades reasoning quality.

More fundamentally, this problem sits at the intersection of two active research directions that have historically been pursued separately. The reasoning community has shown that RL can teach LLMs sophisticated reasoning behaviors — self-verification, error correction, backtracking — using only outcome-based rewards (Guo et al., 2025). The retrieval community has shown that search engines can supply missing knowledge. But no prior work has successfully applied RL to teach an LLM to interleave reasoning and search as a learned, optimizable behavior. SEARCH-R1 aims to bridge this gap, treating the search engine as part of the RL environment and training the LLM to use it strategically.

Prior Approaches and Their Shortcomings

The paper identifies two broad categories of prior work, each with specific limitations that motivate the SEARCH-R1 approach.

Retrieval-Augmented Generation (RAG) is essentially a one-shot retrieval paradigm. Standard RAG (Lewis et al., 2020) works by: (1) encoding the user's question as a query, (2) retrieving relevant passages, (3) concatenating those passages with the question, and (4) feeding everything into the LLM for generation. This pipeline has two critical limitations. First, the retrieval decision is made once, before any reasoning occurs. The LLM has no opportunity to realize "I need different information" mid-reasoning, because retrieval happens upstream of generation. Second, the query formulation is not learned — it is typically a fixed transformation of the input (often just the question itself). For complex multi-hop questions, a single retrieval round based on the surface question text is often insufficient.

The paper acknowledges that existing work has extended RAG to multi-turn settings. IRCoT (Trivedi et al., 2022a) prompts LLMs to interleave reasoning steps with retrieval queries, and ReAct (Yao et al., 2023) prompts LLMs to alternate between reasoning traces and tool-use actions including search. However, the paper makes a crucial distinction: prompting-based approaches are inherently suboptimal. The LLM is not optimized during training to learn how to interact with the search engine — it is merely following a template provided at inference time. The paper states this explicitly in the introduction:

"Prompting advanced LLMs with reasoning capabilities to use search engines during inference is often suboptimal, as the LLM might not fully possess the capability on how to interact optimally with the search engine."

This is not merely about following the format (knowing to use <search> tags). It is about learning strategic behaviors: when to search versus reason from existing knowledge, how to formulate queries that retrieve useful rather than redundant information, when to stop searching because the available evidence is sufficient, and how to recover when search results are irrelevant or misleading. Prompting cannot teach these behaviors because it operates entirely at inference time; the underlying model weights encode no search-specific optimization.

Tool-use training methods face scalability and data bottlenecks. The alternative is to treat the search engine as a tool and train the LLM to use it. Toolformer (Schick et al., 2023) pioneered this approach by fine-tuning LLMs on annotated trajectories where tool calls were explicitly labeled. The paper identifies two specific barriers that prevent this approach from scaling:

1. Reliance on high-quality labeled trajectories. Supervised fine-tuning requires curated demonstrations where every intermediate reasoning step and search query is correctly annotated. For complex multi-turn search-and-reasoning tasks, creating these trajectories at scale is prohibitively expensive — each example requires a human or oracle to decompose a multi-hop question into the right sequence of sub-questions, formulate effective search queries, and demonstrate correct use of retrieved results. The paper notes that such data is "difficult to obtain at scale."

2. Non-differentiability of the search operation. The retrieval step involves discrete operations (selecting query terms, calling an external API, receiving discrete text results) that break gradient flow. This means standard end-to-end supervised learning via gradient descent cannot propagate loss signals through the search engine back to the LLM's query-generation parameters. The paper explicitly calls out this limitation:

"the inherent non-differentiability of the search operation ... renders end-to-end gradient descent-based optimization inapplicable"

This is the key technical challenge that motivates using reinforcement learning. In RL, the search engine is part of the environment, not part of the differentiable computation graph. The LLM receives reward signals based on final answer correctness, and policy gradient methods can optimize the LLM's query-generation behavior even though the intermediate retrieval step is non-differentiable. The reward signal flows backward through the trajectory not via gradients but via advantage-weighted updates.

Prior RL-for-reasoning work ignores search. DeepSeek-R1 (Guo et al., 2025) demonstrated that pure RL with outcome-based rewards can teach LLMs advanced reasoning behaviors — self-verification, self-correction, reflection — without any supervised reasoning data. This is the paper's most direct intellectual predecessor. However, DeepSeek-R1 operates entirely within the parametric knowledge of the model: the LLM reasons using only what it learned during pretraining. As the paper notes:

"DeepSeek-R1 Zero ... primarily focuses on parametric reasoning"

The critical gap is that DeepSeek-R1 never interacts with an external knowledge source. In domains where answers genuinely depend on retrieving facts the model does not know (or facts that have changed since training), pure parametric reasoning hits a ceiling regardless of how sophisticated the reasoning process becomes. SEARCH-R1 can be understood as extending the DeepSeek-R1 Zero paradigm — RL with outcome-only rewards — to the search-augmented reasoning setting, where the rollout trajectory now interleaves LLM-generated tokens with retrieved content.

The paper also notes LeRet (Hsu et al., 2024) as related work that applies RL to retrieval, but clarifies that LeRet focuses on query diversification for retrieval quality rather than on training the LLM to interleave reasoning and search in a multi-turn problem-solving loop.

How This Paper Positions Itself

SEARCH-R1 frames itself as addressing three concrete challenges that emerge when combining RL with search engines, challenges that no prior system has simultaneously resolved:

Challenge 1: RL Framework and Stability. Integrating a search engine into the RL training loop means that the trajectory contains both LLM-generated tokens and tokens retrieved from external sources. Applying the same policy gradient loss to both types of tokens creates what the paper calls "unintended learning dynamics" — the model might learn to exploit patterns in retrieved text rather than improving its own query and reasoning behavior. The solution (detailed later in the technical sections) is retrieved token masking, which restricts the RL loss to only LLM-generated tokens. This is a novel stabilization technique specific to the search-augmented RL setting.

Challenge 2: Multi-Turn Interleaved Reasoning and Search. The training framework must support trajectories where the LLM can call search() multiple times, at variable points during reasoning, based on its own strategic decisions. This is not a fixed pattern — the model might search once, five times, or not at all, depending on the question's difficulty and what information it already possesses. The paper designs a rollout protocol where special tokens (<search>, </search>, <information>, </information>) trigger environment interactions, allowing the RL process to optimize when and what to search dynamically.

Challenge 3: Reward Design. RL-for-reasoning papers like DeepSeek-R1 demonstrated that simple outcome-based rewards (correct/incorrect answer matching) suffice for pure reasoning tasks. But in a search-augmented setting, where the model must learn query formulation, evidence integration, and stopping criteria, it was far from obvious that outcome rewards alone would provide sufficient signal. The paper adopts a deliberately minimal reward function — exact match between predicted and ground-truth answer — and tests empirically whether this sparse signal can drive the emergence of sophisticated search behaviors. The results (previewed in the executive summary) confirm that it can.

By tackling all three challenges within a unified framework compatible with both PPO and GRPO, SEARCH-R1 positions itself not as a new RL algorithm but as a recipe for extending existing RL-for-reasoning methods to the retrieval setting. The paper's claim is not "PPO is better than GRPO for search" but rather "RL with outcome rewards can teach search-and-reasoning, provided you handle retrieved tokens correctly during optimization."

3. Technical Approach

3.1 Reader Orientation

SEARCH-R1 is a training framework — not a new model architecture — that teaches an LLM to autonomously interleave its own chain-of-thought reasoning with real-time search engine queries, all learned through reinforcement learning using nothing more than final-answer correctness as a training signal. The system solves the problem of "how does an LLM learn when to search, what to query, and how to integrate retrieved information during step-by-step reasoning?" by treating the search engine as a non-differentiable environment in an RL loop, where the LLM generates reasoning tokens and search calls interleaved with retrieved passages within a single trajectory, and the policy gradient is computed only over the LLM's own tokens (excluding retrieved content) to ensure stable optimization.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five interacting components arranged in a training loop:

1. Policy LLM ($\pi_\theta$) — the language model being trained. It generates text autoregressively, producing reasoning steps (inside thinking/response tags), search queries (inside <search>/</search> tags), and final answers (inside <answer>/</answer> tags). This model is what gets deployed after training.

2. Search Engine ($\mathcal{R}) — an external, frozen retrieval system (E5 retriever over Wikipedia). It accepts a text query extracted from between <search> and </search> tokens and returns top-k passages, which get wrapped in <information>/</information> tags and inserted back into the trajectory. It is treated as part of the RL environment, not part of the differentiable computation graph.

3. Reference LLM ($\pi_{\text{ref}}) — a frozen copy of the initial policy model, used to compute KL-divergence regularization that prevents the trained policy from drifting too far from its starting distribution.

4. Reward Function ($r_\phi$) — a rule-based evaluator that compares the extracted final answer against the ground-truth using exact string match, returning 1 for correct and 0 for incorrect. There is no learned reward model.

5. Rollout Module — the orchestration logic (Algorithm 1) that manages the interaction loop: it calls the policy LLM to generate tokens, detects when <search> tags appear, pauses generation, queries the search engine, inserts results, and resumes generation. This repeats until the model produces <answer> tags or exhausts an action budget $B$.

Information flow during a single training step: A question $x$ is sampled from the training set → the rollout module initializes an empty trajectory → the policy LLM generates tokens one by one → when it emits </search>, the system extracts the query, calls the search engine, wraps results in <information> tags, and appends them to the trajectory → generation resumes with the retrieved context now visible to the model → this cycle repeats up to $B$ times → when the model emits </answer>, the rollout terminates → the reward function scores the final answer → the RL algorithm (PPO or GRPO) computes a policy gradient update over only the LLM-generated tokens in the trajectory, with retrieved tokens masked out → the KL penalty is applied relative to the reference model.

3.3 Roadmap for the Deep Dive

• First, the RL objective function (Equation 1), which formally defines what is being optimized and how the search engine enters the formulation. Understanding this equation is essential because it establishes the mathematical interface between the LLM and the retrieval environment.
• Second, the rollout protocol and token structure (Algorithm 1, Table 1), which defines the concrete mechanism by which the LLM interleaves reasoning and search. This is the "action space" the RL process operates over.
• Third, the retrieved token masking mechanism, which is the key stabilization technique that distinguishes SEARCH-R1 from naive RL+retrieval. This explains why the loss is only computed over LLM-generated tokens and what goes wrong without masking.
• Fourth, the two RL algorithm variants — PPO and GRPO — adapted for search-augmented trajectories (Equations 2 and 3). These define the exact policy gradient computations and highlight the differences between the two methods.
• Fifth, the training template (Table 1) and reward function (Equation 4), covering how the model is prompted to follow the structured format and how correctness is evaluated.
• Sixth, the hyperparameter and infrastructure configuration, which provides the concrete settings needed to reproduce the training pipeline.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an RL systems paper whose core idea is that policy-gradient methods can optimize an LLM's ability to interleave reasoning with search engine calls, provided that (a) the search engine is treated as part of the environment rather than the computation graph, (b) retrieved tokens are excluded from the policy gradient loss to prevent optimization instability, and (c) a simple outcome-based reward function provides sufficient learning signal when combined with appropriate KL regularization.

The RL Objective Function with a Search Engine

The paper begins by stating the standard RL objective for LLM fine-tuning and then extends it to incorporate the search engine. Understanding this extension is critical because it defines the mathematical framework within which all subsequent design choices (retrieved token masking, rollout protocol, reward design) operate.

Standard RL for LLMs (without search):

The classical formulation, used in RLHF and related work, optimizes:

$\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot | x)} \left[ r_\phi(x, y) \right] - \beta D_{\text{KL}} \left[ \pi_\theta(y | x) \parallel \pi_{\text{ref}}(y | x) \right]$

where $\pi_\theta$ is the policy LLM being trained, $\pi_{\text{ref}}$ is a frozen reference model (typically the initial checkpoint before RL training), $x$ is a prompt sampled from dataset $\mathcal{D}$, $y$ is a complete generated response, $r_\phi(x, y)$ is a scalar reward, and $\beta$ controls the strength of KL-divergence regularization. The expectation $\mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot | x)}$ means we sample prompts from the training data and generate responses from the current policy, then average over both sources of randomness. The KL term $\beta D_{\text{KL}}[\pi_\theta \parallel \pi_{\text{ref}}]$ penalizes the policy for diverging from the reference model, which prevents reward hacking — the model learning to produce outputs that score highly under the reward function but have drifted into degenerate regions of token space.

What this computes: The objective maximizes expected reward while keeping the policy statistically close to its starting point. The training loop samples a batch of prompts, generates responses from the current policy, computes rewards, and updates the policy parameters $\theta$ to increase the probability of high-reward responses. The KL penalty acts as a regularizer that bounds how much the policy can change per update.

Why this form: The KL term is essential because RL fine-tuning with only a reward signal can cause the model to collapse to a narrow distribution of outputs that exploit the reward function (e.g., producing repetitive text that happens to score well). By anchoring the policy to the reference model, the optimization finds responses that are both high-reward and plausible under the original language modeling distribution.

SEARCH-R1's extension (Equation 1):

The paper modifies this objective by conditioning the policy on the search engine:

$\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot | x; \mathcal{R})} \left[ r_\phi(x, y) \right] - \beta D_{\text{KL}} \left[ \pi_\theta(y | x; \mathcal{R}) \parallel \pi_{\text{ref}}(y | x; \mathcal{R}) \right]$

where $\mathcal{R}$ is the search engine, and the trajectory $y$ now interleaves LLM-generated tokens with tokens retrieved from $\mathcal{R}$. The notation $\pi_\theta(\cdot | x; \mathcal{R})$ is defined in the paper as equivalent to $\pi_\theta(\cdot | x) \Join \mathcal{R}$, where $\Join$ denotes the interleaved retrieval-and-reasoning process — the policy generates tokens until it calls search, the search engine returns results, and this cycle repeats.

What this computes: The same tradeoff as the standard objective — maximize expected reward while penalizing divergence from the reference — but now the generation process includes interaction with an external search engine. The distribution $\pi_\theta(y | x; \mathcal{R})$ is not a pure language model distribution; it is a mixture of LLM-generated tokens and retrieved-passage tokens, with the retrieval steps triggered by the LLM's own decisions to emit <search> tokens. The reward $r_\phi(x, y)$ evaluates the final answer extracted from the trajectory, and the KL divergence is computed over the full trajectory, including the conditioning on retrieved context.

Why this form: The key insight is that by conditioning on $\mathcal{R}$, the RL framework can optimize the LLM's interaction strategy with the search engine even though the search operation itself is non-differentiable. The search engine is part of the environment, not part of the computation graph. When the policy generates a query, the environment returns passages deterministically (given a fixed retriever and corpus). The policy gradient can then credit or penalize the tokens that led to that query based on whether the retrieved passages ultimately contributed to a correct answer. This is fundamentally different from supervised fine-tuning, which would require labeled query-answer pairs and cannot propagate loss through the discrete retrieval operation. The RL signal flows backward through the advantage function, not through gradients through the search engine.

Relationship to prior RL formulations: The paper explicitly contrasts this with the standard formulation (Equation 5 in Appendix A), noting that the original formulation "assumes that the entire output sequence y is generated solely by the policy LLM. This assumption does not hold in our setting, where model behavior incorporates both internal reasoning and external information retrieval." The SEARCH-R1 objective relaxes this assumption by making the trajectory distribution depend on both the policy and the environment.

The Rollout Protocol: Multi-Turn Search Engine Calling

The rollout protocol is the concrete mechanism by which the abstract objective in Equation 1 is realized. It is defined in Algorithm 1 of the paper and governs how the LLM generates a trajectory that interleaves reasoning, search queries, and retrieved content.

The core loop (Algorithm 1): The process begins with an input question $x$ and an empty trajectory $y$. An action counter $b$ tracks how many search calls have been made, constrained by a maximum budget $B$ (set to 4 in the default configuration). The outer loop runs while $b < B$, and each iteration proceeds as follows:

1. LLM generation phase: The policy model $\pi_\theta$ generates tokens autoregressively, conditioned on the prompt $x$ and all previously accumulated trajectory content $y$. Generation continues until one of three stop conditions is detected in the generated sequence: the token </search>, the token </answer>, or the end-of-sequence token. The newly generated segment is accumulated into the trajectory.

2. Search trigger detection: If the generated segment contains a <search> ... </search> pair, the system extracts the query text between these tokens, calls the search engine $\mathcal{R}$ with this query, receives the top-k retrieved documents $d$, and wraps them as <information> d </information> before appending them to the trajectory. The action counter increments.

3. Answer detection: If the generated segment contains <answer> ... </answer>, the rollout terminates immediately — the LLM has decided it has sufficient information to answer.

4. Malformed action handling: If the generated segment contains neither a search call nor an answer (e.g., the model produced reasoning but no clear action), the system appends a corrective prompt: "My action is not correct. Let me rethink." This ensures the model receives feedback about format violations and can recover.

What this protocol computes: It implements a partially observable Markov decision process where the state is the concatenation of the original question and all tokens generated or retrieved so far. The LLM's action at each turn is to generate a segment of text ending in either a search call or an answer. The environment responds to search calls by returning passages; it responds to answers by terminating and returning a reward. The key property is that the LLM controls when to search and what to query, and it can use the results of previous searches to inform subsequent queries.

Why this design: The interleaved structure is necessary for complex multi-hop reasoning. A single retrieval round before generation (standard RAG) cannot handle questions where the information needed for the second hop depends on the answer to the first hop. The multi-turn design allows the LLM to implement a strategy like: search for entity A → discover that A is related to entity B → search for entity B → combine facts about A and B to answer. The budget $B = 4$ provides enough turns for most multi-hop questions (the multi-hop datasets used, such as HotpotQA and 2WikiMultiHopQA, typically require 2-3 reasoning hops) while preventing infinite loops. The corrective prompt for malformed actions guides the model toward producing valid search or answer tokens, which is particularly important early in training when the base model may not yet reliably follow the format.

The token structure: The paper uses a specific set of special tokens that serve as the interface between the LLM and the environment. Understanding their roles is essential because they define the action space over which RL operates:

• <search> and </search> are action tokens — they trigger an environment interaction. The text between them is the search query. The LLM learns to formulate these queries through the RL process.
• <information> and </information> are observation tokens — they delimit content returned by the environment. The LLM never generates these; they are inserted by the system.
• thinking and response are reasoning tokens — they contain the LLM's internal chain of thought. The LLM generates text within these delimiters to plan its next action, analyze retrieved information, and decide whether to search more or answer.
• <answer> and </answer> are terminal tokens — they signal that the LLM is producing its final answer. When the system detects </answer>, the rollout ends.

This token structure draws a clear boundary between what the LLM controls (reasoning, search queries, final answer) and what the environment provides (retrieved passages). The RL process can therefore assign credit to the LLM's query-formulation decisions without being confounded by the content of retrieved passages.

The system instruction that structures this behavior (Table 1):

Answer the given question. You must conduct reasoning inside  thinking and  response
first every time you get new information. After reasoning, if you find you lack some
knowledge, you can call a search engine by <search> query </search>, and it will
return the top searched results between <information> and </information>. You
can search as many times as you want. If you find no further external knowledge
needed, you can directly provide the answer inside <answer> and </answer> without
detailed illustrations. For example, <answer> xxx </answer>. Question: question.

The paper deliberately designs this template to be structurally constraining but content-neutral. It tells the model what format to use (reason first, then decide whether to search or answer) but imposes no strategy — it does not recommend specific search query formulations, does not enforce a particular number of searches, and does not require reflection or verification. The phrase "if you find you lack some knowledge" makes search conditional on the model's own assessment of its knowledge state. The paper states this choice explicitly:

"We deliberately limit our constraints to this structural format, avoiding any content-specific biases, such as enforcing reflective reasoning and search engine calling or endorsing specific problem-solving approaches. This ensures that the model's natural learning dynamics during the RL process remain observable and unbiased."

Why this matters for RL: If the template imposed strategic behaviors (e.g., "always search at least twice" or "verify your answer before finalizing"), the RL process might learn to follow those patterns without actually understanding when they are necessary. By providing only format constraints, the paper ensures that any emergent search strategies — self-verification, iterative refinement, stopping early when confident — are genuine learned behaviors driven by the reward signal, not artifacts of the prompt.

Retrieved Token Masking: Stabilizing RL with External Content

This is the paper's key stabilization technique and the mechanism most likely to be overlooked in a superficial reading. It addresses a subtle but critical problem: when a trajectory contains both LLM-generated tokens and retrieved tokens, computing the policy gradient over all tokens can produce "unintended learning dynamics."

The problem: In standard RL for LLMs (both PPO and GRPO), the policy gradient is computed as a sum over all tokens in the trajectory. Each token $y_t$ contributes to the loss proportional to the advantage $A_t$ and the log-probability ratio $\pi_\theta(y_t)/\pi_{\text{old}}(y_t)$. If the trajectory includes tokens that the LLM did not generate — specifically, the text retrieved from the search engine — these tokens would also enter the loss computation. This creates two related pathologies:

1. Credit misattribution: The policy gradient would try to increase or decrease the probability of retrieved tokens based on whether the final answer was correct. But the LLM has no control over what the search engine returns — those tokens are determined by the retrieval model $\mathcal{R}$, not by $\pi_\theta$. Optimizing them is meaningless and introduces noise into the gradient.

2. Exploitation of retrieval patterns: The LLM might learn to generate sequences that, when certain retrieved tokens are present in the trajectory, increase the probability of subsequent tokens that happen to correlate with high reward — even if those patterns do not reflect genuine reasoning. The model could effectively "overfit" to superficial features of retrieved passages.

The paper describes the issue concisely:

"While optimizing LLM-generated tokens enhances the model's ability to interact with the search engine and perform reasoning, applying the same optimization to retrieved tokens can lead to unintended learning dynamics."

The solution: retrieved token masking. SEARCH-R1 introduces a binary mask $I(y_t)$ that is 1 for LLM-generated tokens and 0 for retrieved tokens. The policy gradient loss is then computed only over positions where $I(y_t) = 1$. Concretely, in the PPO objective (Equation 2), the loss becomes:

$J_{\text{PPO}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_{\text{old}}(\cdot | x; \mathcal{R})} \left[ \frac{1}{\sum_{t=1}^{|y|} I(y_t)} \sum_{t=1: I(y_t)=1}^{|y|} \min\left( \frac{\pi_\theta(y_t | x, y_{<t}; \mathcal{R})}{\pi_{\text{old}}(y_t | x, y_{<t}; \mathcal{R})} A_t, \text{clip}\left( \frac{\pi_\theta(y_t | x, y_{<t}; \mathcal{R})}{\pi_{\text{old}}(y_t | x, y_{<t}; \mathcal{R})}, 1 - \epsilon, 1 + \epsilon \right) A_t \right) \right]$

where $|y|$ is the total trajectory length (including retrieved tokens), $\sum_{t=1}^{|y|} I(y_t)$ is the number of LLM-generated tokens (used to normalize the loss), and the inner sum runs only over positions $t$ where $I(y_t) = 1$. The normalization by the count of generated tokens (rather than total tokens) ensures the loss scale is independent of how much text was retrieved.

What this computes: The objective computes a token-level policy gradient only over the tokens the LLM actually produced — its reasoning steps, its search queries, its answer formulation. Retrieved passages are treated as fixed context that conditions the generation but does not receive gradient updates. The advantage $A_t$ for a generated token still reflects whether the overall trajectory led to a correct answer, so the LLM is rewarded for generating queries that retrieve useful information, reasoning that correctly interprets retrieved passages, and answers that are consistent with the evidence — all through the indirect path of the advantage-weighted gradient on generated tokens only.

Why this form: The alternative — computing loss over all tokens — would effectively ask the gradient to "improve" the retrieved text, which is nonsensical and destabilizing. The normalization $\frac{1}{\sum I(y_t)}$ rather than $\frac{1}{|y|}$ ensures that when the model retrieves more passages (making the trajectory longer), the per-token loss scale remains constant. Without this normalization, trajectories with more retrieved content would have each LLM-generated token contributing less to the total loss, creating an unintended bias against search.

Empirical validation (Table 4, Figure 3): The paper reports results with and without retrieved token masking. For Qwen2.5-7B-base, SEARCH-R1 with masking achieves an average of 0.431 across seven datasets; SEARCH-R1 without masking achieves only 0.343, a degradation of approximately 20%. The training curves (Figure 3) show that masking leads to consistently higher and more stable training rewards. This is strong evidence that the masking is not merely a theoretical safeguard but a practical necessity.

Extension to GRPO: The same masking is applied in the GRPO objective (Equation 3) and also in the KL divergence calculation. When computing $D_{\text{KL}}[\pi_\theta \parallel \pi_{\text{ref}}]$, retrieved tokens are excluded so that the KL penalty only constrains the LLM's generation distribution, not its interaction with external content. The paper states:

"The retrieved token masking is also applied when calculating the KL divergence loss $D_{\text{KL}}$."

This ensures consistency: the KL penalty measures divergence on the same token positions where the policy gradient operates.

PPO with Search Engine: Detailed Objective and Advantage Estimation

The paper adapts Proximal Policy Optimization (PPO) to the search-augmented setting. PPO is an actor-critic method that uses a learned value function to estimate advantages and a clipping mechanism to prevent destructively large policy updates.

The PPO objective (Equation 2):

$J_{\text{PPO}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_{\text{old}}(\cdot | x; \mathcal{R})} \left[ \frac{1}{\sum_{t=1}^{|y|} I(y_t)} \sum_{t=1: I(y_t)=1}^{|y|} \min\left( \frac{\pi_\theta(y_t | x, y_{<t}; \mathcal{R})}{\pi_{\text{old}}(y_t | x, y_{<t}; \mathcal{R})} A_t, \text{clip}\left( \frac{\pi_\theta(y_t | x, y_{<t}; \mathcal{R})}{\pi_{\text{old}}(y_t | x, y_{<t}; \mathcal{R})}, 1 - \epsilon, 1 + \epsilon \right) A_t \right) \right]$

where $\pi_\theta$ is the current policy (being updated), $\pi_{\text{old}}$ is the policy from the previous iteration (used to generate the trajectories in the expectation), $A_t$ is the advantage estimate at position $t$, $\epsilon$ is the clipping hyperparameter (set to 0.2), and $I(y_t)$ is the retrieved token mask as defined above.

What this computes: For each LLM-generated token, the objective computes the ratio $r_t(\theta) = \pi_\theta(y_t) / \pi_{\text{old}}(y_t)$ — how much more (or less) likely the current policy makes token $y_t$ compared to the old policy. If the advantage $A_t$ is positive (the trajectory was better than expected), the objective wants to increase $r_t(\theta)$ — but only up to $1 + \epsilon$, the clipping threshold. If the advantage is negative, it wants to decrease $r_t(\theta)$ — but only down to $1 - \epsilon$. The $\min$ operation selects the more conservative of the unclipped and clipped updates, ensuring that the objective is a lower bound on the true expected improvement.

Why this form: The clipping mechanism prevents the policy from changing too much in a single update, which is essential for stable training. Without clipping, a single high-advantage trajectory could cause the policy to increase the probability of its tokens by orders of magnitude, potentially collapsing into a deterministic mode that stops exploring. The $\min$ with clipping makes the objective pessimistic — it ignores improvements beyond the clipping threshold when $A_t > 0$ (to prevent over-optimism) and ignores reductions beyond the threshold when $A_t < 0$ (to prevent over-pessimism). This is the standard PPO formulation (Schulman et al., 2017), now applied token-wise only to LLM-generated positions.

Advantage estimation with GAE: PPO requires an advantage estimate $A_t$ for each token. The paper uses Generalized Advantage Estimation (GAE) with parameters $\lambda = 1$ and $\gamma = 1$. GAE computes advantage as an exponentially-weighted sum of temporal difference errors:

$A_t^{\text{GAE}} = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l}$

where $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ is the TD error, $V(s_t)$ is the learned value function's estimate of expected future reward from state $s_t$, and $r_t$ is the reward at step $t$. With $\gamma = 1$ and $\lambda = 1$, GAE reduces to the Monte Carlo return minus the value baseline: $A_t = R - V(s_t)$, where $R$ is the final reward (1 for correct answer, 0 for incorrect). This is computationally simple — it uses the value function only as a baseline for variance reduction, with the full trajectory reward propagated uniformly.

The value function and its training: A separate value model $V_\phi$ (initialized from the same pretrained weights but with a regression head) is trained to predict expected future reward. The paper sets its learning rate to $1 \times 10^{-5}$, ten times higher than the policy learning rate of $1 \times 10^{-6}$, with a warm-up ratio of 0.015. Training the critic faster than the actor is standard practice in actor-critic methods — the value function must adapt quickly to the changing policy to provide useful baselines.

Hyperparameter specifics for PPO: Policy LLM learning rate = $1 \times 10^{-6}$, value LLM learning rate = $1 \times 10^{-5}$, training for 500 steps, policy warm-up ratio = 0.285, value warm-up ratio = 0.015, GAE $\lambda = 1$, $\gamma = 1$, clip ratio $\epsilon = 0.2$, KL coefficient $\beta = 0.001$.

GRPO with Search Engine: Group-Based Advantage and Simplified Architecture

Group Relative Policy Optimization (GRPO) is the second RL algorithm supported by SEARCH-R1. GRPO eliminates the need for a separate value function (critic) by using the average reward within a group of sampled responses as the baseline.

The GRPO objective (Equation 3):

$J_{\text{GRPO}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, \{y_i\}_{i=1}^G \sim \pi_{\text{old}}(\cdot | x; \mathcal{R})} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{\sum_{t=1}^{|y_i|} I(y_{i,t})} \sum_{t=1: I(y_{i,t})=1}^{|y_i|} \min\left( \frac{\pi_\theta(y_{i,t} | x, y_{i,<t}; \mathcal{R})}{\pi_{\text{old}}(y_{i,t} | x, y_{i,<t}; \mathcal{R})} \hat{A}_{i,t}, \text{clip}\left( \frac{\pi_\theta(y_{i,t} | x, y_{i,<t}; \mathcal{R})}{\pi_{\text{old}}(y_{i,t} | x, y_{i,<t}; \mathcal{R})}, 1 - \epsilon, 1 + \epsilon \right) \hat{A}_{i,t} \right) - \beta D_{\text{KL}}[\pi_\theta \parallel \pi_{\text{ref}}] \right]$

where $G$ is the group size (set to 5 in the default configuration), $\{y_i\}_{i=1}^G$ are $G$ independently sampled trajectories for the same prompt $x$, and $\hat{A}_{i,t}$ is the group-relative advantage.

What this computes: For each prompt $x$, GRPO samples $G$ complete trajectories from the old policy. Each trajectory $i$ receives a reward $r_i$ from the outcome-based reward function. The group-relative advantage for all tokens in trajectory $i$ is computed as:

$\hat{A}_{i,t} = \frac{r_i - \text{mean}(\{r_1, ..., r_G\})}{\text{std}(\{r_1, ..., r_G\})}$

This is the standardized reward — how many standard deviations above or below the group mean this trajectory's outcome lies. Crucially, all tokens within the same trajectory receive the same advantage value. This is fundamentally different from PPO, where GAE computes per-token advantages that can vary within a trajectory based on the value function's step-by-step predictions.

Why this form: The group-based advantage serves the same role as the value function baseline in PPO — it centers the rewards so that above-average trajectories receive positive advantages and below-average trajectories receive negative advantages. But it avoids the complexity of training and maintaining a separate value model. The standardization by standard deviation provides adaptive scaling: when the group has high variance in outcomes (some trajectories are clearly better than others), the advantages are compressed toward zero; when all trajectories are similarly good or bad, the scale adjusts accordingly. This is similar to the advantage normalization often used in PPO implementations but computed group-wise rather than batch-wise.

KL divergence regularization in GRPO: Unlike PPO, where KL divergence is typically incorporated into the reward function as a penalty, GRPO adds it directly to the loss as a separate term $-\beta D_{\text{KL}}[\pi_\theta \parallel \pi_{\text{ref}}]$. The KL divergence is computed over only LLM-generated tokens (retrieved token masking is applied here as well). The paper uses $\beta = 0.001$, matching the PPO setting.

Advantages and disadvantages relative to PPO: The paper's empirical analysis (Section 5.1, Figure 2(a)) reveals a clear tradeoff. GRPO converges faster — the training reward rises more quickly in the first 100-200 steps because there is no critic warm-up phase. PPO trains more stably — GRPO exhibits "reward collapse" after extended training, while PPO maintains stable rewards throughout. The paper explains:

"This is because PPO relies on a critic model, which requires several warm-up steps before effective training begins."

"GRPO leads to reward collapse after training for many steps, whereas PPO remains stable."

The final training rewards of both methods are comparable, suggesting that either algorithm is viable but PPO is "a preferable choice in this setting" due to stability.

GRPO hyperparameters: Group size $G = 5$, policy learning rate = $1 \times 10^{-6}$, 500 training steps, warm-up ratio = 0.285, $\epsilon = 0.2$, $\beta = 0.001$. The group size study (Appendix H, Table 8) shows that smaller groups (size 1, equivalent to REINFORCE) can achieve better generalization on out-of-distribution tasks despite lower training rewards, suggesting a speed-stability-generalization tradeoff.

The Reward Function: Outcome-Based Exact Match

The reward function is deliberately the simplest possible design. For each rollout trajectory, the system extracts the text between <answer> and </answer> tokens, compares it to the ground-truth answer using exact string matching, and returns 1 for a match and 0 otherwise:

$r_\phi(x, y) = \text{EM}(a_{\text{pred}}, a_{\text{gold}})$

where $a_{\text{pred}}$ is the extracted final answer from response $y$, and $a_{\text{gold}}$ is the ground truth answer from the dataset.

What this computes: A binary signal indicating whether the final answer exactly matches the reference answer. There is no partial credit, no intermediate rewards for good search queries, no format reward for proper template usage. The reward is only provided at the end of the complete trajectory.

Why this form: The paper argues against more complex reward designs on both theoretical and practical grounds. On the theoretical side, DeepSeek-R1 (Guo et al., 2025) demonstrated that outcome-based rewards alone can drive the emergence of sophisticated reasoning behaviors — the model discovers self-verification, error correction, and reflection as instrumental strategies for achieving correct final answers. The paper hypothesizes that search behaviors (query formulation, iterative refinement, stopping decisions) will similarly emerge as instrumental strategies when the only path to reward is producing correct answers.

On the practical side, the paper explicitly avoids neural reward models:

"We avoid training neural reward models, following Guo et al. (2025). This decision is motivated by the sensitivity of LLMs to specific forms of rewards in large-scale RL, as well as the additional computational cost and complexity introduced by retraining these models."

Neural reward models introduce their own failure modes — reward hacking, distribution shift between training and deployment, and the need to periodically retrain as the policy improves. A rule-based exact-match reward is invariant to these issues: it always measures the same thing, cannot be exploited (the model cannot "trick" string matching), and requires no additional training.

Format rewards are deliberately omitted. DeepSeek-R1 used format rewards to encourage the model to structure its outputs correctly. SEARCH-R1 does not, because:

"our learned model already demonstrates strong structural adherence"

The initial template in Table 1 is sufficient to teach the format without additional reward shaping. The paper leaves "the exploration of more complex format rewards for future work," suggesting this is a deliberate minimalism rather than an oversight.

Why this is surprising and important: In a search-augmented setting, the reward signal is extremely sparse — the model might generate dozens of reasoning steps and multiple search queries, but receives only a single scalar at the end. It is not obvious a priori that this sparse signal can teach the model to formulate effective search queries, because the credit assignment problem is severe: how does the model learn that query Q1 was good but Q2 was redundant when both occur in the same trajectory and only the final answer is scored? The paper's positive results (Table 2) demonstrate that policy gradient methods with group-based or value-based advantage estimation can solve this credit assignment problem through averaging across many trajectories. Trajectories where Q1 was informative and led to correct answers will, on average, receive higher rewards than trajectories where Q1 was poorly formulated, providing a statistical gradient toward better query formulation even without per-step rewards.

Training Infrastructure and Hyperparameter Configuration

The paper provides detailed infrastructure specifications that are essential for replication.

Hardware and parallelism: Training is performed on a single node with 8 NVIDIA H100 GPUs. The total batch size is 512, with a mini-batch size of 256 and a micro-batch size of 64. Gradient checkpointing is enabled to reduce memory usage. Fully Sharded Data Parallelism (FSDP) with CPU offloading is used to distribute model parameters across GPUs.

Sequence length constraints: Maximum total sequence length = 4,096 tokens, maximum response length = 500 tokens, maximum length of retrieved content = 500 tokens. These constraints prevent unbounded trajectory growth and ensure training fits within GPU memory.

LLM rollout configuration: The paper uses vLLM (an efficient LLM serving engine) for generating rollouts, with tensor parallel size of 1 and GPU memory utilization ratio of 0.6. This means 60% of each GPU's memory is allocated to the inference engine, with the remainder used for training computations. Rollout sampling uses temperature = 1.0 and top-p = 1.0 (effectively, standard multinomial sampling from the full distribution), encouraging exploration during training.

Model saving and checkpoint selection: Model checkpoints are saved every 100 steps. If training diverges (detected by monitoring the training reward curve), the paper evaluates at the most recent stable checkpoint; otherwise, the final checkpoint (step 500) is used for evaluation. This pragmatic approach handles the reward collapse observed with GRPO at later steps.

Training data: The training set is formed by merging the training splits of Natural Questions (NQ) and HotpotQA. This is a deliberate choice: NQ represents single-hop factual questions, while HotpotQA represents multi-hop reasoning questions. Training on a mixture ensures the model encounters both search scenarios (single lookup vs. multi-step retrieval) during RL optimization. Evaluation spans seven datasets covering both in-distribution (NQ, HotpotQA) and out-of-distribution (TriviaQA, PopQA, 2WikiMultiHopQA, Musique, Bamboogle) settings, providing a stringent test of generalization.

Retrieval configuration: The search engine uses E5 (a dense passage retriever) over the 2018 Wikipedia dump. The top-3 passages are retrieved per query (top-k = 3), following the setting from Lin et al. (2023). A study of top-k values (Appendix G, Table 7) shows that top-k = 3 achieves the best performance (average 0.431), outperforming top-k = 1 (0.375, lower recall) and top-k = 5 (0.400, lower precision introducing noise). The maximum number of search actions per rollout $B$ is set to 4.

Maximum action budget rationale: The budget $B = 4$ is a constraint on the number of search calls per trajectory. If the model has not produced an answer after 4 search rounds, the rollout terminates. This prevents infinite loops and bounds the maximum trajectory cost. For the multi-hop datasets in the evaluation (HotpotQA, 2WikiMultiHopQA, Musique), questions typically require 2-3 reasoning hops, so 4 search calls provide sufficient capacity while preventing runaway exploration.

Summary of Design Choices and Their Justifications

• RL over supervised fine-tuning: RL can optimize through the non-differentiable search operation via policy gradients, whereas SFT requires expensive labeled trajectories and cannot propagate loss through discrete retrieval steps.
• Outcome-based reward over process reward or neural reward models: Simpler, immune to reward hacking, requires no additional training, and has been shown sufficient for reasoning in prior work. The paper demonstrates this sufficiency extends to search-augmented settings.
• Retrieved token masking over unmasked loss: Prevents credit misattribution to tokens the LLM does not control and stabilizes training. Empirically validated with a 20% average performance degradation when masking is removed.
• Interleaved multi-turn search over single-round RAG: Enables the model to dynamically adjust its retrieval strategy based on intermediate findings, which is essential for multi-hop reasoning where later queries depend on earlier results.
• PPO as default over GRPO: PPO provides greater training stability and avoids reward collapse, despite slower initial convergence. Both methods achieve comparable final performance.
• Generic structural template over content-specific prompts: Ensures that learned search behaviors emerge from the reward signal rather than being prescribed by the prompt, making the results more generalizable and the behavioral analysis more interpretable.
• Mixed single-hop and multi-hop training data: Trains the model to handle both simple factual lookups and complex multi-step retrieval-and-reasoning chains within the same RL process.
• Top-3 retrieval density: Balances recall (getting relevant information) and precision (avoiding noise that degrades both inference and training). Higher top-k values introduce noise that can discourage the model from using retrieved content.

4. Key Insights and Innovations

Innovation 1: RL as a Solution to the Non-Differentiability Barrier in Search-Augmented Training

The most fundamental conceptual move in this paper is reframing the problem of teaching LLMs to search not as a data problem (we need more labeled search trajectories) but as an optimization problem (we need a training signal that can flow through a non-differentiable retrieval operation). This recharacterization matters because it explains why prior approaches hit a ceiling and points toward a different solution space.

Before SEARCH-R1, the dominant assumption in training LLMs to use search engines was that you needed supervised demonstrations. Toolformer (Schick et al., 2023) and related work approached search as a tool-use skill to be taught through annotated examples: show the model many instances of "here is a question, here is when you should search, here is what query to use, here is how to integrate the result." This framing treats the core difficulty as a data scarcity problem — if only we had enough high-quality trajectories, the model would learn.

The paper identifies a deeper barrier that supervised methods cannot cross, regardless of data volume: the search operation is non-differentiable. When the model generates a query, calls an external API, and receives discrete text results, there is no gradient path from the final answer quality back through the retrieval step to the query-generation parameters. The paper explicitly calls this out:

"the inherent non-differentiability of the search operation ... renders end-to-end gradient descent-based optimization inapplicable"

This is not merely an implementation inconvenience — it means that supervised fine-tuning on labeled trajectories is fundamentally limited. SFT can teach the model to mimic query formulations seen in training, but it cannot optimize query strategies end-to-end because the loss signal stops at the retrieval boundary. If the model generates a slightly suboptimal query — one that retrieves almost-relevant but not quite right information — supervised training cannot propagate a gradient saying "adjust your query formulation by this direction to retrieve better passages."

RL sidesteps this barrier by treating the search engine as part of the environment, not part of the computation graph. The policy gradient does not differentiate through the search engine; it estimates, through repeated sampling and reward averaging, which query-generating behaviors tend to produce trajectories that end in correct answers. The advantage function A_t credits or penalizes the tokens that composed a search query based on statistical association with downstream outcomes, not through chain-rule differentiation. This is a fundamental reconceptualization: the optimization signal flows backward through expectation over stochastic trajectories rather than through deterministic gradients through operations.

This matters beyond the technical mechanism because it opens up a whole class of problems that were previously considered untrainable end-to-end. Any task involving non-differentiable external tools — databases, calculators, code execution, APIs — inherits this same barrier. The SEARCH-R1 formulation (policy conditioned on environment R, outcome reward, retrieved token masking) provides a template that directly generalizes. The paper's contribution is not "we used PPO instead of SFT" but rather "we recognized that the non-differentiability of retrieval makes RL the natural optimization framework, and we showed how to make it work stably."

The evidence that this is a genuine insight rather than an obvious choice is in the failure modes the paper documents and solves. Naively applying RL to search-augmented trajectories — computing policy gradients over all tokens including retrieved ones — produces "unintended learning dynamics" that degrade performance by ~20% (Table 4). The fact that a specific stabilization mechanism (retrieved token masking) was necessary and non-obvious confirms that this is not a trivial application of existing RL methods. The field had not previously recognized that tokens from the environment require different gradient treatment from tokens generated by the policy.

Innovation 2: Outcome-Only Rewards as Sufficient Signal for Search Strategy Learning

The paper's second conceptual contribution is the empirical demonstration that a single binary reward at the end of a multi-turn search-and-reasoning trajectory provides sufficient learning signal to teach sophisticated search behaviors — query formulation, multi-hop decomposition, evidence integration, and self-verification. This finding is counterintuitive in the search setting even though it has precedent in pure reasoning (DeepSeek-R1), and the paper's contribution is establishing that the sufficiency of sparse rewards extends across the retrieval boundary.

The intuition that makes this surprising is the credit assignment problem. Consider a trajectory where the model makes three search calls, retrieves passages, reasons about them, and produces a correct answer. The outcome reward (+1) must somehow teach the model that search query #1 was well-formulated (it retrieved a key entity), query #2 was somewhat redundant (it retrieved information already obtained), and query #3 was unnecessary (the answer was already determined). With only a single scalar signal, there is no direct way to distinguish these contributions. The natural assumption — encoded in much prior work on tool use and retrieval — is that you need process supervision: intermediate rewards for good queries, format penalties for malformed calls, or human feedback on retrieval quality.

The paper explicitly rejects this assumption:

"We adopt a straightforward outcome-based reward function, avoiding the complexity of process-based rewards."

The significance of this choice is not simplicity for its own sake. It is a claim about what kind of learning signal is necessary versus what emerges from statistical averaging. Over many training trajectories, the policy gradient averages the reward across different query formulations. Trajectories where the first query was effective (retrieved relevant entities, enabling subsequent reasoning) will, on average, reach correct answers more often than trajectories where the first query was vague (retrieved noise, derailing the reasoning chain). The gradient therefore pushes the model toward better query formulation without ever explicitly scoring individual queries. The optimization discovers query strategies as instrumental behaviors — means to the end of correct answers — through the statistical association between query-generation tokens and final outcomes.

This connects to a broader insight about credit assignment in RL for language. DeepSeek-R1 (Guo et al., 2025) showed that outcome rewards suffice for teaching reasoning behaviors like self-verification and backtracking. SEARCH-R1 extends this claim: outcome rewards also suffice when the trajectory includes external information acquisition steps. The search engine introduces additional stochasticity (different queries retrieve different passages, which condition different reasoning paths), but the averaging property of policy gradients handles this additional variance. The paper's positive results across seven datasets (Table 2) — including out-of-distribution generalization to datasets not seen during training — provide the evidence that this claim holds.

This matters for the research agenda because it eliminates a major bottleneck. If process rewards were necessary, scaling search-augmented RL would require annotating intermediate steps (query quality, retrieval relevance, reasoning correctness) at enormous cost. The finding that they are not necessary means that search-augmented RL can scale using only answer-level supervision, which is often available from existing QA datasets.

The paper does not claim that outcome rewards are always optimal — it explicitly leaves "more complex format rewards for future work." But establishing sufficiency is a stronger claim than establishing optimality, because sufficiency defines the minimum viable supervision for the approach to work. This is intellectually analogous to the discovery in computer vision that ImageNet labels (image-level categories) suffice to train models that learn object-part detectors as intermediate representations — the supervision signal is at the output level, but useful intermediate behaviors emerge.

Innovation 3: Retrieved Token Masking as a Diagnostic Category for Environment-Augmented RL

The third innovation is less a technique than a conceptual category: the identification that tokens originating from the environment (retrieved passages) require fundamentally different gradient treatment from tokens generated by the policy, and that failing to make this distinction causes a specific, diagnosable failure mode. This might sound like an implementation detail, but it represents a genuine insight about the structure of RL when the trajectory is jointly produced by a learned policy and an external system.

Before this work, RL for LLMs (PPO, GRPO, DPO, and related methods) operated in a setting where the entire trajectory y was generated by the policy π_θ. The token-level policy gradient at position t asks: "should the policy be more or less likely to generate token y_t in context y_{<t}?" This question is well-formed when π_θ actually controls the generation of y_t. When y_t is a token retrieved from Wikipedia by a frozen retriever, this question becomes ill-posed. The policy cannot change the probability of that retrieved token — it can only change the probability of the query that preceded the retrieval. Computing a gradient on the retrieved token itself attempts to optimize something the policy does not control, introducing noise and enabling pathological learning dynamics.

The paper's contribution is naming this problem and demonstrating its empirical magnitude. Without masking, average performance drops from 0.431 to 0.343 on Qwen2.5-7B — a 20% relative degradation (Table 4). The training curves (Figure 3 in Appendix D) show that unmasked training is not merely noisier but systematically worse, with lower and more unstable rewards throughout training. This is not a small hyperparameter sensitivity; it is a categorical failure mode.

The conceptual significance extends beyond this paper. As LLMs are increasingly deployed in agentic settings where they interact with tools, databases, and other external systems, the RL trajectories will increasingly contain tokens the policy does not generate. Retrieved token masking establishes a principle: when computing policy gradients, partition the trajectory into policy-generated and environment-generated tokens, and apply the loss only to the former. This principle generalizes to any setting where the environment inserts tokens into the trajectory — tool outputs, API responses, database query results, code execution output. The paper provides both a diagnostic category (is your RL training unstable when trajectories include external content? you probably need masking) and a solution template.

The paper also extends this masking to the KL divergence term, which is a non-obvious detail. KL divergence measures how much the policy distribution diverges from the reference distribution. Computing it over retrieved tokens would penalize the policy for conditioning on retrieved information differently than the reference model — but the reference model never saw the same retrieved passages during its rollout, making the comparison meaningless. Masking retrieved tokens from the KL computation ensures the regularization only constrains the LLM's generation behavior, not its interaction with external content.

This insight is incremental in the sense that it follows from clearly stating what tokens the policy controls versus what the environment provides. But in practice, the RL-for-LLMs community has largely operated in pure-generation settings where this distinction does not arise. The paper identifies a blind spot in standard RL formulations when applied to tool-augmented settings and provides empirical evidence that the blind spot causes substantial degradation. Establishing this as a first-class concern for agentic RL training — rather than a footnote — is the contribution.

Innovation 4: Search-and-Reasoning as an Emergent Capability Distinct from Prompted Tool Use

The fourth insight is a behavioral finding rather than a methodological one: when trained with RL and outcome rewards, LLMs learn search strategies that qualitatively differ from what prompting elicits, including behaviors (self-verification through search, strategic query reformulation, stopping when sufficient) that the training template never explicitly instructs. This provides evidence that the model is learning a general skill rather than mimicking a prescribed pattern.

The distinction between prompted and learned search behavior is crucial for understanding what SEARCH-R1 actually achieves. Prompting-based methods like IRCoT and ReAct provide the LLM with a fixed template: reason, then search, then reason, then answer. The model follows this pattern regardless of whether the question actually requires search. It might search even when it already knows the answer (wasteful), or it might follow exactly N search steps because the template suggests N rather than because the evidence demands it. The behavior is procedural (executing a script) rather than strategic (making decisions based on information state).

SEARCH-R1 trains the model to optimize for answer correctness directly. The model can learn, through the reward signal, that some questions require zero searches (if the parametric knowledge suffices), some require one search, and some require multiple searches — and crucially, that the number depends on the question, not on a template. The RL process can also teach the model when to stop searching because additional information is unlikely to change the answer, and when to search again because the current evidence is insufficient or contradictory.

The case studies (Appendix I and J) provide qualitative evidence for emergent strategic behaviors. In case study 1 (comparing R1 without search to SEARCH-R1), the pure-reasoning model answers a question about Britney Spears' birthplace using only parametric knowledge and gets it wrong (it answers "Houston" instead of "McComb, Mississippi"). SEARCH-R1 decomposes the question into sub-problems ("what fragrance?", "who created it?", "where was she born?"), searches iteratively, and after finding the answer, performs an additional search to verify the location — a self-verification behavior that the template never instructs. Case study 10 shows the model recognizing that further searches are unlikely to help ("The other three superheroes are not mentioned in the search result. I'll provide the answer based on the information I have") and stopping early rather than exhausting its search budget fruitlessly.

These behaviors are significant because they mirror the emergent reasoning behaviors (self-verification, backtracking, reflection) observed in DeepSeek-R1, but now in the context of information acquisition. The model learns not just how to search but when and why — a meta-cognitive skill about its own knowledge state. This connects to a broader research question about whether RL can teach models to recognize the boundaries of their own knowledge and take instrumental actions (search) to transcend those boundaries.

The empirical evidence that this is learned rather than prompted comes from the training dynamics. Figure 2(d) shows that the number of valid search calls increases over training — starting from a low baseline and rising as the model discovers that searching more leads to higher rewards. If the model were merely following the template, search frequency would be roughly constant. The increase indicates that the model is learning that search is instrumentally valuable, not just procedurally required.

This innovation is fundamentally about capability emergence — the claim that RL with sparse rewards can produce behaviors that are more sophisticated than what was explicitly programmed into the template or reward function. It positions SEARCH-R1 not as an engineering improvement over prompted search (faster, more accurate) but as a qualitatively different kind of system: one that has learned a transferable skill of information-seeking through interaction, rather than one that is executing a retrieval script.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The evaluation spans seven question-answering benchmarks: Natural Questions (NQ; Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), PopQA (Mallen et al., 2022), HotpotQA (Yang et al., 2018), 2WikiMultiHopQA (Ho et al., 2020), Musique (Trivedi et al., 2022b), and Bamboogle (Press et al., 2022). These are categorized into general QA (NQ, TriviaQA, PopQA) and multi-hop QA (HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle). The test or validation splits are used for evaluation, while the training sets of NQ and HotpotQA are merged to form the training data for SEARCH-R1 and all fine-tuning baselines.

• Base model(s). Experiments use Qwen2.5 models (Yang et al., 2024) at two scales: 3B and 7B parameters, each available in base and instruction-tuned (Instruct) variants. The paper justifies Qwen2.5 as a representative modern LLM family and includes both base and instruct versions to study whether RL can bridge the gap between raw pretrained models and post-trained models in search-augmented reasoning. Additional experiments on Qwen2.5-14B are reported in Appendix C. For inference-style baselines (Direct, CoT, IRCoT, Search-o1, RAG), only instruct models are used since base models fail to follow instructions without training. For RL-tuning methods (R1, SEARCH-R1, rejection sampling), both base and instruct variants are evaluated.

• Metrics. The primary metric is Exact Match (EM) accuracy, following prior work (Yu et al., 2024). For each question, the system extracts the text between <answer> and </answer> tokens from the final generated response and compares it character-by-character against the ground-truth answer string. There is no partial credit or manual evaluation — the match must be exact. This is the same metric used as the reward function during training (Equation 4), creating a direct alignment between training signal and evaluation criterion.

• Baselines. The paper compares against seven baselines spanning inference-only, retrieval-augmented, and fine-tuning approaches:

  • Direct Inference: The LLM generates an answer given only the question, with no retrieval and no chain-of-thought instruction.
  • Chain-of-Thought (CoT) (Wei et al., 2022): The LLM is prompted to reason step-by-step before answering, but without any retrieval.
  • IRCoT (Trivedi et al., 2022a): A prompting method that interleaves chain-of-thought reasoning with retrieval queries, where the LLM generates reasoning traces and search calls following a fixed template.
  • Search-o1 (Li et al., 2025): An agentic search-enhanced reasoning approach that prompts the LLM to conduct iterative retrieval and reasoning.
  • RAG (Lewis et al., 2020): Standard retrieval-augmented generation using the question as query for a single retrieval round, with top-3 passages concatenated to the question before generation.
  • Supervised Fine-Tuning (SFT) (Chung et al., 2024): The LLM is fine-tuned on the same merged NQ+HotpotQA training data with multi-turn search-and-reasoning trajectories, using standard next-token prediction loss.
  • R1 (base and instruct) (Guo et al., 2025): RL-based fine-tuning without a search engine, using the same PPO/GRPO framework and training data as SEARCH-R1 but with trajectories containing only reasoning and answer steps (no search calls). This isolates the effect of adding search to RL training.
  • Rejection Sampling with Search Engine (Ahn et al., 2024): For each training prompt, five candidate responses are generated from the instructed LLM with search engine access; trajectories that lead to correct final answers are retained. These selected trajectories form a new training set used for standard SFT, preserving the multi-turn LLM–search engine interaction format.

  All retrieval-based baselines (RAG, IRCoT, Search-o1, SFT, rejection sampling, and SEARCH-R1 itself) use the same retriever (E5 over 2018 Wikipedia), the same number of retrieved passages (top-3), and the same knowledge corpus. This ensures that performance differences reflect the training methodology rather than retrieval quality disparities. For baselines requiring instruction following (Direct, CoT, IRCoT, Search-o1, RAG), only instruct models are evaluated since base models fail to adhere to the required output formats.

• Generation budget / compute accounting. The paper does not measure compute in FLOPs or tokens; instead, all methods are compared under fixed hyperparameter configurations (same number of retrieved passages, same LLM architecture and size, same training data). For SEARCH-R1, the rollout budget is controlled by the maximum action budget B = 4 (up to 4 search calls per question) and the number of retrieved passages per call (top-k = 3). For PPO and GRPO training, the total batch size (512), mini-batch size (256), and number of training steps (500) are held constant. The paper does not report wall-clock time or inference latency, though the architectural description (8×H100 GPUs, vLLM-based rollouts) provides enough information for approximate cost estimation. The key fairness claim is that all compared methods use identical retrieval infrastructure and base models, making accuracy differences attributable to the training methodology rather than compute disparities.

• Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals. Instead, it follows a standard train-test split: training is performed on the merged NQ+HotpotQA training sets, and evaluation is conducted on the test/validation splits of all seven datasets. In-domain evaluation refers to NQ and HotpotQA (datasets whose training splits were used); out-of-domain evaluation refers to TriviaQA, PopQA, 2WikiMultiHopQA, Musique, and Bamboogle (datasets not seen during training). For both PPO and GRPO training, model checkpoints are saved every 100 steps. If training diverges (detected by monitoring the training reward curve), the most recent stable checkpoint is used for evaluation; otherwise, the final checkpoint at step 500 is used. This pragmatic checkpoint selection is a form of early stopping based on training dynamics rather than held-out validation performance.

Main Quantitative Results

The central results are reported in Table 2, covering all seven datasets and eight methods across two model scales and two model variants (base/instruct). The paper also provides training dynamics analyses in Figure 2 and RL method comparisons in Table 3. I organize the results by the key comparisons the paper makes.

SEARCH-R1 vs. Retrieval-Augmented Baselines

The headline claim is that SEARCH-R1 improves performance over RAG baselines by 24% average relative improvement for Qwen2.5-7B and 20% for Qwen2.5-3B. These percentages are computed by comparing the average EM across all seven datasets.

For Qwen2.5-7B-base, SEARCH-R1 achieves an average EM of 0.431 across the seven datasets, compared to 0.304 for standard RAG — an absolute gain of 12.7 percentage points and a relative improvement of approximately 42%. The gap is not uniform across datasets. On NQ (in-distribution), SEARCH-R1-base reaches 0.480 vs. RAG at 0.349 — a 13.1-point absolute gain. On TriviaQA (out-of-distribution), SEARCH-R1-base achieves 0.638 vs. RAG at 0.585 — a 5.3-point gain, showing that the method generalizes but with diminishing returns on datasets where RAG already performs well. On the multi-hop datasets, the gaps are often larger in relative terms: HotpotQA (in-distribution) shows SEARCH-R1-base at 0.433 vs. RAG at 0.299 (45% relative gain), and 2WikiMultiHopQA shows 0.382 vs. 0.235 (63% relative gain). On Bamboogle, the gain is particularly striking: 0.432 vs. 0.208 — more than doubling RAG performance.

For Qwen2.5-7B-instruct, the pattern is similar but with compressed margins. SEARCH-R1-instruct averages 0.385 vs. RAG at 0.304 — a 27% relative improvement. The base model variant (0.431) notably outperforms the instruct variant (0.385), which is a surprising result discussed further in the critical assessment. This means the paper's 24% figure (from the abstract) represents some averaging across base and instruct variants — the actual gains depend on model variant and dataset. Computing the average relative gain across all dataset-variant pairs is not straightforward from the reported numbers alone, but the per-cell comparisons in Table 2 consistently show SEARCH-R1 exceeding RAG by substantial margins.

For Qwen2.5-3B, the improvements are more modest in absolute terms but still notable. SEARCH-R1-base averages 0.303 vs. RAG at 0.270 (12% relative gain). SEARCH-R1-instruct averages 0.325 vs. RAG at 0.270 (20% relative gain). Notably, for the 3B model, the instruct variant slightly outperforms the base variant, inverting the pattern seen with 7B. This suggests that the base-vs-instruct advantage is model-size-dependent, though the paper does not explore this further.

A critical observation in the paper's own analysis: "Larger models are better on learning how to do search. SEARCH-R1 on 7B model shows much larger 'performance gap' compared with 3B model (e.g., compared with second best model - RAG)." This is visible in Table 2: the 7B model's SEARCH-R1 performance advantage is consistently larger in absolute terms than the 3B model's, suggesting that search strategy learning is capacity-dependent.

SEARCH-R1 vs. Pure Reasoning RL (R1 without Search)

The comparison between SEARCH-R1 and R1 (RL without search engine access) isolates the contribution of search to the RL training process. For Qwen2.5-7B-base, SEARCH-R1 averages 0.431 vs. R1-base at 0.276 — an absolute gain of 15.5 points (56% relative improvement). For Qwen2.5-7B-instruct, the gap is 0.385 vs. 0.271 (11.4 points, 42% relative). For the 3B model, the gaps are smaller: SEARCH-R1-base at 0.303 vs. R1-base at 0.229 (7.4 points, 32% relative); SEARCH-R1-instruct at 0.325 vs. R1-instruct at 0.224 (10.1 points, 45% relative). The paper notes this aligns with expectations: "incorporating search into LLM reasoning provides access to relevant external knowledge, improving overall performance."

What is more interesting is the pattern across datasets. R1 without search performs respectably on datasets where parametric knowledge suffices — TriviaQA (0.539 for 7B-base) — but collapses on datasets requiring specific factual retrieval. On NQ, R1-base reaches only 0.297 vs. SEARCH-R1-base at 0.480. On Bamboogle, R1-base reaches 0.296 vs. SEARCH-R1-base at 0.432. This demonstrates that RL alone improves reasoning but cannot compensate for missing knowledge — the model needs retrieval to answer questions outside its parametric memory.

SEARCH-R1 vs. Rejection Sampling

The rejection sampling baseline is the strongest non-RL training baseline because it uses the same search engine interaction format and selects only successful trajectories for SFT. SEARCH-R1 consistently outperforms it. For Qwen2.5-7B-base, SEARCH-R1 averages 0.431 vs. rejection sampling at 0.348. The gap is particularly large on HotpotQA (0.433 vs. 0.331) and Musique (0.196 vs. 0.123). This supports the paper's central claim that RL optimization (which can learn from both successful and unsuccessful trajectories through advantage-weighted updates) is more effective than filtering for success and doing SFT on positive examples alone. Rejection sampling discards information from failed trajectories; RL uses both successes and failures to compute gradients.

SEARCH-R1 vs. Prompted Search Methods (IRCoT, Search-o1)

The prompted baselines (IRCoT and Search-o1) both use the instruct model to interleave reasoning and search at inference time without any training. SEARCH-R1 consistently outperforms both. For Qwen2.5-7B, IRCoT averages 0.239 and Search-o1 averages 0.206, compared to SEARCH-R1-base at 0.431. The large gap (roughly 80-110% relative improvement) confirms the paper's motivating claim: "Prompting advanced LLMs with reasoning capabilities to use search engines during inference is often suboptimal, as the LLM might not fully possess the capability on how to interact optimally with the search engine." The trained model substantially outperforms its prompted counterpart, even though both use the same underlying LLM architecture and retrieval infrastructure.

Generalization: In-Distribution vs. Out-of-Distribution

The evaluation design allows testing generalization: models are trained on NQ+HotpotQA but evaluated on five additional datasets. SEARCH-R1 generalizes strongly to some OOD datasets and weakly to others. For Qwen2.5-7B-base: TriviaQA (OOD) reaches 0.638, the highest absolute score across all datasets; PopQA reaches 0.457; 2WikiMultiHopQA reaches 0.382; Musique reaches only 0.196; Bamboogle reaches 0.432. The variability in OOD performance (from 0.196 to 0.638) suggests that generalization depends on the similarity between the training distribution and the target dataset. Musique, which involves complex multi-hop questions requiring composition of multiple facts, proves hardest for all methods — SEARCH-R1-base achieves only 0.196, though this still substantially exceeds RAG (0.058), R1 (0.083), and rejection sampling (0.123).

The paper does not provide a detailed analysis of why Musique and certain other datasets are harder, or whether the learned search strategies transfer poorly to particular question types. The generalization claim — "these gains hold across both in-distribution and out-of-distribution evaluation" — is technically true (SEARCH-R1 outperforms baselines on all datasets) but masks substantial performance heterogeneity that could be informative about failure modes.

Training Dynamics: PPO vs. GRPO

Table 3 and Figure 2(a) provide the direct comparison between RL algorithms. The key observations:

For Qwen2.5-7B-base, PPO-based SEARCH-R1 achieves 0.431 average, while GRPO-based achieves 0.350 average — a substantial gap in favor of PPO. For Qwen2.5-7B-instruct, the gap narrows: PPO at 0.385 vs. GRPO at 0.396 — essentially tied. For Qwen2.5-3B models, the pattern reverses: GRPO-base at 0.312 vs. PPO-base at 0.303; GRPO-instruct at 0.336 vs. PPO-instruct at 0.325.

Why this matters: The paper presents PPO as the default method due to stability, but the performance data tells a more nuanced story. For the 7B-base model, PPO substantially outperforms GRPO (0.431 vs. 0.350). For instruct models and smaller models, the methods are comparable or GRPO slightly edges ahead. The paper's own explanation — "PPO demonstrates greater training stability" (Section 5.1) and "GRPO leads to reward collapse after training for many steps" — is supported by the training curves in Figure 2(a) but does not fully explain the interaction with model scale and variant. The GRPO reward collapse appears to be more damaging for the 7B-base model than for other configurations, suggesting that the stability advantage of PPO is most consequential for larger base models.

GRPO convergence speed: The paper notes that "GRPO converges faster than PPO across all cases." This is visible in Figure 2(a), where the GRPO training reward rises steeply in the first ~50 steps while PPO rises more gradually. The faster convergence is attributed to PPO requiring critic warm-up, while GRPO computes advantages from group statistics immediately. However, faster convergence does not translate to better final performance — in several configurations, PPO's slower but more stable optimization leads to higher ultimate accuracy.

Response Length and Search Behavior Dynamics

Figure 2(c) and 2(d) track how the model's behavior evolves during training for Qwen2.5-7B-base with PPO. The response length (Figure 2(c)) follows a U-shaped trajectory: it decreases sharply in the first 100 steps (as the model eliminates filler words and adapts to the format), then increases substantially after step 100 (as the model learns to call search more frequently and incorporate retrieved passages). The training reward increases during both phases — modestly during the initial length decrease, and substantially during the later length increase when search becomes more frequent.

Figure 2(d) shows the number of valid search calls increasing from roughly 1.4 at the start of training to approximately 2.0 by step 200, correlating with the reward increase. This is direct evidence that the model learns that more search is instrumentally valuable — it discovers through the reward signal that additional retrieval rounds improve answer accuracy. This is not a behavior imposed by the template (which allows "as many times as you want" but does not encourage a specific number); it emerges from optimization.

SEARCH-R1 at 14B Scale

Appendix C (Table 5) reports results for Qwen2.5-14B. SEARCH-R1-base achieves an average of 0.479, further improving over the 7B results (0.431). The scaling trend is consistent: larger models benefit more from search-augmented RL. SEARCH-R1-14B-base more than doubles RAG performance on Bamboogle (0.528 vs. 0.192) and nearly doubles it on Musique (0.241 vs. 0.051). The paper uses these results to support the claim that "increasing the model size leads to consistent performance gains with SEARCH-R1, highlighting the benefits of LLM size scaling in our approach." However, the 14B results are only reported for SEARCH-R1, R1, and a subset of baselines — the full baseline comparison from Table 2 is not replicated at 14B.

Ablation Studies and Robustness Checks

Retrieved token masking (Table 4): Removing the loss mask on retrieved tokens causes substantial and consistent degradation. For Qwen2.5-7B-base with PPO, SEARCH-R1 with masking achieves 0.431 average; without masking, 0.343 — a drop of 8.8 points (20% relative). The degradation is present across all seven datasets, with particularly large drops on NQ (0.480 → 0.388), HotpotQA (0.433 → 0.325), and Musique (0.196 → 0.108). For Qwen2.5-3B-base (Table 6 in Appendix D), the pattern is similar: 0.303 with masking vs. 0.262 without — a 14% relative degradation. The training curves in Figure 3 (Appendix D) show that unmasked training not only achieves lower final reward but also exhibits greater instability — the training reward curve is both lower and noisier throughout. The paper interprets this as evidence that the masking "mitigates unintended optimization effects and ensures more stable training."

What this ablates: This experiment does not simply remove a convenience feature — it tests whether the conceptual distinction between policy-generated and environment-generated tokens matters in practice. The 20% performance gap is large enough to establish that this distinction is not a theoretical nicety but a practical requirement for stable search-augmented RL. Without masking, the policy gradient tries to optimize tokens the LLM does not control, introducing what the paper calls "unintended learning dynamics" — likely overfitting to superficial patterns in retrieved text.

Base vs. Instruct LLMs (Figure 2(b), Figure 4 in Appendix E): Instruction-tuned models converge faster and start from higher initial performance. For Qwen2.5-7B, the instruct model's training reward begins around 0.25 and rises to ~0.45, while the base model starts near 0.15 and rises to a comparable final level. However, for the 7B scale, the base model ultimately outperforms the instruct model (0.431 vs. 0.385 in Table 2) — meaning that despite slower convergence, the base model achieves higher final accuracy. For the 3B scale, instruct slightly outperforms base (0.325 vs. 0.303), suggesting a scale-dependent interaction. The paper's interpretation: "while general post-training accelerates learning in reasoning-plus-search scenarios, RL can effectively bridge the gap over time, enabling base models to achieve comparable performance." The 7B result complicates this — "comparable" understates that base actually exceeds instruct — but the paper does not offer a hypothesis for why base models might ultimately surpass instruct models at larger scales.

Number of retrieved passages (top-k) in training (Appendix G, Table 7, Figure 6): This is a study of how retrieval density during training affects learning. For Qwen2.5-7B-base with PPO at step 500, top-k=3 achieves the best average performance (0.431), followed by top-k=5 (0.400) and top-k=1 (0.375). The training curves (Figure 6) reveal a more nuanced dynamic: top-k=5 converges fastest initially, reaching the highest training reward within the first 200 steps, but its reward gradually declines and becomes more unstable as training progresses. Top-k=1 and top-k=3 show more consistent improvement, with top-k=3 ultimately achieving the highest reward after 500 steps. The paper hypothesizes two mechanisms: (1) top-k=1 suffers from low retrieval recall, limiting access to necessary information; (2) top-k=5 introduces lower precision due to noisy or irrelevant passages, which "not only degrades inference performance but may also adversely affect RL training — discouraging the model from leveraging retrieved content when it learns that the additional context is often unhelpful or misleading." This second mechanism is particularly interesting: too much retrieval noise during training may teach the model to ignore retrieved content entirely, undermining the purpose of search-augmented RL. The finding that the optimal training-time top-k (3) matches a commonly used inference-time setting (Lin et al., 2023) provides practical guidance.

GRPO group size (Appendix H, Table 8, Figure 7): For Qwen2.5-7B-base with GRPO, three group sizes are tested: 1 (equivalent to REINFORCE), 3, and 5. Group size 1 achieves the best average performance (0.410), followed by size 5 (0.350) and size 3 (0.363). This is a counterintuitive result — larger groups provide more stable advantage estimates but apparently lead to worse generalization, particularly on out-of-distribution datasets. For example, on 2WikiMultiHopQA, group size 1 achieves 0.413 vs. size 5 at 0.297. The training curves (Figure 7) show that larger groups converge faster (higher reward early) but are more prone to instability. The paper notes: "While larger group sizes can accelerate convergence and achieve higher training rewards, smaller group sizes (e.g., size = 1) enable more stable training and better generalization." This is a speed-stability-generalization tradeoff: larger groups help during early training but may cause the policy to overfit to the training distribution or exploit the group-based advantage computation.

PPO vs. GRPO training dynamics (Figure 5, Appendix F): This is not a single ablation but a systematic comparison across four model configurations (3B-base, 3B-instruct, 7B-base, 7B-instruct). The consistent pattern is that GRPO rises faster but PPO maintains stability longer. The reward collapse with GRPO is visible in all four subfigures of Figure 5 — the GRPO curve peaks and then declines, while the PPO curve plateaus or continues a slow increase. The collapse is most dramatic for 3B-base (declining from ~0.35 to ~0.25) and least dramatic for 7B-instruct (declining only slightly at the very end). This ablation establishes that the choice of RL algorithm materially affects training dynamics, even though Table 3 shows that final performance differences are configuration-dependent.

Negative result: ReST-EM for revision models (Appendix K): The paper reports (in passing, as part of the ablation discussion) that an attempt to further optimize the revision model using an RL-based self-improvement method led to performance degradation. While this experiment is in the appendix and related to a different modeling choice, the paper notes it as evidence that naive self-improvement loops can backfire in retrieval-augmented settings, and that the training methodology is sensitive to specific design choices.

Critical Assessment

The experiments in this paper demonstrate that RL can train an LLM to interleave reasoning with search engine calls using only outcome-based rewards, and that this training produces models that outperform both prompted search methods and standard RAG baselines. However, the strength of evidence varies across the paper's specific claims, and several important questions remain unaddressed.

Claim 1: "SEARCH-R1 improves performance by 24% (Qwen2.5-7B) and 20% (Qwen2.5-3B) over various RAG baselines under the same setting."

Evidence: Table 2 provides per-dataset comparisons. For Qwen2.5-7B, averaging the relative improvements across base and instruct variants yields something in the 20-42% range depending on how you compute the average. The exact 24% and 20% figures appear to be averaged across base and instruct variants — the paper does not provide the precise computation, but the per-cell numbers are consistent with these headline figures.

Assessment: The claim is supported at face value, but "various RAG baselines" is somewhat misleading. The primary comparison is against standard single-round RAG. Against prompted multi-turn methods (IRCoT, Search-o1), the gains are much larger — typically 80-110% relative. Against rejection sampling (which also uses multi-turn search), the gains are smaller — roughly 24% for 7B-base and 14% for 7B-instruct. The headline number obscures this heterogeneity. A more precise claim would be: "SEARCH-R1 improves over standard RAG by 20-42% depending on model variant, with substantially larger gains over prompted search methods and smaller but consistent gains over rejection sampling with search."

What is not tested: The paper does not compare against RAG with iterative retrieval (where initial retrieval results are used to reformulate queries for additional retrieval rounds) — a stronger baseline than single-round RAG that would partially close the gap without requiring learned query formulation. It also does not compare against fine-tuned RAG models where the retriever itself is optimized (e.g., REALM, RAG-end2end training). These comparisons would help distinguish whether SEARCH-R1's advantage comes from multi-turn interaction, from learned query formulation, or from RL optimization specifically.

Claim 2: "Outcome-based reward functions are sufficient to guide the LLM to learn meaningful and consistent search behaviors."

Evidence: The case studies (Appendix I and J, Tables 9-20) provide qualitative examples of emergent search behaviors: multi-hop decomposition, self-verification through additional searches, early stopping when evidence is insufficient, and strategic query reformulation. The training dynamics (Figure 2(d)) show search frequency increasing over training, indicating the model learns to search more. The performance gains over R1 (RL without search) in Table 2 demonstrate that adding search to the RL process improves outcomes.

Assessment: The evidence is suggestive but incomplete. The case studies are cherry-picked examples — the paper presents 6 successful cases and 5 failure cases, but does not provide a systematic behavioral analysis (e.g., what fraction of trajectories exhibit self-verification, how often does the model search unnecessarily, what is the distribution of search counts across question types). The increase in search frequency (Figure 2(d)) could indicate learning to search strategically, or it could simply indicate learning that "more search = higher reward" without genuine strategic discrimination. The paper does not analyze whether the model searches more on harder questions (which would indicate strategic behavior) or uniformly on all questions (which would indicate a learned heuristic rather than genuine information-seeking).

A missing experiment: A behavioral analysis that bins questions by difficulty (e.g., number of reasoning hops required, whether the answer is in parametric knowledge) and measures search behavior per bin would distinguish strategic from heuristic search. If the model searches more on multi-hop questions and less on single-hop questions, that would be strong evidence for learned strategy. If search frequency is uniform across question types, the learned behavior might be simpler than claimed.

Claim 3: "SEARCH-R1 generalizes across both in-distribution and out-of-distribution evaluation."

Evidence: Table 2 shows SEARCH-R1 outperforming baselines on all seven datasets, including five not used in training. The OOD gains are particularly strong on TriviaQA (0.638 for 7B-base), Bamboogle (0.432), and PopQA (0.457).

Assessment: The generalization claim is supported in the narrow sense that performance improves on OOD datasets. However, the absolute performance varies dramatically across OOD datasets — from 0.638 (TriviaQA) to 0.196 (Musique) for 7B-base — which is larger than the gap between SEARCH-R1 and baselines on some datasets. This suggests that SEARCH-R1 does not equally improve search-and-reasoning for all question types. The paper does not analyze what makes Musique hard (compositional complexity, entity linking difficulty, retrieval quality) or whether the learned search strategies transfer poorly to specific question structures. A finer-grained analysis would strengthen the generalization claim by identifying where generalization succeeds and fails.

What is not tested: The paper does not evaluate on datasets from different domains (e.g., scientific QA, legal reasoning, medical QA) or different languages. All seven datasets are English-language factoid QA with short-answer ground truth. The generalization claim should be understood as "generalizes across QA datasets with similar structure to the training distribution" rather than "generalizes to arbitrary search-and-reasoning tasks."

Claim 4: "Retrieved token masking stabilizes RL training."

Evidence: Table 4 and Figure 3 show consistent performance degradation without masking — from 0.431 to 0.343 average for 7B-base (20% relative) and from 0.303 to 0.262 for 3B-base (14% relative). The training curves in Figure 3 show that unmasked training is both lower and noisier.

Assessment: This claim is strongly supported. The effect is large, consistent across model scales, and present across all seven datasets. The paper provides a clear mechanism (preventing credit assignment to tokens the policy does not control) and the empirical magnitude confirms that this is a first-order effect, not a minor hyperparameter sensitivity.

A caution: The ablation tests only one alternative — no masking at all. It does not test alternative masking strategies (e.g., masking only certain types of retrieved tokens, or using soft masking with reduced weight on retrieved tokens rather than complete exclusion). The claim that masking is "necessary" is supported, but the claim that this specific binary masking scheme is optimal is not tested.

Missing Experiments and Unexamined Questions

1. Difficulty-stratified analysis: The paper does not analyze whether SEARCH-R1's benefits are concentrated on easy, medium, or hard questions. For comparable work on reasoning (e.g., the reference paper on compute-optimal test-time scaling), difficulty-dependent analysis revealed that some strategies help on easy questions but hurt on hard ones. A similar analysis for SEARCH-R1 — breaking down performance by question difficulty (number of hops, entity rarity, parametric knowledge coverage) — would reveal whether the learned search strategy is universally beneficial or has specific failure regimes. The Musique results (0.196, substantially lower than other datasets) hint that there may be such regimes, but the paper does not investigate.

2. Search engine quality sensitivity: All experiments use E5 over Wikipedia 2018 with top-3 retrieval. How does SEARCH-R1 perform with a weaker retriever (BM25, smaller corpus) or a stronger one (larger corpus, reranking)? If the learned search strategies are robust to retrieval quality, that would strengthen the claim that the model learns genuine information-seeking rather than exploiting retrieval-specific patterns. If performance degrades sharply with retrieval quality, the learned behavior might be brittle.

3. Training data composition: The training data merges NQ and HotpotQA — one single-hop and one multi-hop dataset. How does training on only single-hop or only multi-hop data affect the learned strategies and generalization? This ablation would reveal whether the model learns separate strategies for different question types or a unified search policy.

4. Computational cost analysis: The paper reports hardware configuration (8×H100 GPUs, 500 training steps) but does not compare training cost to baselines. How much more expensive is SEARCH-R1 training than SFT or R1 training? The RL process requires generating rollouts with search engine calls, which is more expensive per step than standard SFT. A FLOPs or wall-clock comparison would help practitioners decide whether the performance gains justify the training cost.

5. Action budget sensitivity: The maximum search budget B = 4 is fixed. How does performance scale with B? If the model learns to use search strategically, increasing B should benefit complex questions without hurting simple ones. If performance saturates at B = 2-3, the current setting might be wasteful for most questions.

6. Token masking ablation granularity: The paper ablates masking vs. no-masking. It does not test whether masking is equally important for PPO and GRPO, or whether the KL divergence masking (applied in addition to the policy gradient masking) independently contributes to stability. The claim that masking is necessary is supported, but the relative importance of masking for the gradient vs. masking for the KL term is unknown.

7. Statistical significance: The paper reports no confidence intervals, standard deviations, or significance tests. For a 500-question test set (MATH-style benchmarks in similar work), point estimates can be noisy. The per-dataset sample sizes vary — NQ and TriviaQA test sets are typically larger (thousands of questions) while Bamboogle is much smaller (125 questions per the original paper). Without variance estimates, it is difficult to assess whether apparent differences between methods (e.g., GRPO vs. PPO for 3B-instruct, where the gap is 0.336 vs. 0.325) are meaningful or noise.

8. The base-vs-instruct reversal at 7B: For 7B, SEARCH-R1-base (0.431) substantially outperforms SEARCH-R1-instruct (0.385). For 3B, instruct slightly outperforms base (0.325 vs. 0.303). The paper does not explain this interaction. Possible hypotheses (not tested): instruction tuning may constrain the policy search space in ways that help smaller models (by providing useful priors) but hurt larger models (by limiting exploration). This is a genuinely interesting finding that receives almost no analysis.

6. Limitations and Trade-offs

6.1 Single Benchmark Family and Single Domain: All Results Are on Factoid QA with Wikipedia Retrieval

The assumption or constraint. Every experiment in this paper — training, evaluation, ablation, and case study — uses exactly one task format: English-language factoid question answering with short-answer ground truth, evaluated via exact string match, with retrieval over the 2018 Wikipedia dump. The seven benchmark datasets (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle) all share this structure. The paper does not evaluate on code generation, mathematical reasoning, scientific QA, multi-lingual tasks, open-ended generation, or any domain where the "correct answer" is not a short extractable string. The authors do not explicitly address this limitation, though the conclusion gestures toward future work on "multimodal reasoning tasks" and "diverse set of tools."

The consequence. A practitioner deciding whether to adopt SEARCH-R1 cannot determine, from this paper, whether the learned search-and-reasoning capability transfers to tasks with fundamentally different structure. Factoid QA has several properties that make it particularly amenable to the approach: (1) answers are short, extractable strings, enabling clean outcome-based rewards via exact match; (2) the knowledge required is typically contained in Wikipedia, meaning the search engine is well-matched to the information need; (3) questions have a single ground-truth answer, eliminating ambiguity in reward assignment. In domains where correctness is subjective (dialogue, creative writing), multi-dimensional (scientific reasoning requiring both factual accuracy and logical validity), or requires long-form generation (summarization, explanation), both the reward function and the search interaction pattern would need substantial redesign. The paper provides no evidence about whether the framework generalizes.

What evidence exists in the paper. The paper's entire empirical contribution is bounded by this domain constraint. All seven datasets (Section 4.1) are factoid QA. The reward function (Equation 4) is exact match, which requires a clean ground-truth string. The search corpus is exclusively Wikipedia 2018 (Appendix B). There is no negative result on a non-QA task, nor any discussion of what would break if the task format changed.

Mitigation status. Not addressed. The conclusion mentions "integration with diverse information sources beyond search" and "applicability to multimodal reasoning tasks" as future work, but the paper does not test any variation in task format, reward structure, or knowledge source.

6.2 Training and Inference Require a Search Engine in the Rollout Loop — Latency and Infrastructure Costs Are Not Analyzed

The assumption or constraint. SEARCH-R1 training requires running a search engine as part of the RL rollout loop. During each training step, the policy LLM generates tokens until it emits a </search> tag, at which point the system pauses generation, calls the search engine, retrieves passages, inserts them into the trajectory, and resumes generation — potentially up to B = 4 times per trajectory (Section 3.2, Algorithm 1). At inference time, the same interleaved search-and-generation process occurs, making the generation inherently sequential and dependent on external API calls. The paper reports hardware configuration (8×H100 GPUs, Appendix B) and training step count (500 steps), but provides no measurement of wall-clock training time, inference latency, or search engine query cost.

The consequence. For a practitioner, the deployment decision hinges not just on accuracy but on whether the system can meet latency and throughput requirements. SEARCH-R1's interleaved architecture introduces serial dependencies that standard RAG avoids. In standard RAG, retrieval happens once before generation — the search and LLM inference can be pipelined. In SEARCH-R1, each search call blocks generation: the LLM must produce a query, wait for search results, process them, and decide whether to search again. A trajectory with 3 search calls incurs 3 round-trips to the search engine. For real-time applications (chatbots, interactive assistants), this serial dependency multiplies the end-to-end latency compared to a single retrieval round. The paper does not report whether the 24% accuracy gain over RAG (Table 2) comes at a 2×, 5×, or 10× latency cost.

The training cost is similarly opaque. RL training with multi-turn search rollouts is more expensive per step than standard SFT or R1 training (no search), because each trajectory involves multiple LLM generation segments interleaved with search engine queries. The paper acknowledges this implicitly by describing the rollout infrastructure (vLLM with GPU memory utilization ratio of 0.6, FSDP with CPU offloading) but never quantifies the total computational budget in GPU-hours or compares it to baseline training costs. A team deciding between SEARCH-R1 and rejection sampling (which uses the same search format but trains with standard SFT on positive examples) cannot evaluate the cost-accuracy tradeoff.

What evidence exists in the paper. None. Section 5 analyzes training dynamics (reward curves, response length evolution) but reports only step counts, not wall-clock time. Appendix B provides detailed hyperparameters and hardware specs but no timing measurements. The paper does not compare SEARCH-R1's training time to SFT, R1, or rejection sampling. There is no inference latency analysis. This is a conspicuous absence for a systems paper that introduces a new training infrastructure.

Mitigation status. Not addressed. The paper does not acknowledge the absence of cost/latency analysis as a limitation.

6.3 The Method Cannot Solve Problems Requiring Knowledge Beyond the Search Engine's Corpus — and Provides No Benefit When the Base Model Already Knows the Answer

The assumption or constraint. SEARCH-R1's effectiveness is bounded on two sides by the relationship between the question, the search corpus, and the base model's parametric knowledge. On the upper bound: if the base model already knows the answer from pretraining, search is unnecessary and potentially distracting. On the lower bound: if the search corpus does not contain the answer, no amount of learned query formulation can retrieve it. The paper's framework operates between these bounds — it helps when the knowledge exists in the corpus but the model cannot access it through simple retrieval or parametric recall alone.

The consequence. The method offers no path to answering questions that require knowledge genuinely absent from the search corpus. This is not a theoretical edge case — it is the defining limitation of any retrieval-augmented system. If a question requires information from paywalled sources, non-public databases, recent events not yet indexed, or tacit knowledge not expressed in text, SEARCH-R1 cannot help regardless of how well it learns to formulate queries. The paper provides indirect evidence for this limitation in the Musique results (Table 2): SEARCH-R1-7B-base achieves only 0.196 EM on Musique, which, while better than RAG at 0.058, is still far below performance on other datasets (0.638 on TriviaQA). The paper does not analyze whether the low absolute performance on Musique reflects retrieval corpus limitations (information not in Wikipedia), question complexity that exceeds the model's reasoning capacity, or both.

Conversely, on questions where the base model's parametric knowledge already suffices, SEARCH-R1 may search unnecessarily, wasting computation and potentially introducing retrieval noise that degrades performance. The paper does not measure how often the model searches when it already knows the answer, or whether unnecessary search ever hurts performance. The case studies (Appendix J) show instances where the model searches multiple times for information it arguably could have recalled (e.g., Britney Spears' birthplace), but there is no aggregate analysis of search efficiency.

What evidence exists in the paper. The Musique results (0.196 for 7B-base in Table 2) hint at a hard ceiling, but the paper does not investigate its cause. The R1 baseline (RL without search) provides an estimate of parametric knowledge coverage: on TriviaQA, R1-7B-base achieves 0.539, indicating substantial parametric knowledge; on NQ, it achieves only 0.297, indicating greater reliance on retrieval. The gap between R1 and SEARCH-R1 (0.539 → 0.638 on TriviaQA; 0.297 → 0.480 on NQ) shows that search helps more when parametric knowledge is weaker — consistent with the claimed bounds — but the paper does not systematically analyze the relationship between parametric knowledge coverage and SEARCH-R1 benefit.

Mitigation status. The paper does not discuss this fundamental capability bound. The design choice B = 4 (maximum search calls) implicitly acknowledges that infinite search is wasteful, but the paper does not analyze whether the model learns to search less on questions where parametric knowledge suffices or more on questions where it doesn't — which would be the hallmark of strategic search behavior.

6.4 The Reward Function Assumes Exact-Match Answer Extraction — and the Framework Provides No Mechanism for Tasks Without Clean Ground-Truth Answers

The assumption or constraint. The entire training signal for SEARCH-R1 comes from a binary exact-match comparison between the extracted answer string and a ground-truth reference (Equation 4, Section 3.4). This requires: (1) every training question must have a single, unambiguous, short-answer ground truth; (2) the model must output its answer in a format from which the answer can be reliably extracted (the <answer> tags); and (3) correctness must be reducible to string equality. The paper acknowledges this limitation only implicitly, by selecting datasets that satisfy these requirements and noting in Section 3.4 that "for instance, in factual reasoning tasks, correctness can be evaluated using rule-based criteria such as exact string matching" — implying that the reward design is task-specific.

The consequence. SEARCH-R1 cannot be directly applied to tasks where correctness is not a binary string-match property. This excludes a large fraction of real-world LLM applications: summarization (multiple valid summaries), translation (multiple valid translations), dialogue (no single correct response), code generation (functional correctness, not string match), mathematical reasoning (equivalent expressions), and any task requiring nuanced human judgment. Extending the framework to these domains would require a fundamentally different reward formulation — a learned reward model, process-based rewards, or human feedback — each of which introduces the complexity and instability the paper deliberately avoided by choosing exact match.

Even within the factoid QA domain, exact match has known limitations. Answers that are semantically correct but phrased differently (e.g., "USA" vs. "United States") score zero. The paper uses NQ, TriviaQA, and other datasets where answers are typically short named entities, minimizing this issue, but does not discuss how reward sparsity affects learning when semantically equivalent answers receive different rewards due to surface-form variation.

What evidence exists in the paper. The paper's entire evaluation uses exact match (Section 4.3, Appendix B), and the reward function is explicitly defined as exact match (Equation 4). This equivalence between training signal and evaluation metric is methodologically clean — what you optimize is what you measure — but it also means the paper provides no evidence about what happens when the reward signal is noisier, more subjective, or multi-dimensional. The success of SEARCH-R1 is conditional on the existence of a clean, extractable ground truth.

Mitigation status. The paper acknowledges this implicitly by framing exact match as "for instance" (Section 3.4) and leaving "more complex format rewards for future work." However, the difficulty of extending to non-exact-match domains is not discussed as a limitation. This is a significant omission, since much of the paper's claimed simplicity ("straightforward outcome-based reward function, avoiding the complexity of process-based rewards") rests on the exact-match assumption.

6.5 No Difficulty-Stratified Analysis — The Method's Benefit May Be Concentrated on Questions of Intermediate Difficulty, with Degradation at Extremes

The assumption or constraint. The paper reports aggregate performance across entire test sets and makes claims about average improvement (24% for 7B, 20% for 3B). It does not analyze whether these gains are uniform across question difficulty, or whether SEARCH-R1 helps on some questions while hurting on others. The paper has access to all the information needed for such an analysis — question type (single-hop vs. multi-hop), the number of search calls the model makes, whether the base model's parametric knowledge covers the answer — but does not perform it.

The consequence. For a practitioner, the aggregate number obscures a critical deployment consideration: if SEARCH-R1 improves performance on medium-difficulty questions but degrades on easy ones (by introducing unnecessary search noise) or fails on hard ones (beyond retrieval corpus or reasoning capacity), then the optimal deployment strategy might be to use SEARCH-R1 selectively — routing questions by difficulty — rather than applying it uniformly. Without difficulty-stratified results, the paper cannot guide this decision.

There are hints in the data that difficulty-dependent effects exist. The performance gap between SEARCH-R1 and baselines varies dramatically across datasets: the R1-to-SEARCH-R1 gap is large on NQ (0.297 → 0.480) but small on TriviaQA (0.539 → 0.638) for 7B-base. This suggests that SEARCH-R1's benefit depends on how much the question relies on retrieval versus parametric knowledge. Within the multi-hop datasets, the gap is large on Bamboogle (0.296 → 0.432) but small on Musique (0.083 → 0.196), suggesting that SEARCH-R1 helps more on some multi-hop structures than others. The paper does not investigate why.

What evidence exists in the paper. The per-dataset breakdown in Table 2 provides coarse evidence of heterogeneity, but there is no systematic difficulty binning. The case studies (Appendix I, J) show qualitative examples of both success and failure, but these are illustrative, not representative. The response length and search count trajectories (Figure 2(c), 2(d)) show aggregate trends but do not break down by question type.

Mitigation status. Not addressed. The paper does not frame the absence of difficulty analysis as a limitation, nor does it provide guidance on when a practitioner should expect SEARCH-R1 to help versus when alternative approaches (pure parametric reasoning, standard RAG, or simply a larger model) would be preferable. The FLOPs-matched comparison framework in the reference paper (compute-optimal test-time scaling) demonstrates how valuable difficulty-conditioned analysis can be for this type of claim — SEARCH-R1 would benefit from a similar treatment.

6.6 The Learned Search Strategy May Be Brittle to Retrieval Quality — and the Paper Tests Only One Retriever and Corpus Configuration

The assumption or constraint. All experiments use a single retrieval configuration: E5 dense retriever over the 2018 Wikipedia dump, with top-3 passages returned per query (Section 4.3, Appendix B). The paper does not test SEARCH-R1 with a different retriever (BM25, a different dense model, a larger corpus), a different number of retrieved passages during inference (only training-time top-k is ablated in Appendix G), or on a different knowledge source. The paper also does not test what happens when retrieval quality degrades — for example, when the corpus is noisier, less comprehensive, or in a different domain.

The consequence. The learned search strategy may be coupled to the specific retrieval characteristics of E5-over-Wikipedia. A model trained to search with a particular retriever has learned, through the RL process, what kinds of queries produce useful results from that retriever and that corpus. If deployed with a different retriever — one with different strengths, weaknesses, or coverage patterns — the learned query formulation strategy may be suboptimal or counterproductive. Similarly, if the corpus distribution shifts (e.g., from Wikipedia to domain-specific documents), the model's learned heuristics for when to search and what to query may not transfer. The paper provides no evidence about whether SEARCH-R1's search strategy generalizes across retrieval configurations or whether it effectively "overfits" to E5-on-Wikipedia.

The training-time top-k ablation (Appendix G) provides indirect evidence of brittleness: changing from top-3 to top-5 during training reduces average performance from 0.431 to 0.400 (Table 7), and top-5 actually causes reward instability after initial fast convergence (Figure 6). This sensitivity to the number of retrieved passages suggests that the learned behavior is tightly coupled to the retrieval environment it was trained in.

What evidence exists in the paper. Only the training-time top-k ablation (Appendix G) varies the retrieval configuration, and it does so only during training, not at inference time. There is no experiment where a model trained with E5 is evaluated using a different retriever, or where the corpus is changed. The paper does not report retrieval quality metrics (recall@k, precision) for the E5-Wikipedia configuration, making it impossible to contextualize the results relative to retrieval quality.

Mitigation status. Not addressed. The paper does not discuss retrieval robustness as a limitation or propose experiments to characterize the coupling between learned search strategy and retrieval configuration. Given that real-world deployments often involve changing corpora (updated Wikipedia dumps, domain-specific document collections) and that different retrievers have different failure modes, this is a significant practical gap.

7. Implications and Future Directions

How This Work Changes the Landscape

SEARCH-R1 is best understood not as a paradigm shift but as a methodological reframing — it changes the conversation around search-augmented LLMs from "how do we prompt or supervise the model to search?" to "how do we set up an RL environment where the model discovers search strategies through outcome optimization?" This is an incremental but consequential shift because it breaks a conceptual logjam that had divided the field into two camps, each with a ceiling they could not see past.

The field before SEARCH-R1. Prior work on LLMs and search operated in two largely non-overlapping paradigms. The retrieval-augmented generation (RAG) community treated search as a preprocessing step — retrieve first, then generate. The tool-use community treated search as a skill to be taught through supervised demonstrations — show the model when and how to call search, then fine-tune on those examples. Both paradigms shared an implicit assumption: the model's search behavior must be prescribed either through a fixed pipeline (RAG) or through labeled trajectories (tool-use SFT). The field's energy went into designing better retrieval architectures and collecting higher-quality demonstration data.

What SEARCH-R1 reframes. The paper's central move is to treat the search engine not as a preprocessing module or a skill to be mimicked, but as part of the RL environment — a non-differentiable system that the policy interacts with and receives rewards through, but that sits outside the gradient computation graph. This reframing matters because it converts a data problem (we need more annotated search trajectories) into an optimization problem (we need a training signal that survives the non-differentiable retrieval boundary). The paper explicitly identifies the technical barrier that makes this conversion non-trivial:

"the inherent non-differentiability of the search operation ... renders end-to-end gradient descent-based optimization inapplicable"

Prior approaches accepted this barrier as a hard constraint and worked around it (SFT on trajectories, prompting at inference). SEARCH-R1 recognizes it as the wrong constraint — policy gradient methods do not need differentiability through the environment. They need only a reward signal at the end, and the ability to sample trajectories with sufficient diversity that statistical credit assignment can work. This is not a new insight in RL generally (policy gradient methods have been optimizing non-differentiable environments for decades), but it is a genuinely new application of that insight to search-augmented LLM training.

Resolving prior contradictions. The paper partially reconciles a tension in the literature that was visible but not explicitly articulated. On one hand, prompting methods like IRCoT and ReAct showed that LLMs can interleave reasoning and search when given the right template — evidence that the capability is latent in the model. On the other hand, these prompted methods underperform simple single-round RAG on many benchmarks — evidence that the latent capability is incomplete and that prompting alone cannot elicit optimal search behavior. SEARCH-R1's results (Table 2: SEARCH-R1 at 0.431 average vs. IRCoT at 0.239 for 7B-base) resolve this tension: the capability is not merely latent — it requires optimization through interaction to develop fully. The LLM possesses the raw capacity to reason and search, but the strategy of when and how to search emerges from the RL process, not from a template. This is consistent with the broader finding from DeepSeek-R1 that sophisticated reasoning behaviors emerge from RL even when they were never demonstrated.

Which research directions become more attractive. The paper makes three research directions substantially more attractive than they were before. First, agentic RL for tool use — expanding from search engines to APIs, databases, code interpreters, and other non-differentiable tools — becomes a direct extension of the SEARCH-R1 framework rather than a separate problem requiring new optimization methods. The retrieved token masking technique provides a template for handling any environment-generated tokens in the gradient computation. Second, scaling search-augmented RL — the paper shows that larger models benefit more from the training (7B > 3B, and the 14B results in Appendix C continue the trend), suggesting that scaling model size while maintaining the RL training infrastructure could yield further gains without architectural changes. Third, self-improvement through search — the paper's finding that RL with outcome rewards teaches the model to search strategically opens the possibility of using SEARCH-R1-trained models to generate higher-quality training data for themselves, creating a virtuous cycle. The rejection sampling baseline (0.348 average) already underperforms SEARCH-R1 (0.431), suggesting that SEARCH-R1's trajectories could serve as better training data for subsequent rounds.

Which research directions become less attractive. The paper effectively closes off the hypothesis that better prompting alone can match trained search behavior. The gap between prompted methods (IRCoT: 0.239; Search-o1: 0.206) and SEARCH-R1 (0.431 for 7B-base) is large enough — roughly 80-110% relative — that incremental prompting improvements are unlikely to close it. Similarly, the rejection sampling baseline (0.348) underperforms SEARCH-R1 by a substantial margin, suggesting that filtering for successful trajectories and doing SFT on them does not recover the full benefit of RL training. This makes pure SFT-based approaches to search-augmented training less attractive relative to RL-based approaches, at least when outcome supervision is available.

The key boundary condition. The paper establishes — more through its omissions than its claims — that SEARCH-R1's methodology is currently bounded by the availability of clean outcome rewards. The entire training signal is a binary exact-match comparison (Equation 4). This works for factoid QA with short-answer ground truth, but the paper provides no evidence that the framework extends to tasks where correctness is subjective, multi-dimensional, or requires long-form generation. This is not a failure of the method so much as a boundary condition for the current results: SEARCH-R1 demonstrates that outcome-based RL works for search-augmented reasoning, but only when "outcome" can be reduced to an unambiguous scalar. Extending beyond this boundary — to summarization, dialogue, creative generation, or tasks requiring process-level correctness — would require reward functions that this paper deliberately avoids. The field now knows that the approach works within this boundary, but does not yet know whether it can be generalized beyond it.

Follow-Up Research This Work Enables

Cheap, online difficulty estimation for adaptive search budgets. SEARCH-R1 currently uses a fixed maximum search budget $B = 4$ for all questions (Section 3.2). The case studies show the model sometimes stops searching early when it recognizes sufficient information has been gathered (Appendix J, Case Study 10: "The other three superheroes are not mentioned in the search result. I'll provide the answer based on the information I have"), but the training process does not explicitly reward search efficiency — it rewards only answer correctness. A natural extension would train the model with a cost-aware reward function that penalizes unnecessary search calls: $r = \text{EM}(a_{\text{pred}}, a_{\text{gold}}) - \alpha \cdot n_{\text{searches}}$, where $\alpha$ is a small cost coefficient. This would test whether the model can learn not just how to search strategically, but how much to search — deploying more retrieval on genuinely difficult multi-hop questions while answering simple factoid questions with zero or one search call. The key experiment would measure the accuracy-vs-search-cost Pareto frontier and compare it against a fixed-budget baseline. A negative result — the model fails to learn cost-sensitive stopping, or accuracy degrades sharply with even small cost penalties — would reveal whether the emergent strategic behaviors observed in case studies are robust enough to survive explicit efficiency pressure.

Cross-retriever robustness: does the learned search strategy transfer? All SEARCH-R1 experiments use E5 dense retrieval over Wikipedia 2018 (Section 4.3). This means the LLM has learned query formulation strategies that work for this specific retriever on this specific corpus. A critical stress test would train SEARCH-R1 with one retriever (e.g., E5) and evaluate with a different retriever (e.g., BM25, a different dense model like Contriever, or a commercial search API). If the learned query formulation transfers — the model maintains most of its accuracy gain even when the retrieval backend changes — that would be strong evidence that the RL process teaches general information-seeking strategies rather than retriever-specific query optimization. If accuracy drops sharply, it would indicate that SEARCH-R1's gains are partially attributable to the model learning to exploit the specific retrieval characteristics of E5 (e.g., the kinds of queries that retrieve high-quality passages from E5 might not work for BM25), and that deployment across different retrieval infrastructures would require retraining. This experiment would also inform whether SEARCH-R1-trained models can be deployed with black-box search APIs where the retriever cannot be modified.

Difficulty-stratified behavioral analysis: when does SEARCH-R1 help, and when does it hurt? The paper reports only aggregate performance per dataset (Table 2) and provides cherry-picked case studies (Appendix I, J). A systematic analysis that bins evaluation questions by difficulty — measured by the base model's parametric knowledge coverage (can R1 answer without search?), the number of reasoning hops required (single-hop vs. 2-hop vs. 3-hop questions), or retrieval difficulty (how often does a standard RAG baseline retrieve the answer in the top-3 passages) — would reveal whether SEARCH-R1's gains are concentrated on questions of intermediate difficulty, where the model can reason but lacks knowledge, or are uniform across all question types. This matters because the decision to deploy SEARCH-R1 versus simpler alternatives (standard RAG for easy questions, pure parametric reasoning for questions the model already knows) depends on whether SEARCH-R1 degrades performance on any difficulty tier. The reference paper on test-time compute scaling (Section 7 of the example analysis) demonstrated that some strategies improve easy questions while hurting hard ones — a similar analysis for SEARCH-R1 would establish whether the learned search strategy has analogous failure regimes. The Musique results (0.196 for 7B-base, far below other datasets) hint at such regimes but are not analyzed.

Combining SEARCH-R1 with process reward models for search quality. The paper deliberately uses only outcome rewards (Equation 4) and argues that this sparsity is sufficient. However, the paper also shows that models sometimes fail because they cannot decompose complex problems effectively (Appendix J, Case Study 2: the model asks repetitive queries and gets misled by irrelevant passages). An experiment that introduces a process reward for retrieval quality — for instance, a learned verifier that scores whether a retrieved passage is relevant to the current reasoning step, or a simple heuristic reward for retrieving passages that contain the ground-truth answer entities — would test whether augmenting the sparse outcome signal with intermediate retrieval-quality feedback further improves learning. The key comparison would be: SEARCH-R1 (outcome-only) vs. SEARCH-R1 + retrieval relevance reward, evaluated on complex multi-hop datasets (Musique, 2WikiMultiHopQA) where retrieval quality is likely the bottleneck. A negative result — process rewards provide no benefit or destabilize training — would strengthen the paper's minimal-reward thesis. A positive result — process rewards accelerate learning or improve final performance on hard questions — would suggest that the optimal reward design for search-augmented RL lies between the paper's extreme minimalism and fully supervised trajectory annotation.

Extending to non-QA domains with learned reward models. The paper's entire empirical evaluation is on factoid QA with exact-match rewards (Section 4.1). A critical generalization experiment would apply SEARCH-R1 to a domain where answers cannot be evaluated via string matching — for example, scientific reasoning (requiring both factual accuracy and logical validity), code generation (functional correctness tested by unit tests rather than string match), or long-form question answering (evaluated by automated metrics like ROUGE or learned evaluators). The experiment would replace the exact-match reward with a learned reward model trained on human preference judgments or automatic evaluation metrics, and test whether SEARCH-R1's RL framework remains stable and effective when the reward signal is noisier and less binary. This would directly test the paper's implicit claim that its framework does not depend on the specific properties of exact-match rewards. A negative result — training becomes unstable with learned rewards, or the model learns to exploit the reward model rather than genuinely improve — would clarify the boundary conditions established in Section 6.4 and would motivate research into reward model robustness specifically for search-augmented RL settings.

Open-domain multi-turn search with dynamic corpus updates. SEARCH-R1 uses a static 2018 Wikipedia dump (Appendix B). In real deployments, the knowledge corpus changes over time — new articles appear, existing articles are updated, and some information becomes outdated. An experiment that trains SEARCH-R1 on one version of Wikipedia and evaluates on a later version (e.g., a 2023 dump with events that postdate the 2018 dump) would test whether the learned search strategy includes temporal awareness — can the model recognize when retrieved information might be outdated and seek newer sources? This would require modifying the search environment to include document timestamps in the retrieved passages, and potentially adding a temporal-awareness component to the reward (e.g., penalizing answers that use outdated facts when newer information is available). The experiment would measure whether SEARCH-R1's multi-turn search capability includes the flexibility to handle evolving knowledge, or whether it assumes a static corpus.

Practical Applications and Downstream Use Cases

Cost-efficient training of search-augmented LLMs for specialized domains. Organizations that need LLMs to answer questions over domain-specific corpora — medical literature, legal documents, internal company knowledge bases — typically face a chicken-and-egg problem: they lack annotated search trajectories for their domain, but SFT-based tool-use training requires exactly such annotations. SEARCH-R1's key practical contribution is that it requires only question-answer pairs (which many organizations already have in the form of FAQs, support tickets with resolutions, or curated Q&A databases) rather than full search-and-reasoning demonstrations. A medical QA system could be trained by: (1) taking a dataset of (question, answer) pairs from medical licensing exams, (2) configuring SEARCH-R1 with a retriever over PubMed or a proprietary medical corpus, and (3) running RL training with exact-match rewards against the known answers. The model would learn to formulate medical queries, retrieve relevant literature, and synthesize evidence into answers — all without a single human-annotated search trajectory. The paper's results (24% improvement over RAG on QA benchmarks, Table 2) suggest that the resulting system would substantially outperform a standard RAG pipeline using the same retrieval infrastructure, while requiring only answer-level supervision that is often already available.

Improved multi-hop reasoning in enterprise search and knowledge management. Enterprise search systems — where employees query internal document collections, wikis, and databases to answer complex questions — currently rely on single-round retrieval followed by LLM summarization. Questions like "Which team is responsible for the Q3 revenue dip in the European market, and what corrective actions were proposed in the last review?" require synthesizing information from multiple documents (financial reports, organizational charts, meeting notes). Standard RAG retrieves documents based on the surface question text and often misses the connections between entities. SEARCH-R1's interleaved reasoning-and-search loop (Algorithm 1) is designed for exactly this pattern: the model can retrieve information about the European market, discover the relevant team, then search for that team's review documents, then synthesize. The paper's strong performance on multi-hop datasets — HotpotQA at 0.433 vs. RAG at 0.299, Bamboogle at 0.432 vs. RAG at 0.208 (Table 2, 7B-base) — provides direct evidence that the learned search strategy handles compositional information needs more effectively than single-round retrieval. The infrastructure requirement is modest: the same RL training setup described in Appendix B (8×H100 GPUs, 500 training steps) can be applied to a proprietary corpus with an organization's existing retriever.

Enabling smaller models to compete with larger ones through learned search. The paper's scaling results show that SEARCH-R1 helps all model sizes but the absolute gains increase with model scale (3B-instruct: 0.325, 7B-base: 0.431, 14B-base: 0.479; Tables 2 and 5). However, a different practical interpretation is available: a 7B model with SEARCH-R1 (0.431) outperforms a 14B model without search training (R1-14B-base: 0.357, Table 5) by 21% relative. This suggests that for organizations with limited GPU budgets for inference, training a smaller model with SEARCH-R1 may be more cost-effective than deploying a larger model with standard RAG. The smaller model has lower per-token inference cost and can run on cheaper hardware, while the learned search capability partially compensates for the smaller parameter count by enabling the model to access external knowledge more effectively. This is not a claim the paper makes explicitly, but the numbers support it: SEARCH-R1-7B-base exceeds the non-search 14B model on 6 of 7 datasets (all except Musique, where both struggle). For latency-sensitive applications where inference must run on-device or on CPU, this opens the possibility of deploying a 3B SEARCH-R1 model (0.325) instead of a 7B RAG model (0.304) — comparable quality with substantially lower hardware requirements.