ArXiv: 2603.11327

🎯 Pitch

Search agents trained with standard RL often plateau on mediocre strategies because they only receive a sparse reward at the very end. MR-Search breaks this deadlock by having the agent explicitly write a self-reflection after each failed episode and feeding it into the next attempt, coupled with a training algorithm that assigns credit at every intermediate turn. This simple loop—try, reflect, retry—enables a single agent to improve its own search strategy in-context at test time, boosting accuracy by up to 19% across eight QA benchmarks without any external process reward models.

1. Executive Summary

This paper introduces MR-Search, an in-context meta-reinforcement learning formulation that trains LLM-based search agents to perform cross-episode exploration via explicit self-reflection—after each trajectory, the agent generates a textual reflection on its previous attempt and conditions subsequent search episodes on that accumulated context. Experiments across eight QA benchmarks (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle, and ASearcher) using Qwen2.5-3B and Qwen2.5-7B base models demonstrate that MR-Search substantially outperforms outcome-reward-only RL baselines (Search-R1) and process-reward methods that rely on external models (PPRM, StepResearch), achieving average relative improvements of 9.2% to 19.3% over the strongest baselines. The method learns to balance exploration and exploitation end-to-end by meta-learning how to generate effective self-reflections, establishing that in-context adaptation across episodes can substitute for externally provided process rewards—but only when the training objective propagates credit across multiple reflective turns via a turn-level discounted advantage formulation rather than treating episodes as independent.

2. Context and Motivation

The Core Problem: Sparse Rewards Cripple RL-Based Search Agent Training

The fundamental challenge this paper addresses is deceptively simple yet practically crippling: when training LLM-based search agents via reinforcement learning, the only feedback available is whether the final answer is correct — yet the process of reaching that answer involves many intermediate decisions that go unrewarded. This matters because search is inherently a multi-step activity: an agent must decide what to search for, when to search, how to interpret retrieved results, whether retrieval was sufficient, and when to commit to a final answer. A binary signal at the end of this complex chain provides almost no information about which of those decisions were good and which were bad.

The consequences of this credit assignment problem are severe and well-documented in the RL literature. The paper explicitly identifies several failure modes in Section 1:

"Due to the sparse nature of outcome rewards, the agent often struggles to learn more complex processes and is susceptible to issues such as inefficient exploration at the early stage, local optima, and inefficient search dynamics"

These are not minor nuisances — they are fundamental barriers to training capable search agents. Inefficient exploration means the agent wastes training samples on unproductive search strategies before stumbling onto effective ones. Local optima mean the agent settles into a mediocre policy (e.g., always issuing a single generic search query and guessing) that earns some reward but never discovers better strategies. And inefficient search dynamics mean the agent performs redundant queries, fails to synthesize information across turns, or abandons promising lines of inquiry too early.

The paper argues these challenges become more pronounced in agentic search specifically, where "multi-turn interactions with tools could amplify small errors and obscure credit assignment" (Section 1). A single poor search query early in a trajectory can propagate forward: retrieved documents will be irrelevant, subsequent reasoning will be based on wrong information, and the final answer will be incorrect. From the RL perspective, every action in that trajectory receives the same negative outcome signal, making it impossible to distinguish the root cause (the bad query) from reasonable decisions that followed from the bad information. This is the credit assignment problem in its most acute form.

Why This Problem Matters: The Shift Toward Autonomous Search Agents

The practical significance of this problem is growing rapidly because of a broader shift in how AI systems are being deployed. The paper contextualizes its work within the emergence of agentic search and deep research capabilities (Section 1):

"Language models with advanced reasoning capabilities have driven substantial progress toward more autonomous and multi-step decision-making behaviors in complex tasks. Examples include agentic search such as deep research and other information seeking, where LMs use search tools and engage in dynamic, multi-turn interactions."

This is not abstract. Systems like OpenAI's deep research, Perplexity, and Google's AI Overviews all operate as multi-turn search agents — they decompose complex questions, issue multiple queries, read and cross-reference documents, and synthesize answers from disparate sources. Training these systems requires RL (because supervised data of high-quality search trajectories is scarce and expensive to collect at scale), yet effective RL requires informative reward signals that are absent in the natural outcome-only formulation.

The gap between what we want to train (agents that search intelligently across turns) and what we can train with naive RL (agents that optimize a terminal binary reward) represents a fundamental bottleneck for the entire paradigm of learned search agents. Without addressing this gap, scaling RL training (more compute, more data, larger models) yields diminishing returns because the learning signal itself is impoverished.

There is also a theoretical significance to this problem. Multi-turn tool use is a form of sequential decision-making under uncertainty, where the agent must balance exploration (trying different search strategies to discover what works) against exploitation (using known-good strategies to maximize correctness). The standard RL formulation with outcome rewards collapses this exploration-exploitation tradeoff into a single binary signal, making it impossible for the agent to learn how to explore effectively — a meta-skill that is arguably more important than any single search tactic.

Where Prior Approaches Fall Short

The paper identifies three families of prior approaches and their specific limitations:

1. Outcome-reward-only RL methods (Search-R1, ReSearch). These methods — epitomized by Search-R1 (Jin et al., 2025a) and ReSearch (Chen et al., 2025) — train LLMs under the ReAct paradigm using PPO or GRPO with only the final answer correctness as the reward signal. The agent interleaves reasoning and tool calls, receives retrieved documents, and eventually produces an answer. If the answer is correct, the entire trajectory is "good"; if incorrect, the entire trajectory is "bad." The paper acknowledges these methods "have demonstrated promising performance" (Section 3.1), but identifies their core limitation concisely:

"the outcome rewards are sparse and delayed, leading to ambiguous credit assignment and ineffective search exploration"

This is the same criticism leveled at early deep RL methods before advances like prioritized experience replay and curiosity-driven exploration — the feedback is too coarse to guide learning of complex multi-step behavior. The paper's Figure 3 visualizes the consequence: Search-R1 with sequential reflection turns shows almost no improvement with additional reflection steps, suggesting the model hasn't learned to use reflection effectively because its training objective never rewarded doing so.

2. Process reward methods using external models (PPRM, StepResearch). To address sparse rewards, several works introduce process rewards: feedback at intermediate steps of the search trajectory rather than only at the end. PPRM (Anonymous, 2026) trains a principle-based process reward model to provide step-wise signals during GRPO-based RL. StepResearch (Wang et al., 2025b) uses step-wise PPO with intermediate rewards and token-level supervision derived from an external verifier. The paper's results in Table 1 confirm these methods do outperform outcome-only baselines — StepResearch achieves 43.4% average accuracy vs. 42.1% for Search-R1 with Qwen2.5-7B, for example.

However, the paper identifies three specific drawbacks to this approach (Section 1):

"these approaches rely on external annotations, which are both costly and difficult to reuse when task requirements change. Moreover, model-based rewards inevitably lead to reward hacking and bias and incur additional computational overhead in the RL training."

The cost argument is practical but crucial: process reward models must be trained on step-level correctness labels, which typically require either human annotation (expensive, slow, domain-specific) or automated evaluation heuristics that may not generalize. The reward hacking concern is equally important — when an RL agent optimizes against a learned reward model rather than ground-truth outcomes, it tends to exploit imperfections in that model, producing trajectories that score highly under the model but are not actually correct. This is well-documented in RLHF literature (where reward over-optimization is a central challenge) and the paper correctly identifies it as a risk for process reward approaches in search.

The computational overhead is non-trivial: running a separate reward model alongside the policy model during training roughly doubles the inference cost per update, slowing down the RL training loop. For large-scale training runs with frontier models, this overhead can be prohibitive.

3. Prompting-based self-reflection methods (Reflexion, Self-Refine). Separate from RL training, prior work has explored whether LLMs can improve their own outputs through prompting-based self-reflection without any fine-tuning. Methods like Reflexion (Shinn et al., 2023), Self-Refine (Madaan et al., 2023), and self-correction (Huang et al., 2023) prompt the model to critique and revise its own outputs. The paper references these in Section 2 ("LLMs with Self-Reflection").

The critical finding from this literature — and the reason the paper does not simply adopt these methods as-is — is that prompting alone is insufficient for reasoning tasks. The paper explicitly cites Huang et al. (2023)'s finding that "large language models cannot self-correct reasoning yet" when relying solely on prompting. The issue is that the LLM's self-critique ability is limited by the same knowledge and reasoning capacity that produced the original error — the model cannot reliably identify its own mistakes without additional training to develop that meta-cognitive skill.

The paper positions itself against this backdrop: prompting-based reflection works for some tasks but fails on complex reasoning; fine-tuning-based reflection (SCoRe, Kumar et al., 2024; Qu et al., 2024) can work but hasn't been applied to multi-turn tool-use settings; and our meta-RL formulation provides a principled training objective for learning when and how to reflect in the specific context of agentic search.

4. Meta-RL approaches in non-LLM domains. The paper draws inspiration from meta-reinforcement learning methods developed in robotics and game-playing domains (Section 2). The key idea from this literature — RL² (Duan et al., 2016), E-MAML (Stadie et al., 2018), and algorithm distillation (Laskin et al., 2023) — is that an agent can be trained across multiple episodes of a task, using information from early exploratory episodes to guide later exploitation episodes. This inverts the standard RL framing: rather than resetting the agent's state after each episode, the agent carries forward learned information (often encoded in an RNN hidden state) that adapts its policy to the specific task instance.

The paper notes a critical difference between these prior meta-RL methods and the LLM-based agentic search setting:

"Unlike traditional meta-RL approaches in robotics and games, we focus on open-domain agentic search tasks with tool interactions and self-reflection, without any reward feedback from the environment during inference."

In robotics or games, the environment provides reward signals at every step (e.g., distance to goal, score). In open-domain QA, there are no intermediate reward signals — only the final answer can be evaluated against the ground truth, and even that is unavailable at inference time. The agent must learn to simulate its own process feedback through self-reflection, using that reflection to guide subsequent attempts, all without ever seeing a ground-truth label during deployment.

A concurrent work (Jiang et al., 2025) proposes meta-RL to encourage exploration in LLM agents with ground-truth state feedback — meaning the environment tells the agent at each step whether it's on track. The paper explicitly distinguishes its contribution: "in contrast, we focus on LLM open-domain agentic task and do not access to any environment feedback during inference." This is a substantially harder setting because the agent must generate its own feedback signal rather than relying on an oracle.

How This Paper Positions Itself Relative to Prior Work

The paper's positioning can be understood along three axes:

Axis 1: Reward source. The paper sits between outcome-reward-only methods (too sparse) and external-process-reward methods (expensive, risk of reward hacking). It proposes a self-generated process reward — not a separately trained reward model, but an implicit signal derived from the agent's own multi-turn performance. By having the agent generate multiple complete answers across episodes and comparing their correctness during training, the paper creates a dense learning signal without external annotation and without additional model overhead. This is the motivation for the grouped relative advantage formulation (Section 3.3): by comparing episode n across multiple meta-episodes rather than in isolation, the model receives a signal about how good this particular episode is relative to alternative approaches to the same question.

Axis 2: Training paradigm. The paper reframes agentic search from a single-episode RL problem to a meta-RL problem across episodes. The key insight is that search is naturally iterative — an agent should be able to look at its first attempt, identify what went wrong, and adjust its strategy for the next attempt. The meta-RL formulation makes this cross-episode adaptation the explicit training objective rather than an emergent property. This distinguishes MR-Search from both ReAct-style RL (which treats episodes as independent) and prompting-based self-reflection (which doesn't modify the model's learned behavior).

Axis 3: Computational architecture. The paper deliberately avoids the critic model that PPO requires and the process reward model that PPRM/StepResearch require. Instead, it uses a critic-free RLOO (leave-one-out) baseline for advantage estimation. This matters for practical scaling: adding a critic network doubles the model parameters that must be stored and updated during training, and adding a process reward model similarly increases computational cost. By keeping the training loop simple — just the policy model, sampled trajectories, and a rule-based verifier for final answers — MR-Search remains computationally lightweight while still providing the dense credit assignment that process reward methods aim for.

The paper's central hypothesis, which the experiments are designed to test, is that structured cross-episode self-reflection can serve as an effective substitute for externally provided process rewards, but only when the training objective properly accounts for the temporal dependencies between episodes (i.e., the discounted turn-level advantage) rather than treating each episode as contributing independently to the objective. The computational efficiency and avoidance of reward hacking are presented as additional practical benefits, but the core scientific claim is about the sufficiency of self-generated multi-turn feedback for training capable search agents.

3. Technical Approach

This is primarily a training methodology paper whose core idea is that search agents should be trained to treat multiple answer attempts as a single adaptive process — generating explicit self-reflections between attempts and conditioning subsequent attempts on that accumulated context — rather than as independent episodes that share no information.

3.1 Reader Orientation

What is being built: a training procedure that teaches an LLM-based search agent to improve its own answers by reflecting on its previous mistakes and searching for additional information, all within a single forward pass at test time. The system is the training algorithm itself — how to set up the RL objective, the episode structure, and the credit assignment mechanism so that the resulting model learns to use self-reflection effectively. The problem it solves is that standard RL for search agents provides only a single binary reward at the end of a long multi-turn trajectory, giving the model no signal about which of its many decisions (which search queries, which reasoning steps, which moments of deciding to stop searching) were good or bad. The solution's shape is to restructure the training process into nested loops of episodes within meta-episodes, where the model generates a complete answer, reflects on what went wrong, tries again with that reflection in context, and receives credit assignment at the granularity of whole episodes rather than individual tokens — all while avoiding the need for a separate critic network or externally trained process reward model.

3.2 Big-Picture Architecture (Diagram in Words)

The MR-Search training system has four major components operating in a nested loop structure:

  1. A base LLM policy (π_θ) — typically Qwen2.5-3B or Qwen2.5-7B — that serves as both the search agent and the self-reflection generator. The same model produces reasoning steps, search queries, final answers, and reflection text, all through standard autoregressive generation conditioned on accumulated context.

  2. A retrieval environment — an external Wikipedia-based search engine (2018 Wikipedia dump with E5 embeddings as the retriever) that takes a query string as input and returns the top-3 retrieved documents. This environment is non-differentiable and treated as part of the MDP transition dynamics; the policy only receives the text of retrieved documents as observation tokens.

  3. A rule-based outcome verifier — a function that compares the model's extracted final answer against the ground-truth answer using exact match after normalization. This is the only source of reward signal in the entire system; there are no learned reward models, no human annotations at intermediate steps, and no environment-provided feedback during inference.

  4. A multi-turn advantage estimator — a critic-free mechanism that computes relative advantages at the episode level (not the token level) by comparing the same episode index across multiple parallel meta-episodes within a group, then propagating those advantages backward through the episode sequence using a discount factor.

Information flows through these components in a specific nested order: a question enters the system → the policy generates a first complete search trajectory with a final answer → the system appends a reflection prompt → the policy generates a second complete search trajectory conditioned on the first trajectory and its reflection → this repeats for N episodes → the verifier scores each episode's final answer against ground truth → the advantage estimator computes relative advantages per episode position across G parallel meta-episodes → the policy is updated using a clipped surrogate objective on the token log-probabilities, with each token in episode n receiving the same episode-level advantage signal.

3.3 Roadmap for the Deep Dive

  • First, the single-episode ReAct formulation — because MR-Search builds directly on this foundation, and understanding the base interaction protocol is necessary before seeing how it is extended across episodes.
  • Second, the meta-episode structure and self-reflection mechanism — how multiple episodes are chained together, what the reflection prompt looks like, and why this transforms the learning problem from single-episode RL to meta-RL.
  • Third, the meta-level training objective — the mathematical formalization of what the system optimizes, including the discount factor and why the objective sums across episodes rather than treating each independently.
  • Fourth, the turn-level advantage estimation and policy optimization — the RLOO-based grouped advantage computation, the discounted cumulative advantage that propagates credit backward, and the clipped surrogate objective applied at the token level with episode-level advantage broadcasting.
  • Fifth, the exploration-exploitation masking extension — an optional mechanism for designating certain episodes as "exploration" (contributing context but not gradient) to encourage long-term adaptation over short-term reward.
  • Sixth, the step-level meta-RL extension — how the same principle can be applied at finer granularity within a single episode by treating each tool-interaction step as a micro-episode with an intermediate answer.

3.4 Detailed, Sentence-Based Technical Breakdown

Background: The Single-Episode ReAct Formulation

Before describing how MR-Search extends across episodes, it is essential to understand the base interaction protocol that operates within a single episode. The paper builds on the ReAct paradigm (Yao et al., 2022), which interleaves reasoning steps with tool-use actions. Given a question, the agent does not produce a single monolithic reasoning chain; instead, it cycles through a sequence of thought-action-observation triplets.

Concretely, when presented with a query, the policy model π_θ generates a sequence of alternating elements. It first produces an internal Thought (τ) — a reasoning step that articulates what the agent currently knows and what it needs to find out. It then produces an external Action (α) — a structured search query enclosed in <search> tags. The retrieval environment processes this query and returns Observations (x) — the top-3 retrieved Wikipedia documents, enclosed in <information> tags and injected into the context as tokens the model did not generate. The model then reads these observations and produces another Thought, followed by another Action, and so on. This cycle continues until the model decides to terminate search and produce a final answer, which it encloses in <answer> tags.

The formal representation of a single episode's trajectory is:

a=(τ0,α0,x0,τ1,α1,x1,…,τT−1)a = (\tau_0, \alpha_0, x_0, \tau_1, \alpha_1, x_1, \ldots, \tau_{T-1})

where $a$ is the complete interaction trajectory, $\tau_i$ is the i-th reasoning thought, $\alpha_i$ is the i-th search action, $x_i$ is the i-th retrieved observation (tool output), and $\tau_{T-1}$ is the final thought containing the answer $o$ without any further actions. The index runs from 0 to $T-1$ because the final round contains only the answer thought.

What this represents: a single complete attempt at answering the question, from initial reasoning through multiple search-and-read cycles to final answer production. Each trajectory is a variable-length sequence whose length depends on how many times the model chooses to search before answering.

Why this structure: the alternation of reasoning and search allows the model to pursue information-seeking strategies that adapt to what it finds. Rather than pre-planning all queries or doing a single search, the agent can refine its search based on retrieved results — reading a document might reveal that it needs to search for a different entity, or that it already has sufficient information to answer.

Given this interaction process, standard RL-based search agent training (as in Search-R1) maximizes the following objective:

J(πθ)=Ea∼πθ[fverifier(o,o∗)]J(\pi_\theta) = \mathbb{E}_{a \sim \pi_\theta}\left[f_{\text{verifier}}(o, o^*)\right]

where $J(\pi_\theta)$ is the expected return under the policy, $\pi_\theta$ is the policy model parameterized by $\theta$, $a$ is a trajectory sampled from the policy, $o$ is the final answer extracted from that trajectory, $o^*$ is the ground-truth answer, and $f_{\text{verifier}}$ is a rule-based verifier (exact match after normalization) that returns a binary or scalar reward.

What it computes: the expected value of the verifier's judgment over trajectories sampled from the policy. In practice, this is approximated by sampling a batch of trajectories, computing the verifier score for each, and updating the policy to increase the probability of trajectories that received high scores.

Why this form (and why it's insufficient): this is the standard REINFORCE-style objective for episodic RL with a terminal reward. The problem is that the reward $f_{\text{verifier}}(o, o^*)$ is a single scalar applied to the entire trajectory — every token in the trajectory receives the same positive or negative signal, regardless of whether specific reasoning steps or search queries were good or bad. This is what the paper means by "sparse outcome rewards" leading to "ambiguous credit assignment." In a trajectory with 5 search queries where the final answer is wrong, the model cannot distinguish the query that retrieved misleading information from the query that retrieved useful information but was misinterpreted.

The Meta-Episode Structure and Self-Reflection Mechanism

The core innovation of MR-Search is to restructure the training process so that multiple complete answer attempts are chained together, with each attempt conditioned on all previous attempts and an explicit textual reflection. This transforms the learning problem from optimizing individual trajectories to optimizing a sequence of trajectories where later episodes can benefit from the "experience" accumulated in earlier ones.

The process begins identically to the standard ReAct formulation: given an input question, the policy generates a first complete episode following the thought-action-observation cycle until it produces a final answer:

a0∼πθ(a)a_0 \sim \pi_\theta(a)

What this represents: the first complete search trajectory for the given question — the same as what a standard RL agent would produce as its only attempt.

After this first episode completes (the model has produced its answer inside <answer> tags), the system injects a reflection prompt into the context. This prompt is a fixed template that instructs the model to:

"Reflect on your current answer to the question and provide another answer by searching for additional external information using search engines."

The detailed reflection prompt (provided in Appendix A.1.3) instructs the model to conduct reasoning inside <thinking> tags, call the search engine with <search> tags if it lacks knowledge, and provide an answer inside <answer> tags — the same interaction protocol as the initial episode, but with the explicit meta-instruction to reflect on the previous answer and improve it.

The model then generates a second episode, but now its context includes the entire first episode (question, thoughts, queries, retrieved documents, answer) followed by the reflection prompt. Formally:

a1∼pθ(a1∣a0)a_1 \sim p_\theta(a_1 \mid a_0)

What this represents: the second complete search trajectory, generated with full visibility into the first attempt. The model can see what it originally searched for, what documents it retrieved, what answer it gave, and it receives an explicit prompt to reflect on that answer and improve it.

This chaining process repeats for $N$ total episodes, where each episode conditions on all previous episodes and their reflection prompts:

a2∼pθ(a2∣a0,a1),…a_2 \sim p_\theta(a_2 \mid a_0, a_1), \ldots

The complete sequence of $N$ episodes forms what the paper calls a meta-episode:

y=(a0,a1,…,aN)y = (a_0, a_1, \ldots, a_N)

where $y$ is the full meta-episode, $a_n$ is the n-th inner episode, and $N$ is the total number of episodes per meta-episode.

What this represents: a single training sample that consists of $N$ complete search trajectories, all answering the same question, generated sequentially with each later trajectory having full access to all earlier trajectories and their explicit self-reflections.

Why this structure (the key design rationale): this is where the paper's "meta-RL" framing becomes concrete. In standard meta-RL, an agent encounters multiple episodes of a task and must learn to use information from early episodes to perform better in later episodes. Here, the "task" is answering a specific question, and the episodes are different attempts at that same question. The meta-learning objective is: learn to generate effective self-reflections and use them to guide subsequent search attempts, such that later episodes are more likely to be correct than earlier ones.

The critical departure from standard meta-RL is that there is no environment-provided reward signal during inference — the model never sees whether episode 0 was correct or incorrect before generating episode 1. Instead, the model must learn to simulate its own feedback through the reflection text it generates between episodes. The reflection text serves as the bridge between episodes, carrying forward the model's own assessment of what went wrong and what additional information is needed.

During training, the reward for each episode (whether its final answer matches ground truth) is known, but only for gradient computation — not as input to the model's context. This means the model must learn, through RL training, to produce reflections that genuinely improve subsequent answers without ever being explicitly told "your first answer was wrong; here's why."

The paper notes a practical scalability concern: because each episode conditions on all previous episodes, the context length grows linearly with the number of reflection steps $N$. The paper acknowledges this and suggests two mitigation strategies: retaining only the immediately preceding episode as context, or summarizing prior episodes before carrying them forward. The paper empirically validates the short-context variant in Section 4.4, finding it "works well" for MR-Search.

Key implementation detail — the generation prompt: The initial episode uses a generation prompt (Appendix A.1.3) that instructs the model to conduct reasoning inside <thinking> and <response> tags, call the search engine via <search> tags, and provide answers in <answer> tags. The reflection prompt uses the same interaction protocol but adds the meta-instruction to reflect and improve. Both prompts use the same base format, ensuring the model's interface with tools is consistent across all episodes.

The Meta-Level Training Objective

Given this multi-episode structure, the paper defines a meta-level objective that sums the expected reward across all episodes within a meta-episode, with a discount factor controlling how much weight later episodes receive relative to earlier ones:

Jmeta(πθ)=Ey∼πθ[∑n=0N−1γnR(sn,an)]=Ey∼πθ[∑n=0N−1γnfverifier(on,o∗)]J_{\text{meta}}(\pi_\theta) = \mathbb{E}_{y \sim \pi_\theta}\left[\sum_{n=0}^{N-1} \gamma^n R(s_n, a_n)\right] = \mathbb{E}_{y \sim \pi_\theta}\left[\sum_{n=0}^{N-1} \gamma^n f_{\text{verifier}}(o_n, o^*)\right]

where $J_{\text{meta}}(\pi_\theta)$ is the meta-level expected return, $y$ is a complete meta-episode sampled from the policy, $N$ is the number of episodes per meta-episode, $\gamma \in (0, 1]$ is the discount factor (set to 1 by default), $s_n = a_{<n}$ is the accumulated context up to episode $n$ (the "meta-state" consisting of all previous episodes), $R(s_n, a_n)$ is the reward for episode $n$, $o_n$ is the answer extracted from the n-th episode, $o^*$ is the ground-truth answer, and $f_{\text{verifier}}$ is the rule-based verifier.

What it computes: the expected sum of discounted verifier scores across all episodes in a meta-episode. Since the discount factor is set to 1 by default, this is simply the sum of correctness scores across episodes — the model is rewarded equally for correct answers in early episodes and late episodes. With $\gamma < 1$, later episodes would be discounted relative to earlier ones, but the paper keeps $\gamma = 1$ for its main experiments.

Why this form: this objective formally captures the meta-learning intuition. The model is trained to maximize the total correctness across all its attempts, not just the final one. This means even if the model gets a question right on the first attempt (episode 0), it is still incentivized to produce high-quality reflections and confirmatory searches in subsequent episodes rather than degrading to incorrect answers. Conversely, if the model gets a question wrong on the first attempt, it is incentivized to identify the error through reflection and correct it in a subsequent episode.

What makes this different from independent episodes: if each episode were treated independently (as in standard RL), the objective would be $\sum_n \mathbb{E}_{a_n} [f_{\text{verifier}}(o_n, o^*)]$ — the sum of expected rewards across independent trajectories. This would provide no gradient signal relating episode 0 to episode 1, because the episodes would be generated independently rather than conditionally. By placing the sum inside the expectation over $y \sim \pi_\theta$, the meta-objective makes the generation of later episodes explicitly dependent on earlier episodes, and the gradient flows through this dependency. When the model generates a helpful reflection in episode 0 that leads to a correct answer in episode 1, the gradient reinforces both the helpful reflection and the effective use of that reflection.

A subtle but important detail: the meta-state $s_n = a_{<n}$ is not a latent vector (as in RNN-based meta-RL) but is instead the literal text of all previous episodes concatenated together, processed through the transformer's self-attention. This is what makes the approach "in-context meta-RL" — the model uses its own in-context learning mechanism to encode and utilize information from prior episodes, rather than maintaining a separate recurrent state.

Turn-Level Advantage Estimation and Policy Optimization

The previous sections established the meta-episode structure and objective. This section describes the actual optimization algorithm — how the meta-objective is turned into gradient updates for the policy parameters.

The paper deliberately avoids using a separate value function (critic) to estimate advantages, as PPO does. The justification is both practical (reducing computational overhead) and principled (avoiding the complexity of training a value function over variable-length meta-episodes with text observations). Instead, the paper uses a grouped relative advantage estimation based on the REINFORCE Leave-One-Out (RLOO) method.

The key insight is that by sampling multiple meta-episodes for the same question and comparing episodes at the same position across meta-episodes, the system can estimate how much better or worse a particular episode is relative to alternative approaches to the same question at the same point in the search process.

Step 1: Sampling a group of meta-episodes. For each training question, the system samples $G$ independent meta-episodes, where $G$ is the group size (set to 5 in the experiments). Each meta-episode contains $N$ episodes (3 in the main experiments). This yields a matrix of $G \times N$ episodes — $G$ different approaches to the question, each producing $N$ sequential answer attempts. Let $\mathcal{G} = \{y_i\}_{i=1}^G$ denote this group, where $y_i$ is the i-th meta-episode.

Step 2: Computing per-episode relative rewards. For each episode position $n$ (0 through $N-1$), the system computes a baseline reward by averaging the verifier scores across all meta-episodes except the current one, then subtracts this baseline from the current episode's reward:

r~i,n=r(si,n,ai,n)−meanj≠i r(sj,n,aj,n)=r(si,n,ai,n)−1G−1∑j≠ir(sj,n,aj,n)\tilde{r}_{i,n} = r(s_{i,n}, a_{i,n}) - \text{mean}_{j \neq i} \, r(s_{j,n}, a_{j,n}) = r(s_{i,n}, a_{i,n}) - \frac{1}{G-1} \sum_{j \neq i} r(s_{j,n}, a_{j,n})

where $\tilde{r}_{i,n}$ is the relative (baselined) reward for the n-th episode in the i-th meta-episode, $r(s_{i,n}, a_{i,n})$ is the raw verifier score for that episode, and $\frac{1}{G-1} \sum_{j \neq i} r(s_{j,n}, a_{j,n})$ is the leave-one-out baseline — the average reward of all other meta-episodes at the same episode position.

What it computes: a centered reward that indicates whether a particular episode performed better than average at its position in the sequence. If the i-th meta-episode's n-th episode is correct while the other $G-1$ meta-episodes' n-th episodes are incorrect, then $\tilde{r}_{i,n}$ will be positive (reward above baseline). If it is incorrect while others are correct, $\tilde{r}_{i,n}$ will be negative. If all are correct or all are incorrect, $\tilde{r}_{i,n}$ will be near zero.

Why this form: the RLOO baseline serves the same purpose as the learned value function in actor-critic methods — it reduces the variance of the gradient estimate by subtracting a baseline that is independent of the current action. The key property that makes RLOO work is that the baseline $\text{mean}_{j \neq i} \, r(s_{j,n}, a_{j,n})$ does not depend on the i-th meta-episode's actions (because it excludes $i$), which guarantees the estimate is unbiased — the expected value of $\tilde{r}_{i,n}$ equals the true advantage $A(s_{i,n}, a_{i,n})$ under the standard REINFORCE assumptions. The paper explicitly cites this property (Bereket & Leskovec, 2025) as an advantage over GRPO, which uses a mean-including baseline that introduces bias.

Step 3: Propagating credit backward with discounted cumulative advantages. The relative reward $\tilde{r}_{i,n}$ captures the immediate quality of episode $n$, but in a sequential process, early episodes should also receive credit (or blame) for their impact on later episodes. A helpful reflection in episode 0 that leads to a correct answer in episode 2 should be reinforced even if episode 0's own answer was incorrect. To capture these long-horizon dependencies, the paper computes a discounted cumulative advantage:

Ai,n=∑n′=nN−1γn′−n r~i,n′A_{i,n} = \sum_{n'=n}^{N-1} \gamma^{n'-n} \, \tilde{r}_{i,n'}

where $A_{i,n}$ is the cumulative advantage for the n-th episode in the i-th meta-episode, $\gamma$ is the discount factor (set to 1 by default, but the formula supports values in (0,1]), $\tilde{r}_{i,n'}$ is the relative reward at episode $n'$, and the sum runs from the current episode $n$ to the final episode $N-1$.

What it computes: for each episode, the sum of all future relative rewards, discounted by how far in the future they occur. With $\gamma = 1$ (the default), this is simply the sum of relative rewards from episode $n$ through episode $N-1$. For the final episode ($n = N-1$), this is just $\tilde{r}_{i,N-1}$. For the first episode ($n = 0$), this is the sum of all relative rewards.

Why this form (and what it achieves architecturally): this is a Monte Carlo return — it estimates the total future advantage attributable to decisions made at episode $n$. The critical consequence is that an episode that produces an incorrect answer but contains a reflection that enables a correct answer in a later episode will receive a positive advantage signal. Without this propagation (i.e., using $\tilde{r}_{i,n}$ directly as the advantage), the model would be penalized for incorrect answers regardless of whether those answers led to productive reflections that ultimately succeeded. The ablation in Table 2 confirms this empirically: removing the discount factor (setting $\gamma = 0$, which makes $A_{i,n} = \tilde{r}_{i,n}$) causes "substantial degradation" in performance and "convergence to poor local optima." The paper's interpretation is that "an incorrect episode does not necessarily imply that intermediate episodes are uninformative."

Step 4: Policy optimization with clipped surrogate objective. With these episode-level advantages computed, the policy is updated using a clipped surrogate objective similar to PPO, but applied at the token level with episode-level advantage broadcasting. The formal objective is:

1G∑i=1G1∣yi∣∑n=1∣yi∣min⁡(π(yi,n∣x,yi,<n;θ)π(yi,n∣x,yi,<n;θold)Ai,n,  clip(π(yi,n∣x,yi,<n;θ)π(yi,n∣x,yi,<n;θold),1−ϵ,1+ϵ)Ai,n)\frac{1}{G} \sum_{i=1}^{G} \frac{1}{|y_i|} \sum_{n=1}^{|y_i|} \min\left( \frac{\pi(y_{i,n} \mid x, y_{i,<n}; \theta)}{\pi(y_{i,n} \mid x, y_{i,<n}; \theta_{\text{old}})} A_{i,n}, \; \text{clip}\left( \frac{\pi(y_{i,n} \mid x, y_{i,<n}; \theta)}{\pi(y_{i,n} \mid x, y_{i,<n}; \theta_{\text{old}})}, 1-\epsilon, 1+\epsilon \right) A_{i,n} \right)

where $G$ is the group size (number of meta-episodes), $|y_i|$ is the number of tokens in the i-th meta-episode, $\pi(y_{i,n} \mid x, y_{i,<n}; \theta)$ is the probability of the n-th token under the current policy, $\pi(y_{i,n} \mid x, y_{i,<n}; \theta_{\text{old}})$ is the probability under the old (pre-update) policy, $A_{i,n}$ is the cumulative advantage for the episode containing token $n$ (broadcast to all tokens in that episode), and $\epsilon$ is the clipping threshold.

What it computes: for each token in each meta-episode, the objective compares the ratio of current to old policy probability for that token against the advantage signal. If the ratio moves too far from 1 (beyond $1-\epsilon$ or $1+\epsilon$), it is clipped to prevent overly large policy updates. The clipped surrogate is then multiplied by the advantage and averaged across all tokens and meta-episodes.

Why this form: the clipping mechanism is standard in PPO — it prevents the policy from changing too much in a single update, which improves training stability. The key design choice specific to MR-Search is broadcasting the episode-level advantage to all tokens in that episode. This means every token in the n-th episode receives the same advantage signal $A_{i,n}$, regardless of whether that token is a reasoning step, a search query, or a retrieved document. The paper notes (Section 3.3) that tool output tokens are masked out from the loss, following Search-R1 (Jin et al., 2025a) — this means the model does not receive gradients for the tokens that represent retrieved Wikipedia text, since those were not generated by the policy but rather injected from the environment.

What distinguishes this from PPO: PPO requires a separate value function (critic network) that estimates $V(s)$ and computes advantages as $A = r + \gamma V(s') - V(s)$. MR-Search avoids the critic entirely by using the grouped RLOO baseline, making it a critic-free algorithm. This eliminates the need to train and store a separate network, reducing memory and computational overhead.

What distinguishes this from GRPO: GRPO (used by Search-R1) groups trajectories and computes relative rewards, but does so at the level of single episodes (each meta-episode is one episode) and does not propagate credit across turns within a meta-episode. MR-Search's key extension is the multi-turn discounted advantage that connects episodes within a meta-episode.

What distinguishes this from MT-GRPO: MT-GRPO (Zeng et al., 2025) also operates on multi-turn trajectories, but uses a different advantage formulation. The paper's ablation (Table 2) shows that MR-Search's RLOO-based estimator outperforms MT-GRPO when both are trained with episode turn-level rewards: MR-Search achieves 46.0% average vs. 44.3% for MT-GRPO on Qwen2.5-7B. The paper attributes this to the unbiasedness of the RLOO estimator compared to GRPO-style estimates.

Exploration-Exploitation Masking

The paper introduces an optional extension that encourages the policy to prioritize long-term gains from improved context adaptation over short-term episode feedback. The mechanism is a reward mask applied during advantage computation.

The insight is that in the standard formulation, all episodes contribute equally to the advantage signal according to Eq. (8). This means the model receives gradient updates for the correctness of every episode, including early episodes that may serve primarily as information-gathering steps. The paper hypothesizes that for some tasks, it may be beneficial to designate certain episodes as "exploration only" — they contribute context for subsequent episodes but do not directly receive gradient updates from their own rewards.

The modified advantage computation is:

Ai,n=∑n′=nN−1γn′−n r~i,n′ mn′A_{i,n} = \sum_{n'=n}^{N-1} \gamma^{n'-n} \, \tilde{r}_{i,n'} \, m_{n'}

where $m_{n'} \in \{0, 1\}$ is a binary mask indicating whether episode $n'$ is an exploitation episode (1) or exploration episode (0).

What it computes: the same discounted cumulative advantage as before, but with the relative rewards for exploration episodes zeroed out. This means exploration episodes contribute zero to the advantage signal regardless of whether their answers were correct or incorrect. The gradient for these episodes flows only indirectly — through their influence on the context that subsequent exploitation episodes use.

Why this form: by zeroing out exploration rewards, the policy gradient is driven entirely by the performance of exploitation episodes. This encourages the policy to use early episodes productively to gather information and adapt its strategy, rather than trying to maximize correctness in those early episodes directly. The paper draws a connection to prior work on decoupling exploration and exploitation in meta-RL (Stadie et al., 2018; Liu et al., 2021).

When this is useful: The paper reports in Section 4.4 that this strategy is "helpful for ASearcher, which requires multi-turn search." For simpler benchmarks (NQ, HotpotQA), the standard formulation (all episodes unmasked) performs better. The intuition is that ASearcher requires longer interaction chains with more tool calls, making the exploration-exploitation distinction more meaningful — early episodes need to explore the information space broadly before committing to an answer strategy, and penalizing them for not producing correct answers prematurely could discourage this exploration.

Implementation detail: the paper designates "the first two episodes as exploration and the last two as exploitation" in their experiments. The specific split (number of exploration vs. exploitation episodes) is treated as a hyperparameter that could be tuned per task.

Step-Level Meta-RL Extension

The paper notes that the core principle of MR-Search — treating semantically meaningful sub-units as episodes within a meta-RL framework — extends naturally to finer granularities than full answer trajectories. Specifically, within a single search episode, each tool-interaction step can be treated as a micro-episode.

The motivation is that even within a single episode, the agent makes multiple decisions (what to search for, whether to continue searching, which retrieved information to trust) and may accumulate enough information to form a partial answer before the episode concludes. By prompting the model to produce an intermediate answer after each tool call, the system can obtain per-step reward signals that provide denser supervision.

Architecture: The paper implements this by prompting the model to produce an intermediate answer after each tool call (illustrated in Figure 8 in the Appendix). Each tool-interaction step — reasoning, search query, retrieved documents, intermediate answer — becomes a micro-episode. The verifier scores each intermediate answer against the ground truth, providing step-level rewards. These step-level rewards are then used in the same RLOO-based advantage formulation, with the discount factor propagating credit across steps within a single search trajectory.

Why this works: this transforms long trajectories into sequences of reflection-like steps, where the model receives feedback on whether it is on track at each intermediate point. This provides "localized credit assignment" that helps the model learn which search queries are productive and which are not, without requiring external process reward models. The paper reports in Table 3 that the step-level variant achieves substantial improvement over Search-R1 (the outcome-reward-only baseline), though slightly underperforms the full episode-level MR-Search on most benchmarks. For example, on Qwen2.5-7B, the step-level variant achieves 48.6% on NQ vs. 50.2% for full MR-Search, but still substantially above Search-R1's 45.9%.

Summary of Design Choices and Their Justifications

  • Meta-episode structure (N episodes, chained by reflection) over independent episodes: transforms the learning problem from optimizing individual trajectories to optimizing adaptive strategies across attempts, enabling the model to learn from its own experience in-context.
  • RLOO-based grouped advantage over PPO with a critic or GRPO with per-episode baselines: provides unbiased advantage estimates without the computational overhead of a value network, and the grouping across meta-episodes makes advantage estimates comparable at each episode position.
  • Discounted cumulative advantage over immediate relative rewards: propagates credit across episodes so that early exploratory episodes that enable later success are reinforced even if their own answers are incorrect, addressing the core credit assignment problem.
  • Episode-level advantage broadcasting over token-level advantage estimation: simplifies the optimization while maintaining effectiveness, since the key decisions (whether to search, what to search for, when to stop) are best evaluated at the level of whole episodes rather than individual tokens.
  • Tool output masking in the loss: prevents the model from being penalized or rewarded for tokens it did not generate (retrieved documents), which would introduce noise and bias into the gradient.
  • Self-reflection as textual context over latent recurrent state: leverages the LLM's in-context learning capability and makes the reflections interpretable, debuggable, and potentially reusable across tasks — a practical advantage over black-box recurrent state.
  • Critic-free optimization over actor-critic methods: reduces computational overhead (no separate value network to train and store), simplifies the training pipeline, and avoids the instability that can arise from value function approximation errors in complex text-based environments.
  • Constant discount factor (γ=1\gamma = 1) over geometric discounting: treats all episodes within a meta-episode as equally important, which makes sense because the meta-episode has a fixed finite length and all episodes contribute to the total reward objective.

4. Key Insights and Innovations

Innovation 1: Reframing Agentic Search as Meta-Reinforcement Learning Across Self-Reflection Episodes

The paper's most fundamental contribution is not the self-reflection mechanism itself—prior work already explored prompting-based self-reflection (Shinn et al., 2023; Madaan et al., 2023) and even fine-tuning for self-correction (Kumar et al., 2024; Qu et al., 2024). Rather, the conceptual breakthrough is recasting multi-attempt search as a meta-RL problem where the agent meta-learns how to generate effective self-reflections, transforming what was previously a brittle prompting hack into a principled training objective.

Prior to this work, the dominant assumption—implicit in Search-R1, ReSearch, and related RL-based search agent training—was that each search trajectory is an independent episode. The agent produces one answer, receives a reward, and the policy is updated. If you wanted multiple attempts, you would sample multiple independent trajectories (parallel sampling) and pick the best via majority voting or a verifier. The meta-RL framing inverts this: the agent is trained to treat a sequence of attempts as a single adaptive process, where early episodes serve as information-gathering steps whose value lies not in their own correctness but in how they improve later attempts.

This is a fundamental shift, not an incremental refinement, for three reasons. First, it changes what the policy is optimizing: not the expected correctness of a single trajectory, but the expected improvement trajectory across episodes. A model that consistently gets the answer wrong on the first try but correct on the third try would score poorly under independent-episode RL (two wrong, one right) but well under MR-Search's meta-objective (the sequence shows effective adaptation). Second, it changes what information flows between attempts: rather than treating episodes as isolated, the policy now generates explicit textual bridges (reflections) that carry forward its own diagnosis of what went wrong. Third, it changes the credit assignment structure: the discounted cumulative advantage (Eq. 8) means early episodes receive credit for enabling later success, even when their own answers are incorrect.

The significance extends beyond performance gains. This reframing provides a unified explanation for why prompting-based self-reflection often fails on complex reasoning (Huang et al., 2023): the model has never been trained to generate useful reflections for itself, only to produce reflections that sound plausible to a human reader. MR-Search's meta-training objective closes this gap by making the downstream utility of the reflection—did it lead to a better answer?—the explicit optimization target. The reflections are no longer evaluated on surface plausibility but on their causal contribution to improved search outcomes.

Evidence for this framing's importance comes from the training dynamics (Figures 4, 6, 7): MR-Search exhibits more stable convergence and higher training reward than Search-R1, and critically, the model trained with MR-Search shows a steep improvement curve when given additional reflection turns at test time (Figures 3 and 5), while Search-R1 with the same reflection mechanism shows marginal gains. This suggests Search-R1 learned a policy optimized for single-shot correctness, while MR-Search learned a policy optimized for iterative improvement—qualitatively different learned behaviors emerging from the meta-RL objective.

Innovation 2: Self-Generated Cross-Episode Feedback as a Substitute for External Process Rewards

The paper makes a diagnostic contribution that challenges a growing consensus in the agentic search literature: that dense, step-level process rewards—whether from human annotations, learned reward models, or LM judges—are necessary to overcome the credit assignment problem in multi-turn search (Deng et al., 2025; Wang et al., 2025b; Anonymous, 2026). MR-Search demonstrates that structured comparison of multiple complete answer attempts, coupled with backward credit propagation, can provide comparably effective learning signals without any external process reward model.

This is not an obvious result. The standard argument for process rewards is that multi-turn search trajectories are long (5+ tool calls), the outcome signal is binary and delayed, and without intermediate feedback the agent cannot distinguish productive search queries from unproductive ones. MR-Search does not dispute this diagnosis—it agrees that credit assignment is the bottleneck. Rather, it proposes a fundamentally different solution: instead of annotating intermediate steps, generate multiple complete trajectories for the same question and compare them at the same episode position. The grouped RLOO advantage (Eq. 7) answers the question "was this particular search strategy better than alternative strategies at this stage of the process?" not by evaluating the strategy against a learned reward model, but by comparing outcomes across parallel meta-episodes.

This matters for three practical reasons that go beyond raw accuracy. First, it eliminates the distribution shift problem that plagues learned reward models: a process reward model trained on one model's outputs may not accurately score another model's outputs, requiring retraining whenever the policy changes. MR-Search's reward signal comes directly from the verifier applied to the policy's own outputs, so there is no distribution shift. Second, it eliminates reward hacking: the agent cannot learn to exploit imperfections in a learned reward model because there is no learned reward model—only the ground-truth verifier on final answers, which is by definition correct. Third, it eliminates computational overhead: no separate reward model to train, store, and run during RL updates.

The evidence for this substitution claim is in Table 1. MR-Search achieves 46.0% average accuracy on Qwen2.5-7B, compared to 43.4% for StepResearch and 39.3% for PPRM—both of which use external process reward models. On the smaller Qwen2.5-3B, the gap is even larger: 41.4% for MR-Search vs. 38.0% for StepResearch. This suggests that self-generated cross-episode feedback is not merely competitive with external process rewards but can be more effective, likely because it avoids the reward model's inevitable approximation errors.

The negative result—that the step-level meta-RL variant (Table 3) underperforms full episode-level MR-Search on most benchmarks—is also informative. It suggests that the granularity of credit assignment matters: comparing entire episodes (which bundle together search strategy, reasoning quality, and answer extraction) may provide a more reliable signal than comparing individual tool-interaction steps, where the connection between any single step and final correctness is noisier.

Innovation 3: Temporal Credit Propagation as the Enabling Mechanism for Multi-Turn Self-Improvement

The paper identifies and empirically isolates a specific mechanism—discounted cumulative advantage propagation across reflection turns—as the critical factor that enables effective multi-turn self-improvement training. This is not a novel algorithm in the RL literature (Monte Carlo returns are standard), but its application to self-reflection in LLM agents reveals a previously unappreciated necessity: without backward credit flow, multi-turn self-reflection training collapses.

The ablation in Table 2 provides the decisive evidence. Setting the discount factor γ = 0—which makes the advantage for each episode depend only on that episode's own reward, with no propagation from future episodes—causes "substantial degradation" in performance: 43.8% average vs. 46.0% for full MR-Search. The paper's interpretation is revealing: "an incorrect episode does not necessarily imply that intermediate episodes are uninformative." This is the core diagnostic insight: in a self-reflection chain, the value of an early episode lies primarily in the quality of the reflection it generates, not in the correctness of its own answer.

This finding has implications that extend beyond MR-Search. It explains a subtle failure mode in prior work on training LLMs to self-correct: if the training objective treats each correction attempt as an independent episode (each receiving reward based on its own answer), the model is penalized for producing an incorrect initial answer even when that incorrect answer leads to a productive reflection and eventual success. This creates a conflicting gradient signal—the model is simultaneously encouraged to produce correct answers (short-term reward) and to learn to generate useful reflections (long-term benefit), but the long-term benefit is not credited to the actions that enabled it.

MR-Search's solution—propagating credit backward through the discounted cumulative advantage—is conceptually straightforward but architecturally non-obvious. It requires structuring the training loop so that episodes are generated sequentially (not independently), that rewards at all positions are known before any gradient is computed, and that the advantage estimator can look forward to determine the total future value of each episode. This is not possible in standard online RL setups where episodes are processed as they are generated; it requires the batched, grouped architecture that MR-Search introduces.

The comparison with MT-GRPO (Zeng et al., 2025) in Table 2 reinforces this point. MT-GRPO also operates on multi-turn trajectories with turn-level rewards, but uses a different advantage formulation. MR-Search outperforms it (46.0% vs. 44.3% average), suggesting that the specific mechanism of unbiased RLOO estimation combined with full backward discounting is more effective than GRPO-style multi-turn credit assignment. The paper attributes this to the unbiasedness property of the leave-one-out baseline.

Innovation 4: In-Context Meta-Learning as a Bridge Between Prompting-Based and Training-Based Self-Improvement

The paper synthesizes two previously disconnected research threads—prompting-based self-reflection (which works at inference time but is unreliable for complex reasoning) and training-based self-correction (which can be reliable but requires expensive fine-tuning and often process reward models)—into a unified framework where the model is trained to perform effective self-reflection at inference time using only its own in-context learning mechanism.

This is a conceptual bridge that resolves a tension in the literature. Prompting-based methods (Reflexion, Self-Refine) are appealing because they require no training and work with any model, but they fail when the model's self-critique ability is insufficient to identify its own errors (Huang et al., 2023). Training-based methods (SCoRe, Qu et al., 2024) can teach models to self-correct reliably, but they typically require carefully constructed training data with paired incorrect-to-correct trajectories, and they haven't been demonstrated in multi-turn tool-use settings.

MR-Search's insight is that these approaches are not alternatives but complements that can be unified through meta-RL. The model is trained (via RL) on the meta-skill of generating useful reflections, but at inference time, it applies this skill purely through in-context learning—conditioning on its own previous trajectories and reflection text, without any external feedback or separate evaluation model. The training objective teaches the model what makes a reflection useful (does it lead to better subsequent answers?), but the mechanism for using that reflection at test time is exactly the same in-context conditioning that prompting-based methods rely on.

This matters because it suggests a general principle for building self-improving LLM agents: train the meta-cognitive skill through outcome-driven RL, deploy through in-context adaptation. The heavy lifting is done during training, where ground-truth outcomes are available and can provide reliable learning signals; at inference, the model leverages its learned ability to generate diagnostic reflections and adapt its strategy accordingly, without needing any oracle feedback.

The test-time scaling results (Figures 3 and 5) provide compelling evidence for this synthesis. When MR-Search is allowed additional reflection turns beyond what it was trained with (extrapolating from 3 to more turns at test time), performance continues to improve steeply. In contrast, Search-R1 with the same reflection mechanism shows only marginal gains—it was never trained to use reflections effectively, so adding more reflection turns doesn't help. This demonstrates that MR-Search has genuinely learned a transferable self-reflection skill, not merely memorized a fixed number of refinement steps.

The exploration-exploitation masking extension (Section 3.4) further reinforces this bridge concept. By designating early episodes as "exploration only" (contributing context but receiving no direct gradient from their own rewards), the training procedure mirrors how prompting-based reflection is used at inference time—early attempts are information-gathering steps whose value is realized in later attempts. The difference is that MR-Search learns what information to gather and how to reflect on it, rather than relying on a fixed prompt template to elicit useful reflection behavior.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper uses eight benchmarks spanning two categories. For single-hop QA: NQ (Kwiatkowski et al., 2019; 3,610 test samples), TriviaQA (Joshi et al., 2017; 11,313 test samples), and PopQA (Mallen et al., 2022; 14,267 test samples). For multi-hop QA: HotpotQA (Yang et al., 2018; 7,405 test samples), 2WikiMultiHopQA (Ho et al., 2020; 7,405 test samples), Musique (Trivedi et al., 2022; 2,417 test samples), and Bamboogle (Press et al., 2022; 125 test samples). An additional synthetic dataset, ASearcher (Gao et al., 2025), requiring longer-horizon multi-turn search, is split into 90% training and 10% evaluation sets by the authors. For training all fine-tuning approaches, the NQ and HotpotQA training sets are merged into a unified dataset, following the setup of Jin et al. (2025b).

  • Base model. Experiments use Qwen-series models (Yang et al., 2024), specifically Qwen2.5-3B-Base and Qwen2.5-7B-Base. These are chosen as representative base models at a scale where RL training for agentic search is computationally feasible yet challenging—the 3B model in particular struggles with multi-turn search behavior under naive RL, providing a strong test of whether the proposed training approach can elicit capabilities that outcome-reward-only methods fail to unlock. All models use the 2018 Wikipedia dump (Karpukhin et al., 2020) as the knowledge source with E5 embeddings (Wang et al., 2022a) as the retriever, with the number of retrieved documents fixed to three across all methods for fair comparison.

  • Metrics. The primary metric is Exact Match (EM) accuracy after answer normalization. EM evaluates to true if and only if the predicted answer exactly matches any ground-truth answer after normalization. For all methods, a single trajectory per question is sampled and the EM of the last valid prediction is reported, following Jin et al. (2025a). No majority voting or best-of-N selection is applied in the main results—each question gets one trajectory, and the accuracy is the fraction of questions where that single trajectory's final answer is correct. For the ASearcher dataset, both EM and F1 are reported (Figure 4).

  • Baselines. The paper compares against six baselines organized into three categories. Inference without fine-tuning: (1) Direct inference—the base model prompted to answer without retrieval; (2) Search-o1 (Li et al., 2025b)—a search-enhanced reasoning framework that integrates agentic RAG with a reason-in-document module, used with retrieval at inference time. Fine-tuning with outcome rewards only: (3) ReSearch (Chen et al., 2025)—RL-based framework training LLMs to interleave reasoning with explicit search actions using outcome rewards; (4) Search-R1 (Jin et al., 2025a)—extends RL-based reasoning by enabling LLMs to autonomously generate search queries during multi-turn reasoning, trained with GRPO and outcome rewards. Fine-tuning with external process rewards: (5) PPRM (Anonymous, 2026)—a principle process reward model providing step-wise signals to guide GRPO-based RL; (6) StepResearch (Wang et al., 2025b)—trains search agents with step-wise PPO using intermediate rewards and token-level supervision from an external verifier. For the sequential reflection and parallel sampling comparisons in Section 4.3, the paper also compares against Search-R1 extended with the same multi-turn reflection mechanism at inference time (Search-R1-S for sequential, Search-R1-P for parallel).

  • Generation budget / compute accounting. All methods are evaluated at a single trajectory per question—there is no best-of-N sampling or budget sweep in the main results. The key resource constraint is the number of tool calls per episode, set to a maximum of 3 for NQ/HotpotQA and 5 for ASearcher. The number of reflection episodes (N) is fixed at 3 during training. For the test-time scaling analysis (Figures 3 and 5), additional reflection turns are extrapolated beyond training by appending the entire interaction history to the context at each turn, with no limit explicitly stated beyond what is shown in the figures. Training uses a group size of G = 5 for advantage calculation, a learning rate of 1e-6 with AdamW, and 300 total training steps. Context length is set to 8K tokens for NQ/HotpotQA and 16K for ASearcher.

  • Cross-validation / statistical protocol. The paper does not employ cross-validation for strategy selection (unlike the example paper which used two-fold CV for compute-optimal policy selection). Results in Table 1 are single-run evaluations. For the test-time scaling analysis (Figures 3 and 5), shaded regions show standard deviation across 3 runs, providing some measure of statistical reliability for the scaling curves. The training dynamics plots (Figures 4, 6, 7) show training step-level curves but no confidence intervals.

Main Quantitative Results

Headline Comparison Against Baselines (Table 1)

The central quantitative result appears in Table 1, which reports EM accuracy across all seven benchmarks (excluding ASearcher) for both model sizes. On Qwen2.5-7B-Base, MR-Search achieves an average accuracy of 46.0% across the seven benchmarks, compared to 42.1% for Search-R1 (the strongest outcome-reward-only baseline), 43.4% for StepResearch (the strongest external process-reward baseline), and 38.1% for ReSearch. This represents a relative improvement of 9.2% over Search-R1 and 6.0% over StepResearch.

On Qwen2.5-3B-Base, the relative advantage is larger: MR-Search achieves 41.4% average vs. 34.7% for Search-R1—a 19.3% relative improvement. The gap between MR-Search and the process-reward baselines is also more pronounced on the smaller model: 41.4% vs. 35.7% for PPRM and 38.0% for StepResearch. This pattern—MR-Search providing larger relative gains on the smaller model—is consistent with the paper's motivation: smaller models struggle more with sparse outcome rewards because they have less capacity to discover effective search strategies through undirected exploration.

The per-benchmark breakdown reveals where MR-Search's gains are concentrated. On single-hop QA (NQ, TriviaQA, PopQA), the improvements are modest but consistent: MR-Search achieves 50.2% vs. 45.9% for Search-R1 on NQ (+4.3 points), 66.6% vs. 63.2% on TriviaQA (+3.4 points), and 47.2% vs. 44.9% on PopQA (+2.3 points). These are tasks where a single well-targeted search query often suffices, so the benefit of multi-episode reflection is incremental.

The largest gains appear on multi-hop QA benchmarks, where information must be synthesized across multiple searches. On HotpotQA, MR-Search achieves 46.8% vs. Search-R1's 43.9% (+2.9 points) and StepResearch's 43.9%. On 2WikiMultiHopQA: 43.6% vs. 38.7% (+4.9 points). On Musique: 22.1% vs. 18.1% (+4.0 points). On Bamboogle—the most difficult multi-hop benchmark with only 125 test questions—MR-Search achieves 45.2% vs. 40.0% for Search-R1 (+5.2 points) and 43.5% for StepResearch.

Notably, MR-Search also outperforms Direct Inference (the base model without retrieval) by enormous margins—46.0% vs. 18.1% average on Qwen2.5-7B—confirming that the search tool is essential and that MR-Search's training enables effective tool use.

ASearcher Results (Figure 4)

On the more complex ASearcher dataset, which requires longer-horizon multi-turn search, MR-Search achieves 41.3% EM vs. 36.9% for Search-R1—a 10.2% relative improvement. The F1 metric shows a similar pattern (Figure 4), though the exact F1 values are only readable from the plot and not reported numerically in the text. The training dynamics (Figure 4, center panel) show MR-Search maintaining consistently higher training accuracy than Search-R1 throughout the 300 training steps, with MR-Search converging around 0.565 training accuracy vs. approximately 0.513 for Search-R1. The tool call frequency (Figure 4, right panel) shows MR-Search calling the search engine more frequently (averaging approximately 3-4 calls vs. 2-3 for Search-R1 toward the end of training), suggesting MR-Search learns to conduct more thorough searches rather than settling for the first retrieved information.

Training Dynamics (Figures 4, 6, 7)

The training curves reveal several patterns supporting the effectiveness of MR-Search's credit assignment. On Qwen2.5-7B-Base (Figure 4, and more extensively in Appendix Figures 6 and 7), MR-Search exhibits more stable convergence than Search-R1. The training accuracy curve for MR-Search rises more smoothly and reaches a higher plateau (approximately 0.565 vs. 0.513 on ASearcher). The test accuracy curve (Figure 4, leftmost panel for ASearcher, and Appendix Figures 6-7 for NQ/HotpotQA) shows MR-Search consistently above Search-R1 throughout training, with the gap widening in later steps—suggesting that the benefits of multi-turn credit assignment compound as training progresses.

The tool call frequency (rightmost panels in Figures 4, 6, 7) shows an interesting dynamic: MR-Search learns to call the search engine more frequently than Search-R1, with the number of tool calls per trajectory increasing over the course of training. On Qwen2.5-7B (Figure 4, ASearcher), MR-Search averages approximately 3.5 tool calls by step 300 vs. roughly 2.5 for Search-R1. This is significant because it demonstrates that MR-Search's training objective does not inadvertently penalize longer search trajectories—the meta-objective rewards thoroughness when it leads to correct answers, and the backward credit propagation ensures that early search queries in a successful trajectory receive positive gradient signal.

Test-Time Scaling (Figures 3 and 5)

Figures 3 and 5 show how performance scales when additional reflection turns are allowed at test time beyond the 3 turns used during training. MR-Search exhibits a steep improvement curve: performance continues to increase as more reflection turns are added, demonstrating effective extrapolation. In contrast, Search-R1 with the same sequential reflection mechanism (Search-R1-S) shows only marginal gains—its curve is nearly flat beyond the first turn, confirming that it did not learn to use reflections productively during training.

The paper also compares against Search-R1 with parallel sampling (Search-R1-P), where multiple independent trajectories are generated and the most frequent answer is selected. MR-Search substantially outperforms both Search-R1 variants, and its advantage grows with more turns. The shaded regions (standard deviation across 3 runs) show that the improvement is consistent across runs, particularly at higher turn counts where the gap is largest. On TriviaQA (Figure 5, second subplot), MR-Search reaches approximately 67% accuracy at 5 turns vs. roughly 63% for Search-R1-P and 62% for Search-R1-S. On Bamboogle (Figure 5, rightmost subplot), MR-Search reaches approximately 48% at 5 turns vs. roughly 40% for both Search-R1 variants.

Ablation of Discount Factor and Training Algorithm (Table 2)

Table 2 presents ablations on Qwen2.5-7B-Base that isolate key design choices. The full MR-Search achieves 46.0% average. Setting γ = 0 (removing future credit propagation, so each episode's advantage depends only on its own reward) reduces average accuracy to 43.8%—a drop of 2.2 percentage points. The degradation is not uniform across benchmarks: on HotpotQA, the drop is from 46.8% to 44.3% (-2.5 points); on 2WikiMultiHopQA, from 43.6% to 41.7% (-1.9 points); on Bamboogle, from 45.2% to 42.6% (-2.6 points). This confirms the paper's central claim about the necessity of backward credit propagation for effective multi-turn training.

Replacing the RLOO-based advantage estimator with PPO (using a separate value function) within the MR-Search framework reduces average accuracy to 42.0%—below even Search-R1 (43.5% in this table). This is a striking negative result: the additional complexity of training a value function not only fails to help but actively hurts performance, likely because value function approximation in the complex, variable-length text observation space is noisy and destabilizes training. PPO underperforms most dramatically on NQ (43.9% vs. 50.2% for MR-Search) and HotpotQA (41.3% vs. 46.8%).

Replacing the RLOO estimator with MT-GRPO (Zeng et al., 2025) achieves 44.3% average—better than PPO and the γ = 0 ablation, but below full MR-Search (46.0%) and slightly above Search-R1 (43.5%). This confirms that the RLOO-based unbiased advantage estimation provides benefits over GRPO-style baselines, consistent with the paper's theoretical argument about unbiasedness.

A subtle but important detail in Table 2: both PPO and MT-GRPO underperform Search-R1 on single-hop NQ (43.9% and 46.1% vs. 46.4% for Search-R1), while MR-Search does not (50.2%). The paper interprets this as evidence of "stronger generalization and robustness" for MR-Search—the multi-turn training with unbiased advantages does not degrade single-hop performance while substantially improving multi-hop performance, whereas other multi-turn formulations trade off single-hop accuracy.

Ablation Studies and Robustness Checks

Exploration-exploitation masking (Table 3): Designating the first two episodes as exploration (zero reward mask) and the last two as exploitation produces mixed results. On the main seven benchmarks, this variant achieves lower average accuracy than full MR-Search—for example, on HotpotQA, it achieves 44.7% vs. 46.8% for MR-Search; on 2WikiMultiHopQA, 39.4% vs. 43.6%. However, on ASearcher—the most complex dataset requiring longer interaction chains—the exploration variant achieves 43.2% vs. 41.3% for standard MR-Search and 36.9% for Search-R1. This is a 6.3 percentage point gain over Search-R1 and nearly 2 points over standard MR-Search. The paper attributes this to the exploration masking encouraging the model to use early episodes for broad information gathering without being penalized for incorrect answers during exploration, which is beneficial when the task requires many tool interactions. This is a conditional finding: exploration masking helps on long-horizon tasks but hurts on shorter-horizon tasks where early episodes can and should produce correct answers.

Step-level meta-RL (Table 3): Treating each tool-interaction step as a micro-episode with intermediate answer evaluation achieves 38.4% on ASearcher and generally underperforms full episode-level MR-Search on most benchmarks. On NQ, it achieves 48.6% vs. 50.2% for full MR-Search; on HotpotQA, 42.3% vs. 46.8%; on Musique, 16.3% vs. 22.1%. However, it still substantially outperforms Search-R1 on most benchmarks (e.g., 48.6% vs. 45.9% on NQ), confirming that the meta-RL principle extends to finer granularities even if the optimal granularity is at the episode level. The performance gap between step-level and episode-level MR-Search suggests that comparing entire search strategies (which bundle together query formulation, result interpretation, and answer extraction) provides a more reliable learning signal than comparing individual tool-interaction steps, where the connection between any single step and final correctness is noisier.

Context management—keeping only one preceding episode (Table 3): The "Short Context" variant, which retains only the immediately preceding episode as context rather than all previous episodes, achieves 40.5% on ASearcher and generally performs slightly below full MR-Search but still well above Search-R1. On Bamboogle, it actually outperforms full MR-Search (47.2% vs. 45.2%), suggesting that for some tasks, longer context may introduce noise or distraction. On most benchmarks, the gap is small—for example, 48.1% vs. 50.2% on NQ, 65.9% vs. 66.6% on TriviaQA—indicating that the method is robust to context truncation and does not critically depend on access to the full interaction history.

Discount factor ablation (Table 2, discussed above): Setting γ = 0 removes backward credit propagation and causes a 2.2 percentage point average drop. The degradation is most pronounced on multi-hop benchmarks (Bamboogle: -2.6 points; HotpotQA: -2.5 points), consistent with the intuition that multi-hop tasks benefit more from sequential refinement where early episodes inform later ones.

Training algorithm comparison (Table 2, discussed above): PPO within the MR-Search framework underperforms Search-R1, demonstrating that a critic-based approach is actively harmful in this setting. MT-GRPO performs better than PPO but below RLOO-based MR-Search, confirming the value of unbiased leave-one-out estimation.

Search-R1 with MR-Search's reflection mechanism (Figures 3 and 5): This is a critical ablation showing that simply adding reflection prompts at inference time to a model not trained for multi-turn reflection provides almost no benefit. Search-R1-S (sequential reflection) and Search-R1-P (parallel sampling) both underperform MR-Search by large margins, and Search-R1-S's curve is nearly flat—adding more reflection turns does not improve performance. This demonstrates that MR-Search's gains come from the training objective, not from the reflection prompting format itself.

Training dynamics across model sizes (Figures 6 and 7): The Appendix provides training dynamics for Qwen2.5-3B (Figure 6) and Qwen2.5-7B (Figure 7) on the NQ/HotpotQA training mixture. Both model sizes show the same qualitative patterns: MR-Search achieves higher training accuracy, more stable convergence, and more frequent tool calls than Search-R1. This replication across model scales strengthens the claim that the meta-RL formulation provides general benefits, not just benefits that appear at a specific model size.

Critical Assessment

Claim 1: MR-Search significantly outperforms outcome-reward-only RL baselines.

The evidence for this claim is robust and multi-faceted. Table 1 shows consistent gains across seven benchmarks and two model sizes, with average relative improvements of 9.2% (Qwen2.5-7B) and 19.3% (Qwen2.5-3B) over Search-R1. The training dynamics (Figures 4, 6, 7) show that these gains emerge during training and are not artifacts of evaluation. The test-time scaling curves (Figures 3 and 5) demonstrate that MR-Search produces qualitatively different learned behavior—the model continues to improve with additional reflection turns, unlike Search-R1 which plateaus immediately.

However, several caveats apply. First, the Search-R1 comparison uses a single trajectory per question with the last valid prediction evaluated—this is the same protocol used for MR-Search, making it fair, but it means we are comparing single-trajectory performance. Search-R1 might benefit more from majority voting or best-of-N than MR-Search does (since MR-Search already gets multiple attempts through its sequential episodes), and this interaction is not explored. Second, the paper does not compare against Search-R1 trained with equivalent total compute—MR-Search generates N episodes per meta-episode during training, which requires N times more forward passes per question than Search-R1. The reported improvements could partially reflect increased effective training data (each question is seen N times per batch rather than once) rather than the meta-RL formulation per se. A compute-matched comparison where Search-R1 gets N times more training questions or N times more training steps would isolate the formulation's contribution from the data amplification effect. Third, the paper uses the same hyperparameter settings for Search-R1 and MR-Search (learning rate, group size, number of steps), but the optimal hyperparameters likely differ between the methods, particularly learning rate and group size.

Claim 2: Self-generated cross-episode feedback can substitute for external process reward models.

Table 1 shows MR-Search achieving 46.0% average vs. 43.4% for StepResearch and 39.3% for PPRM on Qwen2.5-7B. This supports the claim that self-generated feedback is competitive with or superior to external process rewards. However, the comparison is between methods developed by different research groups using potentially different implementations, hyperparameter tuning efforts, and computational budgets. The paper does not re-implement StepResearch or PPRM within its own codebase under controlled conditions—it compares against reported numbers and open-source implementations, which introduces confounds from implementation quality and hyperparameter optimization. A stronger test would be to implement all methods within the same training framework (same base model, retriever, number of training steps, batch size) and compare under identical computational budgets. The fact that MR-Search outperforms these baselines despite the potential confounds is suggestive but not definitive evidence of inherent superiority.

Additionally, the paper's claim about avoiding reward hacking is not directly tested. The paper argues that because MR-Search uses only the ground-truth verifier (not a learned reward model), it is immune to reward hacking. This is logically true but uninteresting—any method that uses only ground-truth verification is immune to reward model exploitation. The relevant comparison is whether MR-Search achieves comparable or better performance than methods that do use learned process rewards, which Table 1 addresses. But there is no experiment showing that the process reward baselines do suffer from reward hacking, or that this explains their performance gap relative to MR-Search. The reward hacking concern is asserted but not empirically demonstrated in this paper's setting.

Claim 3: Temporal credit propagation is the enabling mechanism for multi-turn self-improvement training.

The ablation in Table 2 provides clean evidence: setting γ = 0 drops average accuracy from 46.0% to 43.8%. This is a 2.2 percentage point gap that is consistent across most benchmarks. The paper correctly interprets this as evidence that backward credit flow matters. However, the ablation only tests two extreme values—γ = 0 and γ = 1. Intermediate discount factors (e.g., γ = 0.5, 0.9) are not tested, nor are alternative credit assignment schemes (e.g., assigning credit only to episodes where the answer changed, or using the PRM-style scoring of intermediate reflections). A continuous sweep of γ values would reveal whether the benefit is monotonic in γ or whether there is an optimal intermediate value. Given that γ = 1 treats all future episodes as equally important regardless of temporal distance, there might be benefits to partial discounting that are not explored.

Furthermore, the γ = 0 condition changes two things simultaneously: it removes future credit propagation AND it means each episode is optimized independently of others (since the advantage for episode n depends only on rn, not on rn+1, rn+2, etc.). This makes γ = 0 effectively equivalent to treating episodes as independent—similar to Search-R1's objective but with episodes generated sequentially rather than independently. The degradation could be due to either the loss of credit propagation or the inherent difficulty of optimizing independent episodes that share context. The paper does not disentangle these two factors.

The comparison with MT-GRPO (which does some form of multi-turn credit assignment) shows a smaller gap (46.0% vs. 44.3%), suggesting that any form of cross-turn credit assignment helps, but the RLOO-based unbiased estimator provides additional benefit. However, the specific source of MT-GRPO's underperformance is unclear—it could be the biased advantage estimation, the grouping strategy, or other implementation details.

Claim 4: The approach generalizes across different task types and difficulty levels.

The paper evaluates on single-hop QA (3 benchmarks), multi-hop QA (4 benchmarks), and a synthetic long-horizon dataset (ASearcher). MR-Search outperforms baselines on all eight—this is the strongest evidence for generalization. However, all benchmarks share a common format: factoid questions with short, extractable answers evaluable via exact match. The paper acknowledges this limitation explicitly: "We do not evaluate our method on long-form benchmarks, where responses are substantially longer. Verification in such settings is inherently challenging." The approach's reliance on a rule-based verifier for exact match scoring means it cannot directly transfer to tasks where correctness is multi-dimensional, subjective, or requires open-ended generation. This is not a flaw in the experiments—the paper is honest about the scope—but it means the generalization claim is limited to a specific, albeit broad, class of factoid QA tasks.

The paper also does not evaluate on out-of-distribution questions (different domains, different question styles, different languages). The training data comes from NQ and HotpotQA; the evaluation includes these same domains (NQ, HotpotQA) plus five held-out benchmarks. While the held-out benchmarks are not seen during training, they share the Wikipedia-based, English-language, factoid QA format. True generalization would require evaluation on substantially different distributions.

Methodological concerns and missing experiments:

  • Single trajectory evaluation. The main results use a single trajectory per question. This is a low-budget evaluation that advantages methods producing high-quality first attempts. MR-Search gets N sequential attempts within that single trajectory, while Search-R1 gets one. This is not inherently unfair—it reflects MR-Search's core design—but it means the comparison does not control for the total number of tool calls or tokens generated. A budget-matched comparison where Search-R1 is allowed N independent trajectories (with majority voting or best-of-N selection) and the total tool calls are equalized would test whether the sequential reflection structure is inherently more efficient than parallel sampling.

  • No oracle reflection baseline. The paper does not compare against a version of MR-Search where the reflection text is generated by a human or a stronger model. Such a comparison would establish an upper bound on how much better reflections could be and reveal whether MR-Search's learned reflections are near-optimal or far from the ceiling.

  • No comparison against supervised fine-tuning on expert trajectories. An alternative approach to training search agents is to collect high-quality search trajectories (from humans or stronger models) and perform supervised fine-tuning. The paper compares only against RL-based methods. A supervised baseline would help establish whether RL training is necessary at all for this task.

  • Small sample size for Bamboogle. Bamboogle has only 125 test questions. Performance differences on this benchmark—where MR-Search achieves 45.2% vs. 40.0% for Search-R1 on the 7B model—correspond to only about 6-7 question differences. The paper reports standard deviations for some experiments (Figures 3 and 5) but not for the main table results, making it difficult to assess whether the Bamboogle improvement is statistically significant.

  • No analysis of what the reflections actually contain. The paper provides case studies (Appendix Tables 4-7) showing successful reflection trajectories, but does not systematically analyze reflection quality—how often reflections correctly identify errors, how often they suggest productive new search directions, or how reflection quality correlates with downstream answer correctness. Such analysis would strengthen the claim that MR-Search learns to generate useful reflections specifically, rather than just learning to produce more search queries.

  • Hyperparameter sensitivity unexplored. The paper fixes group size G = 5, number of episodes N = 3, learning rate 1e-6, and number of training steps 300. There is no sensitivity analysis for any of these. The dependence on group size is particularly important—a larger G provides more accurate baseline estimates but consumes more compute per update. The paper does not report whether performance is robust to smaller group sizes (which would be important for practical deployment with limited GPU memory).

What the experiments genuinely demonstrate:

The experiments convincingly demonstrate that training a search agent to generate multiple answer attempts with explicit self-reflection in context, and optimizing using a meta-RL objective with turn-level credit propagation, produces better single-trajectory performance than training with outcome-reward-only RL on the same base models and benchmarks. This is the paper's core empirical contribution, and the evidence—consistent across eight benchmarks, two model sizes, multiple ablations, and training dynamics analyses—is solid.

The experiments also demonstrate that this training approach produces agents that can effectively use additional reflection turns at test time (Figures 3 and 5), which is a qualitatively different capability from agents trained for single-shot performance. This is perhaps the most compelling evidence that MR-Search has genuinely learned a meta-skill rather than simply overfitting to the training distribution of 3-episode sequences.

What remains less clear is whether the specific mechanisms—RLOO-based advantage estimation, backward credit propagation with γ = 1, episode-level rather than token-level or step-level credit assignment—are all individually necessary, or whether the primary driver of improvement is simply the meta-episode structure itself (generating multiple attempts sequentially with reflection prompts) combined with any reasonable multi-turn RL objective. The ablation comparing against Search-R1 with the same reflection mechanism at inference time (Figures 3 and 5) confirms that the meta-episode structure alone is insufficient without appropriate training. But the comparison against MT-GRPO (Table 2) suggests that multiple multi-turn RL formulations can work, with MR-Search's specific advantage estimator providing incremental gains. A more comprehensive ablation that varies the meta-episode structure (e.g., 2 episodes vs. 3 vs. 5, different reflection prompt formats, different context windows) while holding the RL algorithm constant would help isolate the structural contributions from the algorithmic ones.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Claims

The assumption or constraint.
MR-Search's training procedure requires sampling $G \times N$ complete trajectories per training question (group size × number of episodes), each involving up to 3–5 tool calls with retrieval from Wikipedia. The paper reports a group size of $G = 5$ and $N = 3$ episodes, meaning 15 full search trajectories are generated for every single training question in each update step. Additionally, the test-time scaling analysis (Figures 3 and 5) involves extrapolating beyond the 3 training turns by appending the entire interaction history to context, which grows linearly with the number of turns. The paper does not account for this increased compute in any budget-matched comparison against baselines.

The consequence.
The headline improvements over Search-R1 (9.2% relative on Qwen2.5-7B, 19.3% on Qwen2.5-3B) are achieved with substantially more computation per training question. Search-R1 generates one trajectory per question per update; MR-Search generates 15. If Search-R1 were given an equivalent compute budget—either by training on 15× more unique questions, taking 15× more gradient steps, or using a larger batch size—its performance might close some or all of the gap. The paper does not disentangle whether the gains come from the meta-RL formulation itself or simply from the increased effective training data (each question contributes 15 trajectories of learning signal rather than 1). This matters enormously for practitioners: if the benefit is primarily data amplification, then simply sampling more trajectories per question with any RL algorithm might achieve similar results at lower implementation complexity.

What evidence exists in the paper.
The paper does not perform a compute-matched comparison. The training dynamics plots (Figures 4, 6, 7) show training steps on the x-axis and report training accuracy, but "one training step" for MR-Search involves 15 trajectory generations while "one training step" for Search-R1 involves sampling a comparable number of trajectories (since GRPO also uses a group of trajectories for its baseline). The paper does not report total FLOPs or wall-clock time, making it impossible to assess whether MR-Search is genuinely more compute-efficient or simply uses more compute. The test-time scaling analysis (Figures 3 and 5) shows MR-Search improving with additional turns, but the cost of those additional turns (extra tool calls, longer context windows) is not factored into any efficiency metric.

Mitigation status.
Not addressed. The paper does not discuss this compute discrepancy or attempt to control for it. The ablation in Table 2 compares different algorithms within the MR-Search framework but never compares against a version of Search-R1 trained with comparable total trajectory count. The authors do not suggest that future work should perform FLOPs-matched comparisons—this is my assessment of what is needed.

Limitation 2: The Method Is Strictly Bounded to Factoid QA with Exact-Match Verification

The assumption or constraint.
MR-Search's entire training pipeline depends on a rule-based verifier ($f_{\text{verifier}}$) that performs exact match after answer normalization. Every reward signal—whether for the meta-level objective (Eq. 6), the RLOO baseline (Eq. 7), or the discounted cumulative advantage (Eq. 8)—comes from comparing an extracted answer string against ground-truth answer strings. The paper explicitly acknowledges this scope constraint:

"We do not evaluate our method on long-form benchmarks, where responses are substantially longer. Verification in such settings is inherently challenging, and how to reliably assess progress and final correctness for long-form generation remains an open research question." (Limitations section)

Additionally, the paper states: "our current study focuses on agentic search with a fixed Wikipedia search tool" and does not extend to "environments involving multiple heterogeneous tools."

The consequence.
MR-Search cannot be directly applied to any task where correctness is not binary and exactly matchable. This excludes: long-form QA where answers are paragraphs rather than short strings; open-ended generation tasks (summarization, creative writing, dialogue); tasks requiring multi-dimensional evaluation (where an answer might be partially correct, factually accurate but poorly reasoned, or correct but incomplete); and any task where ground-truth answers are unavailable or ambiguous. Even within the factoid QA domain, the exact-match metric is brittle—it penalizes answers that are semantically correct but syntactically different from the reference (e.g., "May 22, 1964" vs. "22 May 1964"). The paper uses answer normalization, but normalization heuristics are task-specific and can introduce their own errors.

Furthermore, the reliance on Wikipedia as the sole knowledge source means the method is untested on tasks requiring real-time web search, database queries, code execution, or interaction with APIs beyond document retrieval. The paper's claim about "agentic search" is thus narrower than the term might suggest—this is agentic search over a static Wikipedia corpus using a single retrieval tool.

What evidence exists in the paper.
All eight benchmarks (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle, ASearcher) are factoid QA datasets with short extractable answers and exact-match evaluation. The paper's Limitations section explicitly acknowledges the lack of long-form evaluation and the fixed-tool constraint. There is no experiment showing MR-Search on tasks requiring open-ended generation, multi-tool coordination, or learned reward models for verification.

Mitigation status.
The paper acknowledges these limitations transparently in its Limitations section but does not attempt to address them. The step-level meta-RL extension (Section 3.4, Table 3) uses intermediate answers at each tool call and evaluates them with the same exact-match verifier—this provides denser supervision but does not escape the fundamental requirement for matchable ground-truth answers at every intermediate step. Future work on learned verifiers or LLM judges for open-ended tasks is mentioned as a direction but not explored.

Limitation 3: The Self-Reflection Mechanism Has No Guarantee of Monotonic Improvement and Can Degrade Correct Answers

The assumption or constraint.
MR-Search trains the policy to generate multiple sequential answer attempts, each conditioned on all previous attempts and an explicit reflection prompt. The meta-objective (Eq. 6) sums rewards across all episodes, meaning the model is incentivized to maximize total correctness, not to ensure that each episode is better than the previous one. There is no architectural constraint or training signal that prevents the model from "revising" a correct answer into an incorrect one during the reflection process.

The paper does not explicitly discuss this risk, but it is a direct consequence of the design. Since all episodes contribute equally to the objective (with $\gamma = 1$), a trajectory that goes correct → incorrect → correct receives the same total reward (2 correct, 1 incorrect) as one that goes incorrect → correct → correct, even though the first trajectory exhibits a regression that would be problematic in deployment.

The consequence.
At test time, additional reflection turns could degrade answer quality. The test-time scaling curves (Figures 3 and 5) show MR-Search's performance increasing with additional turns on average, but average performance can mask per-question regressions—some questions might get worse with more turns even as the aggregate improves. The paper does not report whether MR-Search ever turns correct answers into incorrect ones during the reflection process, or how frequently this occurs.

This matters practically because a deployment cannot know in advance whether adding more reflection turns will help or hurt a specific question. If the model sometimes degrades correct answers, then blindly applying more reflection turns (the test-time scaling strategy the paper advocates) could reduce accuracy on those questions, even as aggregate accuracy improves. A robust system would need a stopping criterion or selection mechanism—for instance, using the verifier (if available) to pick the best answer across all episodes—but MR-Search is designed for settings where no verifier is available at inference time.

What evidence exists in the paper.
The case studies in Appendix A.2.2 provide anecdotal evidence. In Case Study 2 (Table 5), the model's first reflection produces an incorrect answer ("P-3C Orion"), but the second reflection corrects this to "Hawker Siddeley Nimrod." This is a correct → incorrect → correct pattern. The case studies are selected to demonstrate successful trajectories, so they do not reveal how often the final answer is worse than the first. The paper does not report per-episode accuracy breakdowns that would reveal regression rates. The training dynamics and test-time scaling curves are all aggregate metrics that can hide per-question degradation.

Mitigation status.
Not addressed. The paper does not implement a within-chain selection mechanism (e.g., taking the most common answer across episodes, using the PRM or verifier to select the best episode). The meta-objective treats all episodes symmetrically, providing no incentive for monotonic improvement. The exploration-exploitation masking variant (Table 3) masks rewards for early episodes but does not prevent later episodes from being worse than early ones. A stopping criterion or answer selection mechanism would be a natural extension but is not explored.

Limitation 4: Evaluation Uses a Single Trajectory per Question, Obscuring the Efficiency Tradeoff Against Parallel Sampling

The assumption or constraint.
The main results (Table 1) evaluate all methods using a single trajectory per question, reporting the exact match of the last valid prediction. For MR-Search, this single trajectory contains $N$ sequential episodes (3 during training, extrapolated to more at test time), each with up to 3–5 tool calls. For Search-R1 and other baselines, this single trajectory contains one episode.

This evaluation protocol is inherited from Search-R1 (Jin et al., 2025a) and provides a clean comparison of single-trajectory quality. However, it does not control for the total number of tool calls, tokens generated, or wall-clock time between methods. MR-Search's single trajectory costs approximately $N$ times more tool calls and generates substantially more tokens than Search-R1's single trajectory, yet they are compared as if they consume the same inference budget.

The consequence.
The reported gains (9.2% relative improvement on Qwen2.5-7B, 19.3% on Qwen2.5-3B) could be partially or entirely attributable to MR-Search simply doing more computation per question at inference time. A fairer comparison would be budget-matched: give Search-R1 an equivalent inference budget by allowing it $N$ independent trajectories (parallel sampling) and selecting the answer via majority voting or best-of-N weighted selection, then compare against MR-Search's $N$ sequential episodes with the same total tool calls. If parallel sampling with majority voting achieves comparable accuracy to sequential reflection at the same inference cost, then MR-Search's sequential structure provides no efficiency advantage—it would simply be an alternative way to spend compute, not a superior one.

The paper does partially address this in Figures 3 and 5, where it compares MR-Search against Search-R1 with parallel sampling (Search-R1-P) using the most frequent answer among $N$ trajectories. MR-Search outperforms Search-R1-P at all turn counts. However, this comparison is only shown in the test-time scaling context, not in the main results table. The main Table 1 does not include a Search-R1-P baseline with budget-matched parallel sampling, so readers of the headline results do not see whether the sequential structure is inherently more efficient than simply generating more independent answers.

What evidence exists in the paper.
Figures 3 and 5 provide the only budget-comparable evidence. On TriviaQA (Figure 5), at 5 turns, MR-Search achieves approximately 67% vs. Search-R1-P's 63%. On Bamboogle, it is approximately 48% vs. 40%. These gaps suggest sequential reflection does provide benefits beyond parallel sampling, even at matched turn counts. However, these figures only appear in the analysis section (Section 4.3), not in the main results table, and they are shown for a subset of benchmarks.

The paper does not report the total token count or wall-clock time for any method, making it impossible to assess whether the sequential structure introduces latency that would be unacceptable in interactive applications. Sequential episodes are inherently serial—each episode depends on the previous one—while parallel sampling can be executed simultaneously given sufficient hardware.

Mitigation status.
Partially addressed in Figures 3 and 5, but not in the main results table. The paper does not report latency, token counts, or FLOPs, and does not frame its main comparison as budget-matched. A practitioner reading only the abstract and Table 1 would not know that MR-Search uses $N$ times more inference compute per question than the baselines it is compared against.

Limitation 5: The Method Has Been Validated on Only Two Model Sizes from a Single Model Family

The assumption or constraint.
All experiments use Qwen2.5-Base models (Qwen2.5-3B-Base and Qwen2.5-7B-Base). The paper states it "believe[s] this model is representative of the capabilities of many contemporary LLMs" (this statement is implicit in the choice—the paper does not explicitly claim representativeness but treats the results as general findings). The training setup, hyperparameters (learning rate 1e-6, group size 5, training steps 300, context lengths 8K–16K), and retrieval configuration (E5 embeddings, top-3 documents, 2018 Wikipedia dump) are all fixed to this specific model family and scale.

The consequence.
Several aspects of MR-Search's effectiveness could be model-dependent in ways that limit generalization to other architectures or scales. Qwen2.5 models may have specific in-context learning capabilities, self-reflection tendencies, or tool-use behaviors that differ from other model families (e.g., Llama, Gemma, Mistral). The optimal number of episodes ($N = 3$), the reflection prompt format, and the group size ($G = 5$) may not transfer to models with different context window sizes, attention mechanisms, or pre-training data distributions.

More critically, the paper's central claim—that meta-RL with self-reflection can substitute for external process rewards—might only hold at the 3B–7B scale. At smaller scales (e.g., 1B parameters), the model may lack sufficient in-context learning capacity to effectively condition on previous episodes and reflections. At larger scales (e.g., 70B+), the base model's single-shot performance may be high enough that the marginal benefit of multi-episode reflection diminishes, or the computational cost of generating $G \times N$ trajectories per update may become prohibitive. The paper provides no evidence about scaling trends—does the relative improvement over Search-R1 increase, decrease, or stay constant as model size grows?

What evidence exists in the paper.
The paper tests exactly two model sizes: 3B and 7B parameters. The relative improvement is larger on the 3B model (19.3% vs. 9.2%), which is consistent with the intuition that smaller models benefit more from structured exploration because their single-shot performance is lower. But two data points cannot establish a trend, and there is no experiment with a larger model or with a model from a different family. The paper does not ablate the number of episodes $N$ as a function of model size, nor does it test whether the group size $G = 5$ is appropriate for both scales.

Mitigation status.
Not addressed. The Limitations section does not mention model family dependence. The paper does not claim to have tested other model families or scales, but it also does not caution readers that the findings might not transfer. Future work scaling MR-Search to "frontier base models" is mentioned in the Limitations section, but only in the context of scaling training runs, not in the context of verifying that the method works on different architectures.

Limitation 6: No Mechanism for Early Stopping or Adaptive Episode Count

The assumption or constraint.
MR-Search generates a fixed number of episodes $N$ per meta-episode during training (set to 3). At test time, the number of reflection turns is a hyperparameter chosen by the user—the paper shows performance improving with additional turns up to 5–6 (Figures 3 and 5), but there is no learned mechanism for the model to decide when to stop reflecting. The model always produces exactly as many episodes as it is prompted to produce, regardless of whether the answer has converged, whether additional searches are yielding redundant information, or whether further reflection is likely to degrade rather than improve the answer.

This is a consequence of the fixed meta-episode structure. The training objective (Eq. 6) sums over a predetermined number of episodes; it never teaches the model to output a "stop" token or to recognize when its current answer is sufficient. The reflection prompt (Appendix A.1.3) instructs the model to provide an improved answer but provides no criterion for deciding that no improvement is needed.

The consequence.
At deployment, the user must pre-specify how many reflection turns to allow—a decision that trades off accuracy against latency and compute cost. Setting the number too low risks leaving potential improvements on the table; setting it too high wastes compute on unproductive additional searches and risks the degradation problem discussed in Limitation 3. The optimal number of turns likely varies by question difficulty, but MR-Search provides no way to estimate this per question.

This stands in contrast to methods that use verifier scores to decide when to stop searching (e.g., beam search with a confidence threshold, or iterative refinement until the verifier score stabilizes). MR-Search is designed for settings where no verifier is available at inference time, so such score-based stopping criteria are unavailable. But the paper does not explore whether the model could be trained to output its own confidence estimate or to produce a special termination token when it believes further reflection would be unproductive.

The test-time scaling curves (Figures 3 and 5) show MR-Search's performance continuing to improve through 5–6 turns, but these are aggregate curves. On some questions, performance likely plateaus or degrades after 2–3 turns; on others, it might continue improving through 10+ turns. Without adaptive stopping, the user must choose a single turn count for all questions, leaving efficiency on the table.

What evidence exists in the paper.
The test-time scaling analysis (Figures 3 and 5) shows performance as a function of turn count, but always with the same number of turns applied uniformly to all questions. The paper does not report per-question convergence behavior or attempt to identify questions where additional turns are productive vs. wasteful. The case studies in Appendix A.2.2 show the model continuing to revise its answer across multiple turns, sometimes correcting errors and sometimes confirming previous answers, but no case shows the model voluntarily stopping.

Mitigation status.
Not addressed. The paper treats the number of episodes as a fixed hyperparameter both at training and inference time. Adaptive stopping is not discussed in the Limitations section or suggested as future work. This is a significant gap for practical deployment, where per-query compute budgets need to be allocated efficiently across a distribution of question difficulties.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual reframing rather than a paradigm shift: it recasts agentic search training from a single-episode RL problem to a meta-RL problem across self-reflection episodes, and in doing so, reveals that the bottleneck in training capable search agents is not the RL algorithm itself but the impoverished structure of the learning signal. The magnitude is an architectural reframing with substantial practical payoff—9.2% to 19.3% relative improvement across eight benchmarks is large enough to change what practitioners consider the "baseline" approach to training search agents, but the underlying components (ReAct-style tool use, RLOO advantage estimation, PPO-style clipped surrogates) are all borrowed from prior work. The novelty is in how they are composed.

The most significant methodological shift is the elevation of credit assignment structure from an implementation detail to a first-class design dimension. Prior work on RL for search agents treated credit assignment as something the RL algorithm handles automatically—PPO or GRPO with a value function baseline was assumed sufficient. MR-Search demonstrates that how credit flows across time (episodes within a meta-episode) matters more than which policy gradient estimator is used, provided the estimator respects that temporal structure. The ablation in Table 2 makes this concrete: PPO (a more sophisticated algorithm with a learned value function) underperforms Search-R1 within the MR-Search framework (42.0% vs. 43.5% average), while the simple RLOO estimator with proper turn-level discounting achieves 46.0%. This inverts the typical RL narrative—algorithmic sophistication matters less than whether the reward structure matches the task's temporal dependencies. For the field, this means future work on training LLM agents should prioritize reward structure design (what gets rewarded, when, and how credit propagates) over algorithmic innovation in the policy gradient estimator.

The paper also resolves a latent contradiction in the self-reflection literature. Prompting-based self-reflection (Reflexion, Self-Refine) shows that LLMs can sometimes improve their own outputs through iterative refinement, yet the self-correction literature (Huang et al., 2023) demonstrates that "LLMs cannot self-correct reasoning" when relying solely on prompting. MR-Search's framework explains this discrepancy: prompting elicits a behavior the model was never trained to optimize. The model can produce plausible-sounding reflections, but whether those reflections actually lead to better answers is uncorrelated with their surface quality because the model has no training signal connecting reflection quality to downstream task performance. MR-Search closes this loop by making the downstream utility of the reflection the explicit optimization target. The implication is that prompting-based self-improvement methods are not wrong—they are undertrained. With appropriate meta-RL training, the same reflection format that fails under zero-shot prompting becomes a reliable mechanism for iterative improvement.

The paper also redirects research investment away from external process reward models—at least for factoid QA with accessible ground-truth answers. The finding that MR-Search outperforms PPRM and StepResearch (methods that invest substantial effort in training process reward models) suggests that, when outcome verification is cheap and reliable, structured self-comparison across episodes is a more effective use of research effort than building and maintaining external reward models. This does not mean process reward models are obsolete—they remain essential for tasks where outcome verification is impossible or extremely sparse—but it shifts the burden of proof: future work proposing process reward models for tasks with verifiable outcomes must now demonstrate superiority over self-comparison approaches like MR-Search, not merely over outcome-reward-only baselines.

However, the paper also narrows the scope of claims that can be made about test-time scaling for search agents. The test-time scaling curves (Figures 3 and 5) show MR-Search improving with additional reflection turns, but these improvements come from additional tool calls and longer context windows—not from "thinking harder" in the sense of internal reasoning without external information. This contrasts with the test-time scaling narrative in mathematical reasoning (where models improve through longer chain-of-thought without external tools). The paper demonstrates that for search agents, test-time scaling is fundamentally about acquiring more external information, not about deeper internal reasoning. This distinction matters because it bounds the ceiling: performance cannot improve beyond what the retrieval corpus contains, regardless of how many reflection turns are added.

Finally, the paper establishes that in-context meta-learning is a viable training paradigm for LLM agents in settings where environment feedback is unavailable at inference time. This bridges the gap between meta-RL (which typically assumes dense environment rewards during meta-training and meta-testing) and practical LLM deployment (where only the final output can be evaluated, and even that only during training). The mechanism—training the model to generate its own feedback signal (reflection text) and then conditioning on that self-generated signal—is a template that could apply to any sequential decision-making task where ground-truth outcomes are available for training but not for inference.

Follow-Up Research This Work Enables

Compute-matched comparison against Search-R1 with equivalent trajectory budget. The most immediate open question is whether MR-Search's gains come from the meta-RL formulation or simply from generating 15× more trajectories per training question (G=5 groups × N=3 episodes). A strong follow-up would train Search-R1 with the same total trajectory count per update—either by using 15× larger batches, 15× more gradient steps, or 15× more unique questions—and compare against MR-Search at matched total FLOPs. If Search-R1 closes the gap, then MR-Search's contribution is primarily a data amplification strategy (generating more learning signal per question) rather than a fundamentally better training objective. If the gap persists, the formulation itself is validated as non-redundant with data quantity. This experiment is straightforward to run within the existing codebase and would substantially strengthen the paper's claims.

Scaling MR-Search to 70B+ models and measuring whether relative gains persist or diminish. The paper tests only 3B and 7B models. The relative improvement is larger on the smaller model (19.3% vs. 9.2%), consistent with smaller models benefiting more from structured exploration. But two data points do not establish a trend. A scaling study across 1B, 3B, 7B, 14B, and 70B parameter models—all trained with identical MR-Search and Search-R1 protocols—would determine whether the meta-RL advantage follows a power law, plateaus, or eventually reverses at larger scales where single-shot performance may be high enough to make multi-episode refinement unnecessary. If the relative gain asymptotically approaches zero at large scales, then MR-Search is primarily a technique for making small models competitive, not for pushing frontier performance. If the gain persists or grows, it suggests meta-RL captures a capability that does not emerge from scale alone.

Replacing the rule-based verifier with a learned verifier for open-ended generation tasks. The paper's most significant scope constraint is its reliance on exact-match verification, which limits applicability to factoid QA. The obvious extension is to train a learned outcome verifier (e.g., an LLM judge fine-tuned on human preference data for long-form answers) and use it in place of the exact-match function in Eqs. (6–8). The critical question is whether the RLOO-based advantage estimation remains stable when the verifier is noisy or miscalibrated—a binary exact-match signal has zero noise, but a learned verifier introduces stochasticity that could inflate the variance of the advantage estimate. A strong follow-up would test MR-Search on a long-form QA benchmark (e.g., ELI5, ASQA) using a learned verifier, measuring both answer quality (via automated metrics and human evaluation) and training stability (variance of the advantage estimate across training). If the approach transfers, it dramatically expands MR-Search's applicability. If training becomes unstable, the paper would reveal a fundamental tension between self-generated feedback and verifier noise—the meta-RL formulation may require near-deterministic outcome signals to provide reliable credit assignment.

Adaptive episode count via learned stopping criteria. The current method uses a fixed number of episodes N, chosen as a hyperparameter. A more efficient system would learn to stop reflecting when further episodes are unlikely to improve the answer. One concrete approach: add a special "stop" action that the model can output instead of a reflection, and train it using the meta-RL objective where the model receives a small negative reward for each additional episode (to penalize unnecessary computation) plus the verifier reward for correctness. This creates a tradeoff between thoroughness and efficiency that the model must learn to navigate per question. A strong evaluation would measure not just final accuracy but accuracy per unit of inference compute (tool calls or tokens), showing that the adaptive variant achieves comparable accuracy to fixed-N MR-Search with significantly fewer average turns. The test-time scaling curves (Figures 3 and 5) show that performance continues improving through 5–6 turns in aggregate, but per-question analysis would likely reveal that easy questions plateau after 1–2 turns while hard questions benefit from 5+. An adaptive policy could capture these differences automatically.

Cross-model-family replication to establish generality of the meta-RL formulation. All experiments use Qwen2.5 models. The paper makes no claims about model-family specificity, but the effectiveness of in-context meta-learning could depend on the base model's pre-training mixture, context window utilization, or instruction-following capabilities. A replication study using Llama-3, Gemma, and Mistral base models at comparable scales (3B–8B parameters) with identical training protocols would establish whether MR-Search's benefits are universal or specific to Qwen's architecture and training data. If the method transfers cleanly, it becomes a general-purpose training recipe for search agents. If it fails on some families, analyzing why (e.g., does Llama-3 struggle with the reflection prompt format? Does Gemma's context utilization degrade with long meta-episodes?) would reveal which base model properties are prerequisites for effective in-context meta-learning.

Combining MR-Search with process reward models to test whether the approaches are complementary or redundant. The paper positions MR-Search as an alternative to external process rewards, but it does not test whether combining both yields further gains. A natural experiment: train a process reward model on MR-Search's own trajectories (which include explicit reflections) and use it to provide per-step rewards in addition to the episode-level RLOO advantages. If performance improves beyond either approach alone, it suggests that self-comparison and learned process rewards capture complementary aspects of credit assignment—self-comparison provides a global signal about episode quality, while process rewards provide local signals about individual reasoning steps. If performance does not improve or degrades, it suggests redundancy or conflicting gradients, which would strengthen the paper's claim that self-comparison alone is sufficient when outcome verification is possible. The experiment would also test whether MR-Search's reflections provide a naturally rich source of training data for process reward models, potentially making them cheaper and more accurate to train.

Measuring and mitigating the correct-to-incorrect reversion rate. The paper does not report how often MR-Search turns a correct answer into an incorrect one during reflection. This is a critical safety and reliability metric for any deployed system. A simple analysis: for each question, track the correctness of each episode's answer and compute the probability that episode n+1 is incorrect given that episode n was correct. If this probability is non-trivial (say, >10%), then blindly applying more reflection turns is risky. Potential mitigations include: (1) training with an asymmetric reward that penalizes correct→incorrect transitions more heavily than incorrect→incorrect transitions; (2) using the model's own confidence (e.g., token probabilities at the answer span) as a proxy for whether to continue reflecting; or (3) generating multiple parallel reflections at each turn and selecting the most common answer (a within-episode majority vote). This analysis would ground the test-time scaling results in per-question reliability rather than aggregate trends.

Practical Applications and Downstream Use Cases

Training small on-device search agents that rival cloud-scale models through iterative refinement. The paper's largest relative gains appear on Qwen2.5-3B (19.3% improvement over Search-R1), and MR-Search achieves 41.4% average accuracy on this small model—competitive with much larger models using simpler training. For on-device deployment scenarios (smartphones, laptops, edge devices) where a 3B model is the maximum that fits in memory, MR-Search provides a training recipe that squeezes substantially more capability out of limited parameters. The test-time scaling property (Figures 3 and 5) means the deployed system can trade latency for accuracy: on easy questions where the first answer is likely correct, stop after one episode; on harder questions where the model is uncertain, allow 3–5 reflection episodes with additional searches. The key practical requirement missing from the paper is a cheap uncertainty estimator—perhaps the model's own token probabilities at the answer span, or a lightweight classifier trained on the reflection text—to decide per question how many turns to allocate. Without this, the deployment would need to use a fixed turn count, leaving some efficiency on the table.

Cost-efficient training data generation for self-improvement pipelines. A growing paradigm in LLM development is using models to generate their own training data (e.g., STaR, ReST, rejection sampling fine-tuning). The quality of generated data depends heavily on the model's ability to produce correct answers, especially for complex multi-step questions. MR-Search's meta-episode structure naturally produces multiple answer attempts per question, each with an explicit reflection that diagnoses what went wrong. This is high-quality training data for subsequent supervised fine-tuning: correct answers from later episodes can be used as target outputs, and the reflection text provides natural rationales. Moreover, the reflections themselves can be used to train a separate "self-critique" model or to identify common failure modes. Compared to standard sampling (which produces independent trajectories that may all be wrong), MR-Search's sequential structure increases the probability that at least one episode in the chain is correct, improving the yield of usable training data per question. The paper's numbers suggest this: on Qwen2.5-7B, MR-Search's test-time scaling reaches ~67% on TriviaQA at 5 turns vs. ~64% for Search-R1 single-shot, meaning the sequential process produces a correct answer for roughly 3% more questions that could be added to the training pool.

Batch evaluation of search agent quality without ground-truth answers. In production settings where search agents are deployed to answer user questions, evaluating agent quality typically requires human annotation or comparison against ground-truth answers—both expensive. MR-Search's cross-episode structure enables a form of self-consistency evaluation: if the model's answer converges across multiple reflection episodes (i.e., episodes 2, 3, and 4 all produce the same answer), that answer is more likely to be correct than a single-episode answer. Conversely, if answers oscillate across episodes, the question is likely difficult for the model. This provides a cheap, automated quality signal that could be used for monitoring, filtering, or routing (e.g., escalating oscillating questions to a larger model or human reviewer). The paper does not analyze this convergence property, but the framework naturally supports it—one would simply compare answers across episodes and use agreement as a proxy for confidence. A deployment could set a threshold (e.g., 3 consecutive episodes with the same answer) and treat that as high-confidence; questions below threshold get additional resources.

Fast bootstrapping of search agents for new domains or knowledge bases. The paper trains on NQ and HotpotQA and evaluates on five held-out QA benchmarks, demonstrating that MR-Search's learned self-reflection skill transfers across question distributions. This suggests a practical workflow for deploying search agents on proprietary or domain-specific corpora: take a base model, train MR-Search on whatever labeled QA data is available (even a few thousand examples), and deploy with test-time reflection. Because the method uses only outcome verification (no process labels needed), the training data requirement is minimal—any corpus where questions have verifiable answers suffices. The meta-learned reflection skill means the model can adapt its search strategy to the new corpus's structure without requiring domain-specific reflection examples. For an enterprise with a private document collection and a set of FAQ-like question-answer pairs, MR-Search provides a recipe that is substantially simpler than training a process reward model or collecting expert search trajectories—just run meta-RL training on the available QA pairs and deploy with inference-time reflection.