In-Context Reinforcement Learning for Tool Use in Large Language Models

🎯 Pitch

ICRL shows that large language models can master tool use without any supervised fine-tuning data. By simply showing a few examples inside the RL training rollouts and then gradually removing them, models leap from imitation to autonomous tool calling. This pure RL approach slashes data costs and beats traditional SFT+RL pipelines by up to 9 points on knowledge-intensive benchmarks.

1. Executive Summary

This paper introduces In-Context Reinforcement Learning (ICRL), an RL-only framework that eliminates the need for supervised fine-tuning by embedding few-shot demonstrations directly into RL rollout prompts to teach LLMs how to invoke external tools—then gradually phases out those demonstrations through a curriculum that transitions the model from imitation to autonomous tool use. Evaluated on the Qwen2.5 model family across five challenging QA benchmarks (TriviaQA, HotpotQA, 2Wiki, Musique, Bamboogle), ICRL achieves state-of-the-art performance without any SFT or labeled tool traces, outperforming the strongest baselines by up to +8.94 average EM on Qwen2.5-3B and +7.34 on Qwen2.5-7B, with double-digit gains concentrated on multi-hop reasoning tasks like 2Wiki and Musique. The method also generalizes across tool domains—matching or exceeding SFT+RL baselines on math reasoning benchmarks (AIME2024/AIME2025) with code execution tools—establishing that in-context demonstrations during RL rollouts can substitute for expensive cold-start supervision, provided the curriculum is designed to avoid premature reduction of in-context examples that would otherwise degrade multi-turn reasoning quality.

2. Context and Motivation

The Core Problem: Cold-Start Dependence Makes Tool Use Training Brittle and Expensive

The paper addresses a fundamental tension in training LLMs to use external tools. Tool augmentation—search engines, Python interpreters, calculators—is widely recognized as essential for overcoming the fixed knowledge limitations of pretrained models. However, the dominant training paradigm creates a dependency on high-quality supervised data before any reinforcement learning can begin. Specifically, most existing approaches follow a cold-start pipeline: first apply supervised fine-tuning (SFT) on labeled tool-use trajectories, then optionally fine-tune further with reinforcement learning (RL) using verifiable rewards. The SFT stage requires thousands of annotator- or model-generated demonstrations showing exactly how to decompose a query, when to invoke tools, how to format tool calls, and how to integrate retrieved information into a final answer.

This dependency matters for several practical reasons that the paper highlights implicitly through its experimental design rather than through explicit economic arguments:

Annotation cost scales poorly across domains. Every new tool (search, code execution, calculator, database query, API call) and every new task domain (factual QA, math reasoning, multi-hop reasoning) potentially requires a separate set of supervised demonstrations. The paper's comparison to O2-Searcher (Mei et al., 2025), which requires cold-start SFT for each tool-use setting, and to ReTool (Feng et al., 2025), which requires annotated data for code-augmented reasoning, makes this point concretely: ICRL matches or exceeds these methods without any SFT data at all.
Distribution mismatch between SFT and RL objectives. SFT trains the model to mimic fixed trajectories—what a human or a stronger model did for a particular query. RL trains the model to maximize a reward—what works according to an outcome measure. When the SFT trajectories are suboptimal or cover only a narrow slice of possible successful strategies, the model inherits those limitations before RL ever begins. The RL phase can correct some but not all of these baked-in biases, because the model starts from a policy that already strongly prefers the SFT-demonstrated behaviors.
SFT data quality is itself a bottleneck. Synthesizing tool-use demonstrations typically requires a stronger model (e.g., GPT-5.2 as used in this paper) to generate plausible reasoning-and-search traces. The quality of the resulting policy is upper-bounded by the quality of the synthetic demonstrations, and errors in those traces—incorrect search queries, poor integration of retrieved information, premature answer generation—get learned by the student model before RL can correct them.

Conflicting Demands: RL Without SFT Fails, But SFT Is Expensive

The paper is motivated by an empirical observation that creates a genuine methodological dilemma. Directly applying RL from scratch to tool-use tasks yields poor performance, as the model has no initial understanding of how to structure tool calls, when to search, or how to format its outputs. This is not just slow convergence—it is a fundamental exploration problem. The action space is vast (the model can generate arbitrary text at each step, including arbitrary search queries), the reward signal is sparse (binary feedback on final answer correctness, plus format penalties), and randomly initialized tool-use behavior almost never produces a correct answer, meaning the model receives no positive reinforcement signal to guide learning.

The standard solution—cold-start SFT—solves the exploration problem by providing the model with a reasonable initial policy: mimic these demonstrations, which show exactly how to search, reason, and answer. But this solution reintroduces the data dependency problem described above. The field is thus stuck between two unpalatable options: RL without SFT (ineffective exploration, no learning) or SFT+RL (expensive data requirements, distribution mismatch).

The paper's core insight is that these two options are not the only ones. There is a third path: provide soft supervision during RL exploration itself, not through a separate pretraining phase, but through in-context demonstrations embedded directly in the rollout prompts. This allows the model to see examples of successful tool use while it explores, guiding its initial behavior toward the right action space without ever requiring those demonstrations to be learned as fixed weights through SFT.

Where Existing Approaches Fall Short

The paper identifies specific limitations across three categories of prior work:

Direct prompting methods are cheap but static. Approaches like Chain-of-Thought (CoT) prompting (Wei et al., 2022) or few-shot tool-use prompting can elicit some tool-use behavior from instruction-tuned models without any training. However, the paper's baseline results (Table 3) show this works poorly: CoT achieves an average EM of only 1.52% on Qwen2.5-3B and 12.84% on Qwen2.5-7B across the five QA benchmarks. These methods cannot improve with experience—the model doesn't learn from its successes or failures, and the prompting strategy is fixed regardless of the query. Moreover, few-shot prompts consume precious context window space and increase inference cost without any compounding benefit across queries.

Retrieval-based methods are bound by retrieval quality, not reasoning quality. Methods like standard RAG, IRCoT (Trivedi et al., 2023), and Search-o1 (Li et al., 2025a) improve over direct prompting by incorporating retrieval, but they use fixed retrieval strategies that cannot adapt to model-specific strengths and weaknesses. The model has no ability to learn when to search versus reason from internal knowledge, what to search for, or how many search turns to execute—these are all determined by the prompting template or the retrieval pipeline, not learned from task feedback.

Existing RL-based methods still require SFT. Search-R1 (Jin et al., 2025a), ZeroSearch (Sun et al., 2025), and ParallelSearch (Zhao et al., 2025) all use RL to train tool-use policies, achieving strong results (Search-R1 reaches 38.16 average EM on Qwen2.5-7B; ParallelSearch reaches 41.78). However, these methods inherit the cold-start assumption or rely on existing instruction-tuned models without the benefit of in-context guidance during RL. The paper's results (Table 3) show ICRL substantially outperforms all of them, suggesting that the in-context guidance during RL exploration provides benefits beyond what RL alone—even with strong base models and careful reward design—can achieve.

Methods that do use SFT are data-hungry. O2-Searcher (Mei et al., 2025) achieves respectable performance (37.26 average EM on Qwen2.5-3B) by combining cold-start SFT with RL, but Table 4 directly compares this to ICRL: ICRL achieves 40.16 average EM on the same model size without any SFT, outperforming O2-Searcher on four of five datasets. This is a direct head-to-head validation of the paper's central claim—that in-context demonstrations during RL can replace an entire supervised pretraining phase while achieving better results.

ReTool (Feng et al., 2025) extends the SFT dependency to code execution. As shown in Table 7, ReTool uses an SFT+RL pipeline specifically for teaching models to write and execute Python code for math reasoning, achieving strong results (67.0% on AIME2024, 49.3% on AIME2025 with Qwen3-8B). But again, this requires annotated code-writing trajectories. ICRL achieves comparable performance (64.1% and 51.7% respectively) with no SFT, demonstrating that the in-context curriculum approach generalizes across tool modalities.

How This Paper Positions Itself

The paper's positioning is explicit but subtle. It does not claim that SFT is useless or that RL alone is sufficient—the preliminary results (and common knowledge in the field) show that RL from scratch fails precisely because of the exploration problem. Instead, the paper argues that the right question is not "SFT or RL?" but rather "what is the most efficient way to provide the model with initial guidance toward successful tool use?"

The proposed answer is to embed that guidance in the prompt rather than in the weights. This reframing has several implications that distinguish ICRL from prior work:

First, guidance is dynamic rather than frozen. In SFT, the demonstrations are baked into the model's parameters once and for all. In ICRL, the demonstrations can be progressively removed as the model internalizes the behavior, creating a natural curriculum. This is the core algorithmic contribution—not just "use few-shot prompts during RL," but "use a curriculum that reduces the number of shots and eventually removes them entirely." The ablation in Figure 2 shows this curriculum design matters crucially: a three-stage schedule (3→2→0 shots) substantially outperforms a four-stage schedule (3→2→1→0 shots), because the latter encourages premature stopping of multi-turn reasoning.

Second, the model learns when to use tools, not just how. SFT demonstrations show specific sequences of actions for specific queries. The model learns to imitate those sequences. ICRL, by contrast, shows the model general patterns—"here is how you format a search query, here is how you think about the results, here is how you know when you have enough information to answer"—and then lets RL optimize the decision of when to apply those patterns. The learning curves in Figure 3 support this: during the 0-shot phase (after demonstrations are removed), the number of valid search calls increases, meaning the model is learning to use tools more frequently and effectively even without explicit examples, driven purely by the reward signal.

Third, the framework is domain-agnostic. The paper demonstrates ICRL on two qualitatively different tool-use domains: web search for factual QA (Section 3) and code execution for math reasoning (Section 4.3). The same framework—few-shot demonstrations during RL rollouts, gradual phase-out, format+accuracy rewards—works in both settings without modification. This suggests that ICRL is not a search-specific hack but a general approach to tool-use training that could extend to API calling, database querying, or multi-agent coordination.

Fourth, ICRL sidesteps the on-policy/off-policy mismatch that plagues SFT+RL pipelines. When an SFT-trained model is subsequently fine-tuned with RL, the SFT trajectories are off-policy relative to the RL-updated model—they were generated by a different policy (the SFT model) under different conditions (teacher-forced demonstrations rather than reward-guided exploration). ICRL avoids this entirely because the demonstrations exist only in the prompt, not in the training objective. The model never learns to imitate them directly; it learns from the reward signal while the demonstrations merely shape the initial exploration distribution.

The Difficulty Multi-Hop Reasoning as a Stress Test

The paper's choice of evaluation benchmarks is deliberate and informative. TriviaQA, HotpotQA, 2Wiki, Musique, and Bamboogle are not random QA datasets—they form a gradient of reasoning complexity:

TriviaQA and HotpotQA involve single-hop or simple multi-hop reasoning where the model can often answer with one or two search queries and straightforward information integration.
2Wiki and Musique are designed specifically to test compositional reasoning across multiple documents—the model must retrieve information from different sources, verify facts, and compose them into a coherent answer.
Bamboogle (Press et al., 2023) is an adversarial benchmark specifically constructed to be difficult for retrieval-augmented models, requiring non-obvious search queries and careful integration of partial information.

The pattern of results in Table 3 reveals something important about where ICRL's advantages are largest. On Qwen2.5-3B, ICRL outperforms Search-R1 by +7.3 on 2Wiki, +9.7 on Musique, and +7.2 on Bamboogle—but the gains are more modest on TriviaQA (+8.2, but from a lower Search-R1 baseline) and HotpotQA (+3.0). This suggests that the in-context curriculum is particularly effective at teaching the multi-turn, compositional reasoning that SFT+RL pipelines struggle to capture in static demonstrations. When the required tool-use pattern is complex and query-dependent—involving multiple searches, conditional reasoning about retrieved information, and decisions about when to stop searching—teaching it through a curriculum of gradually-removed demonstrations appears more effective than teaching it through fixed SFT trajectories.

The ablation in Figure 2 reinforces this interpretation. The four-stage curriculum (with an intermediate 1-shot stage) causes premature stopping—over 80% of queries finish within two search turns—and dramatically degrades performance on datasets that require multi-hop reasoning. On TriviaQA, the four-stage model achieves only 20.8 EM versus 75.4 for the three-stage model. This is not a small difference; it is a catastrophic failure mode that demonstrates how curriculum design can determine whether the model learns to reason deeply or learns to answer as quickly as possible. The paper frames this as a finding about curriculum pacing—the model needs sufficient exposure to multi-turn reasoning patterns before being asked to generate them autonomously—but it is also evidence about the central challenge in tool-use training: avoiding premature exploitation of shallow strategies that achieve partial format rewards but fail on complex queries.

3. Technical Approach

3.1 Reader Orientation

ICRL is a training framework for teaching LLMs to use external tools—specifically search engines and Python interpreters—through reinforcement learning alone, with no supervised fine-tuning stage. The problem it solves is the cold-start dependency in tool-use training: RL from scratch fails because the model cannot explore effectively in the vast action space of possible tool calls and reasoning chains, but the standard fix (SFT on human- or model-generated tool-use trajectories) is expensive and brittle across domains. The "shape" of the solution is to provide soft guidance during RL exploration by embedding few-shot demonstrations directly into the rollout prompts—then progressively removing those demonstrations as the model internalizes tool-use behavior, creating a curriculum that transitions from imitation-guided exploration to autonomous reward-driven learning.

3.2 Big-Picture Architecture (Diagram in Words)

The ICRL system has four major components that interact in a multi-stage training loop:

Base Policy LLM ( $\pi_\theta$ ) — an instruction-tuned model (Qwen2.5-Instruct or Qwen3-Instruct) that generates tool-augmented reasoning chains. It serves as the policy being optimized. Initially, this model has no tool-use training; it can follow instructions but does not know how to structure search queries, integrate retrieved information, or decide when to stop searching.
Rollout Prompt Template with Few-Shot Demonstrations ( $P_N$ ) — a structured prompt containing $N$ example question-answer pairs that demonstrate the full tool-use workflow: reasoning in <thinking> tags, search queries in <search> tags, retrieved information in <information> tags, and final answers in <answer> tags. These demonstrations are prepended to every training query during the initial RL stages and are progressively removed as training advances ( $N \rightarrow N-1 \rightarrow \dots \rightarrow 0$ ).
External Tool ( $T$ ) — a retrieval function (Serper API for web search, returning top-3 documents per query; or a Python interpreter for code execution) that takes the model's generated tool calls as input and returns observations that are appended to the rollout context. The tool is treated as a black-box response mechanism: the model generates special tokens indicating a tool call, the tool executes and returns text, and the model conditions on this returned text for subsequent generation steps.
Reward Function ( $r_\phi$ ) — a composite reward that combines binary answer accuracy (exact match with ground truth) and format correctness (adherence to the required XML tag structure, with penalties for specific violations like missing <answer> tags or no use of <search>). This reward is computed per rollout trajectory and used by GRPO to compute group-relative advantages that drive policy updates.

Information flows as follows: a training query enters the system → the current curriculum stage determines how many few-shot examples ( $N$ ) are prepended to the query → the policy model samples $k = 8$ trajectory rollouts (each containing reasoning, tool calls, tool responses, and a final answer) → the external tool executes any search queries in real time, returning top-3 documents → the reward function scores each complete trajectory → GRPO computes advantages by normalizing rewards across the group of 8 rollouts → the policy is updated using only the model-generated tokens (tool responses are masked from the loss) → after $T$ steps at the current curriculum stage, the number of few-shot examples is reduced and training continues.

3.3 Roadmap for the Deep Dive

The explanation proceeds in five steps, ordered to build understanding from the abstract formalism to the concrete training loop:

First, the tool-use MDP formulation (Section 2.1): the formal definition of tool-augmented generation as a conditional sequence model with structured interaction. This establishes the notation and decision space that the RL objective operates over.
Second, the RL objective with tool-specific loss masking (Section 2.2): the GRPO-based policy gradient objective and the critical detail that tool-returned tokens are excluded from optimization. This explains what the model is optimizing and why standard RL cannot be applied naively.
Third, the ICRL curriculum and rollout template (Section 2.3): the core innovation—how few-shot demonstrations are embedded in rollout prompts and progressively removed. This explains how exploration is guided without SFT.
Fourth, the reward design (Section 2.3): the composite accuracy+format reward function and the specific format violation penalties. This explains what signal drives learning and how it balances task success with structural compliance.
Fifth, the training procedure end-to-end (Algorithm 1): the full training loop as pseudocode, including the curriculum schedule, rollout sampling, advantage computation, and policy updates. This ties all components together into a concrete algorithm.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an algorithm design paper whose core idea is that in-context demonstrations during RL rollouts can substitute for supervised fine-tuning as a mechanism for guiding exploration in tool-use tasks, and that a curriculum which gradually removes these demonstrations enables the model to transition from imitation-guided behavior to autonomous reward-driven tool use.

Tool Use as a Conditional Generation Process

The paper formalizes tool-augmented generation as a sequential decision process where the model interleaves internal reasoning, tool calls, and final answer generation. The key modeling choice is that tool interactions are embedded directly in the autoregressive text stream using structured XML tags, making the entire process—reasoning, querying, observing, answering—a single sequence generation problem.

Formally, given a query $q$ and an external tool $T$ , the model generates a response $y = (y_1, y_2, \dots, y_{|y|})$ where each token $y_t$ is conditioned on the query, all previously generated tokens, and the history of tool interactions up to that point. The conditional distribution is:

$\pi_\theta(y \mid q, T) = \prod_{t=1}^{|y|} \pi_\theta(y_t \mid y_{<t}, q, H_t)$

where $\pi_\theta$ is the policy model parameterized by $\theta$ , $y_{<t}$ represents all tokens generated before position $t$ , and $H_t$ denotes the sequence of prior actions and tool observations up to step $t$ .

What it computes: the probability of a complete tool-augmented response $y$ given a query $q$ and access to a tool $T$ . The product runs over all tokens in the response, where each token's probability depends on everything that came before it—the model's own reasoning, any search queries it issued, and the text returned by the tool in response to those queries. The history $H_t$ is the distinguishing element: it captures the fact that tool outputs become part of the conditioning context for future tokens, enabling the model to incorporate retrieved information into subsequent reasoning.

Why this form: this formulation reduces tool use to a standard autoregressive generation problem with augmented context. Alternative formulations could treat tool calls as separate, non-textual actions (e.g., API calls with structured parameters that the model emits as special tokens and receives as structured returns), but embedding everything in the text stream means the entire interaction—reasoning, deciding to search, formulating queries, reading results, deciding when to stop—is a single sequence that standard RL for language generation can optimize without architectural changes. The model never leaves the text modality; the tool is simply a function that inserts text into the context window at specific points.

The structured format uses four XML tag types to delineate different interaction modes:

<thinking> and </thinking>: enclose internal reasoning steps. The model writes its thought process here—what it knows, what it needs to find out, how it interprets retrieved information, and when it has enough to answer.
<search> and </search>: enclose a query string to be sent to the search engine. When the model generates a closing </search> tag, the training system intercepts this text, sends it to the Serper API, and inserts the results.
<information> and </information>: enclose the text returned by the tool. These tags are generated by the training system, not the model—they mark where external information enters the context stream.
<answer> and </answer>: enclose the final answer. The text between these tags is extracted and compared against the ground truth for computing the accuracy reward.

The tool $T$ is modeled as a deterministic response function $T: \mathcal{V}^* \rightarrow \mathcal{V}^*$ mapping a text query to a text observation. For search, $T(q')$ returns the top-3 documents retrieved by BM25 from a corpus, formatted with titles and snippets. For code execution, $T(q')$ would be a Python interpreter returning execution output. The key property is that $T$ is stochastic from the model's perspective (the model doesn't know exactly what will be returned) but deterministic from the training system's perspective (given the same query, the search engine returns the same results, though in practice live search results may vary).

The model has three types of actions it can take at each generation step, implicitly decided by what text it generates:

Internal reasoning: generating text inside <thinking> tags. No external interaction occurs; the model is processing information internally.
Tool invocation: generating text inside <search> tags (or equivalent for code execution). This triggers the training system to call the tool and insert results.
Answer generation: generating text inside <answer> tags. This signals that the model considers its reasoning complete and is providing a final response.

There is no explicit "stop searching" action separate from generating an answer—the model transitions from search to answer when it generates an <answer> tag, which implicitly terminates the search loop.

The RL Objective with Tool-Specific Loss Masking

Once tool-augmented generation is formulated as a sequential decision process, the reinforcement learning objective follows naturally. The model is trained to maximize expected reward over queries sampled from a training distribution, with a KL penalty to prevent the policy from diverging too far from a reference model (usually the initial instruction-tuned model before RL training):

$\max_{\pi_\theta} \mathbb{E}_{q \sim \mathcal{D}, y \sim \pi_\theta(\cdot \mid q, T)} \left[ r_\phi(q, y) \right] - \beta \cdot D_{\text{KL}} \left[ \pi_\theta(y \mid q, T) \parallel \pi_{\text{ref}}(y \mid q, T) \right]$

where $\pi_\theta$ is the policy being optimized, $\pi_{\text{ref}}$ is the reference policy (frozen copy of the initial model), $r_\phi$ is the reward function (detailed in the next section), $\beta$ is the KL penalty coefficient (set to 0.001), and $\mathcal{D}$ is the training distribution of queries.

What it computes: the expected reward (accuracy + format) that the model achieves when following its current policy $\pi_\theta$ , minus a penalty proportional to how different the current policy's output distribution is from the reference model's distribution. The expectation is over both the query distribution (what questions we care about) and the policy's own generation distribution (what answers the model actually produces when following its current parameters). Maximizing this objective means the model should learn to generate high-reward tool-use trajectories while staying close enough to its original instruction-following behavior that it doesn't catastrophically forget general language capabilities.

Why this form: the KL penalty is standard in RLHF and related methods to prevent reward hacking—the model could otherwise learn to produce nonsensical text that happens to match the ground truth answer format or exploit other reward function blind spots. The reference model provides an anchor: the policy is penalized for deviating too far from its pretrained/instruction-tuned behavior, which acts as a regularizer during early training when the reward signal is sparse. The coefficient $\beta = 0.001$ is small enough that the model can learn substantially new behavior (tool use) but large enough to prevent collapse into degenerate policies.

The paper adopts GRPO (Group Relative Policy Optimization) as the specific RL algorithm. GRPO is a variant of policy gradient methods that replaces the learned value function (critic) with a group-based baseline: for each query, the model samples a group of $N = 8$ complete trajectories, computes the reward for each, and uses the mean and standard deviation of rewards within the group to normalize advantages. This eliminates the need for a separate critic network, reducing memory and computational overhead.

The GRPO loss for a single query $q$ is:

$\mathcal{L}_{\text{GRPO}}(\theta) = \mathbb{E}_{\tau_i \sim \pi_{\theta_{\text{old}}}(q), q \sim \mathcal{D}_{\text{RL}}} \left[ \frac{1}{\sum_{i=1}^{N} |\tau_i|} \sum_{i=1}^{N} \sum_{t=1}^{|\tau_i|} \text{CLIP}(r_{i,t}(\theta), A_i, \epsilon) \right] - \beta \cdot D_{\text{KL}}[\pi_\theta \parallel \pi_{\text{ref}}]$

where:

$\tau_i$ is the $i$ -th trajectory (complete response) in the group of $N = 8$ trajectories sampled from the old policy $\pi_{\theta_{\text{old}}}$ for query $q$ .
$|\tau_i|$ is the number of tokens in trajectory $i$ .
$r_{i,t}(\theta) = \pi_\theta(\tau_{i,t} \mid q, \tau_{i,<t}) / \pi_{\theta_{\text{old}}}(\tau_{i,t} \mid q, \tau_{i,<t})$ is the per-token importance weight: the ratio of the new policy's probability of generating token $\tau_{i,t}$ to the old policy's probability. This measures how much more (or less) likely the current policy makes each token compared to when it was sampled.
$\text{CLIP}(r, A, \epsilon)$ is the standard PPO-style clipping function: $\min(r \cdot A, \text{clip}(r, 1-\epsilon, 1+\epsilon) \cdot A)$ with $\epsilon$ typically 0.2. This prevents the policy from updating too aggressively on tokens where the importance ratio has changed dramatically.
$A_i$ is the group-normalized advantage for trajectory $i$ , computed as:

$A_i = \frac{R(\tau_i) - \text{mean}(\{R(\tau_j) \mid j = 1, \dots, N\})}{\text{std}(\{R(\tau_j) \mid j = 1, \dots, N\})}$

where $R(\tau_i)$ is the scalar reward assigned to trajectory $i$ .

What the GRPO loss computes: for each query, the model samples 8 complete answers using its current (pre-update) parameters. Each answer gets a reward. The advantage for each answer is how much better or worse its reward is compared to the average of the 8 answers, measured in standard deviation units. Then, for each token in each answer, the loss encourages the model to increase the probability of tokens that appeared in better-than-average answers (positive advantage) and decrease the probability of tokens that appeared in worse-than-average answers (negative advantage), but clips the update if the probability ratio has changed too much (preventing destructive large updates). The KL penalty is subtracted to keep the overall policy close to the reference.

Why GRPO over alternatives: standard PPO requires training a separate value function (critic) to estimate advantages, which doubles memory usage and introduces a second optimization objective that can be unstable. GRPO replaces the critic with a group-based Monte Carlo baseline: by sampling 8 trajectories per query and normalizing within that group, the algorithm gets a low-variance advantage estimate without a learned critic. The cost is 8× more sampling per query, but this is parallelizable and avoids the complexity of critic training. For tool-use tasks specifically, the critic would need to evaluate partial trajectories mid-search, which is difficult because the value of a partial answer depends on what the search engine will return—a fundamentally hard prediction problem that GRPO sidesteps entirely by only evaluating complete trajectories.

A critical implementation detail: loss masking for tool-returned tokens. The trajectories $\tau_i$ contain both model-generated tokens and tool-returned tokens (the search results inside <information> tags). If the policy gradient included tool-returned tokens, the model would receive gradients for "generating" text it did not actually produce—the search engine generated it. This would create spurious learning signals: the model might learn to output tokens that happen to appear in search results (because those tokens are correlated with correct answers) rather than learning to issue effective search queries.

The paper's solution is loss masking: only tokens that the language model actually generates contribute to the policy gradient. Specifically, the sum over tokens in the GRPO objective:

$\sum_{t=1}^{|\tau_i|} \text{CLIP}(r_{i,t}(\theta), A_i, \epsilon)$

is computed only over positions $t$ where $\tau_{i,t}$ was generated by $\pi_{\theta_{\text{old}}}$ , not over positions where the token was inserted by the tool. In practice, this is implemented by setting the loss weight to zero for all tokens between <information> and </information> tags (and any other tool-returned spans). The importance ratio $r_{i,t}(\theta)$ is still computed for all tokens (since it's needed for KL computation), but gradients only flow through model-generated positions.

Why this is necessary: without loss masking, the model would try to "generate" the search results, which is both impossible (it has no control over what the search engine returns) and harmful (it would learn spurious correlations between search result tokens and rewards that don't reflect the model's actual decision-making). This is a direct consequence of embedding tool interactions in the text stream: the text contains two qualitatively different types of content—model decisions (reasoning, queries, answers) and environment responses (search results)—and the optimization must distinguish between them. In standard RL for text generation without tools, every token is model-generated, so this distinction doesn't arise.

The ICRL Curriculum: In-Context Demonstrations for Exploration

This is the paper's core algorithmic contribution. The central problem that ICRL solves is the exploration bottleneck in tool-use RL. When an instruction-tuned model without tool-use training is asked to "solve this problem by searching the web and reasoning step by step," it does not know:

What XML tags to use for search queries versus answers versus reasoning.
When to search versus when to reason from internal knowledge.
How to formulate effective search queries.
How to integrate retrieved information into its reasoning chain.
How many search turns are typically needed.

Random exploration in this space almost never produces a correct answer, meaning the model receives zero accuracy reward and the format reward alone is too weak to guide it toward the right behavior. The model might eventually stumble on a correct format through random chance, but the sample complexity would be prohibitive.

ICRL solves this by embedding few-shot demonstrations directly into the rollout prompts during RL training. These demonstrations show the model exactly what a successful tool-use trajectory looks like, but critically, the model is not trained to imitate them through SFT—it sees them only as context during generation, and it is trained through RL rewards on its own outputs.

At the start of training, every rollout prompt is constructed as:

[System instruction about how to use tools and tags]
[Example 1: full question → reasoning → search → results → reasoning → answer]
[Example 2: full question → reasoning → search → results → reasoning → answer]
[Example 3: full question → reasoning → search → results → reasoning → answer]
[Actual training question]

The model then generates its own response to the actual training question, conditioning on these three examples as context. The resulting policy is denoted:

$\pi_\theta(y \mid P_N, q, T) = \prod_{t=1}^{|y|} \pi_\theta(y_t \mid P_N, y_{<t}, q, H_t)$

where $P_N$ represents the prompt containing $N = 3$ demonstration examples (the paper starts with 3-shot). The demonstrations are generated by GPT-5.2: the authors randomly sample three questions and use GPT-5.2 to generate complete tool-use trajectories formatted according to the template in Table 1. These three examples are fixed throughout the entire training process at a given curriculum stage—they are not re-sampled or dynamically selected per query.

What this policy computes: the probability of generating a tool-use trajectory $y$ when the model sees $N$ examples of successful tool use in its context window. The examples serve as in-context demonstrations: the model's attention mechanism can attend to the example reasoning patterns, the example search query formulations, and the example answer formats while generating its own response. Because the model has strong instruction-following and in-context learning capabilities from its instruction tuning, it can generalize from these examples to produce similarly structured outputs for new queries.

Why this helps with exploration: the few-shot demonstrations reshape the model's initial output distribution. Without demonstrations, the model assigns high probability to all sorts of outputs—some with wrong formats, some with no search, some with nonsensical reasoning. With demonstrations, the model's probability mass concentrates on outputs that resemble the demonstrated patterns: using the right XML tags, including reasoning before and after search, formulating search queries as natural language questions, and providing answers in <answer> tags. This means that when the model samples 8 trajectories per query for GRPO, a non-trivial fraction of those trajectories will be well-formed enough to occasionally produce correct answers, generating positive accuracy rewards that drive further learning.

The progressive reduction curriculum. The defining feature of ICRL is that the number of demonstrations is not fixed. As training progresses, the authors periodically reduce $N$ :

Stage 1 (3-shot): Train with $P_3$ for $T$ steps. The model learns to produce structured tool-use outputs by conditioning on 3 examples.
Stage 2 (2-shot): Reduce to $P_2$ (two examples) and train for $T$ more steps. The model must now generate tool-use behavior with less explicit guidance, relying more on what it learned in Stage 1.
Stage 3 (0-shot): Remove all examples ( $P_0$ , only the system instruction) and train for a final $T$ steps. The model must generate tool-use behavior entirely autonomously, driven only by the reward signal and whatever tool-use capabilities it internalized from the earlier stages.

After each reduction, the updated policy conditioned on the reduced prompt is:

$\pi_\theta(y \mid P_{N-1}, q, T) = \prod_{t=1}^{|y|} \pi_\theta(y_t \mid P_{N-1}, y_{<t}, q, H_t)$

where $P_{N-1}$ contains one fewer demonstration than the previous stage.

What the curriculum achieves: Stage 1 provides the exploration guidance—the model sees examples and can produce well-structured outputs. The reward signal then reinforces which of these structured outputs actually lead to correct answers. Stage 2 reduces the "scaffolding," forcing the model to rely more on its own learned parameters rather than copying from the context. Stage 3 removes all scaffolding, requiring fully autonomous tool use. This is a scaffolding-and-fading approach: provide support early, gradually withdraw it as competence develops.

Why this form over alternatives: the paper considered two alternative curriculum designs, evaluated in the ablation (Section 4.1, Figure 2):

3→2→0 (three stages): the model moves from heavy prompting to moderate prompting directly to zero prompting, without an intermediate "minimal guidance" stage. This forces the model to learn enough in the 2-shot stage to function independently.
3→2→1→0 (four stages): the model has an additional intermediate stage with exactly 1 demonstration. This provides more gradual support but, as the results show, causes premature convergence to shallow strategies—the model learns to answer after 1-2 search turns even when more are needed, because the single demonstration implicitly suggests that one search is often sufficient.

The paper's choice of 3→2→0 over 3→2→1→0 is an empirical finding about curriculum pacing: too gradual a reduction teaches the model to depend on the minimal demonstration rather than learning deep multi-turn reasoning. The 3→2→0 curriculum forces a sharper transition from guided to autonomous behavior, which apparently encourages the model to internalize the reasoning patterns more thoroughly during the 2-shot stage in preparation for the 0-shot stage.

The system instruction remains constant. Throughout all curriculum stages, the first part of the prompt—the system instruction explaining the task and the XML tag format—stays the same:

"Solve the following problem step by step. You must conduct reasoning inside thinking... every time you get new information. After reasoning, if you find you lack some knowledge, you can call a search engine by <search> query </search> and it will return results between <information>...</information>. You can search as many times as you want. Finally, provide the answer inside <answer>...</answer>."

This instruction provides the declarative knowledge (what the tags are, what the model should do), while the few-shot examples provide the procedural knowledge (how to use them in practice). The curriculum removes the procedural examples but keeps the declarative instruction, so the model always knows what output format is expected.

The training data partitions matter. Algorithm 1 shows that different curriculum stages use different subsets of the training data: $\mathcal{D}^{(3)}, \mathcal{D}^{(2)}, \mathcal{D}^{(0)}$ . The paper does not specify whether these are disjoint or overlapping splits of the Natural Questions training set, but the notation $\mathcal{D}^{(N)}$ with superscripts indexed by the number of shots implies that each stage uses a designated subset. This prevents the model from simply memorizing answers to specific training questions that it saw in earlier stages with more demonstrations—each stage forces the model to apply its tool-use skills to (potentially) new questions with less guidance.

Reward Design: Balancing Accuracy and Format Compliance

The reward function $r_\phi(q, y)$ is the only learning signal the model receives. Unlike SFT, which provides token-level supervision (exactly what to generate at each step), the RL reward is computed once per complete trajectory and provides a scalar feedback. The paper designs a composite reward that combines two components:

$r_\phi(q, y) = \alpha \cdot \text{reward}_{\text{acc}} + (1 - \alpha) \cdot \text{reward}_{\text{format}}$

where $\alpha = 0.8$ balances the two terms.

Accuracy reward ( $\text{reward}_{\text{acc}}$ ): binary, 1.0 if the model's final answer (extracted from between <answer> and </answer> tags) exactly matches the ground truth answer string, and 0.0 otherwise. The comparison uses exact match (EM): no partial credit for near-correct answers, no lenient matching. For datasets like Natural Questions where answers are typically short entities (names, dates, numbers), exact match is a reasonable correctness signal. However, this means the accuracy reward is extremely sparse: most trajectories, especially early in training, receive 0.0 accuracy reward regardless of how close their reasoning was to being correct.

Format reward ( $\text{reward}_{\text{format}}$ ): computed as:

$\text{reward}_{\text{format}} = 1.0 - \sum_{v \in \mathcal{V}} \text{penalty}(v)$

where $\mathcal{V}$ is the set of format violations detected in the model's response, and $\text{penalty}(v)$ assigns a fixed cost to each specific violation type. The violations and their penalties (from Table 2) are:

Violation	Penalty	Rationale
No `<answer>` tag	0.5	Must provide structured answer
Unbalanced `<answer>` tags	0.2	Proper XML structure required (e.g., missing closing tag)
No `<thinking>` tag	0.15	Should demonstrate reasoning
Unbalanced `<thinking>` tags	0.1	Proper XML structure required
No `<search>` usage	0.1	Should utilize available tool
Empty answer content	0.2	Answer must be substantive (tags present but nothing between them)

What the composite reward computes: a weighted combination of "did the model get the right answer?" (80% weight) and "did the model follow the output format rules?" (20% weight). The accuracy term provides the primary learning signal—it creates selection pressure for trajectories that actually solve the problem. The format term provides a shaping reward—it gives the model positive feedback for at least producing well-structured outputs even when the answer is wrong, which guides early exploration before any accuracy rewards are achieved.

Why $\alpha = 0.8$ and these specific penalties: the paper does not provide an ablation on $\alpha$ , but the choice reflects a deliberate tradeoff. Setting $\alpha$ too high (e.g., 0.95) would mean the model receives almost no useful signal until it occasionally produces a correct answer, which could take prohibitively many training steps. Setting $\alpha$ too low (e.g., 0.5) would let the model optimize primarily for format compliance—producing correctly-tagged but content-free outputs—without sufficient pressure to actually answer questions correctly. The 0.8/0.2 split means that a perfectly formatted but wrong answer gets reward 0.2, while a correctly formatted and correct answer gets reward 1.0. The gap (0.8) is the accuracy bonus, which is large enough to drive learning toward correctness once the format is mastered.

The specific penalty values encode priorities among format requirements. The highest penalty (0.5) is for missing the <answer> tag entirely—a response without an answer is fundamentally useless regardless of reasoning quality. Unbalanced tags (0.2 for answer, 0.1 for thinking) penalize malformed XML that would break downstream parsing. Missing <thinking> (0.15) penalizes the model for skipping reasoning, which likely leads to poor answers. Missing <search> (0.1) is the lightest penalty because there exist some questions the model might answer correctly from internal knowledge without searching—the authors want to encourage search but not mandate it for every query.

Why composite rewards over accuracy-only: accuracy-only rewards create a classic sparse reward problem. In the early stages of training, before the model has learned any tool-use behavior, the probability of generating a trajectory that happens to contain the correct answer is near zero. With only binary accuracy feedback, the model would receive 0.0 reward for every trajectory and have no gradient signal—the policy gradient would be zero because all advantages would be zero (all trajectories equally bad). The format reward provides a dense shaping signal: even when the answer is wrong, the model can still receive partial credit for correctly using the XML tags. This nonzero reward creates variance in the group of 8 trajectories (some will have better format than others), which produces nonzero advantages that drive the policy gradient toward at least producing well-structured outputs. Once the model reliably produces well-structured outputs, some of those outputs will occasionally be correct, and the accuracy reward takes over as the primary optimization signal.

Training Procedure: The Full ICRL Algorithm

Algorithm 1 in the paper provides the complete pseudocode. The training loop operates as follows, combining the GRPO optimizer, the in-context demonstrations, the curriculum schedule, and the loss masking into a single procedure:

Inputs:

$\pi_\theta$ : initial policy (Qwen2.5-Instruct or Qwen3-Instruct, pre-trained and instruction-tuned but without tool-use training).
$\pi_{\text{ref}}$ : reference model (frozen copy of $\pi_\theta$ before any RL updates).
$T$ : external tool (Serper API for search; Python interpreter for code execution).
$P_N$ : initial few-shot prompt with $N = 3$ demonstrations generated by GPT-5.2.
$\mathcal{D}^{(N)}, \mathcal{D}^{(N-1)}, \dots, \mathcal{D}^{(0)}$ : training data partitions for each curriculum stage.
$r_\phi$ : composite reward function.
$T$ : number of RL steps per curriculum stage (the paper does not specify the exact value, but training curves in Figure 3 suggest hundreds to low thousands of steps—enough for the learning curves to stabilize at each stage).

Training loop (multiplied across curriculum stages $k = N, N-1, \dots, 0$ ):

For each curriculum stage $k$ (starting at $k = N = 3$ and ending at $k = 0$ ):

Construct the prompt $P_k$ : select $k$ demonstration examples from the original $P_N$ . The paper does not specify whether the selection is the first $k$ examples, a random $k$ , or a fixed subset. Given that the demonstrations are generated once by GPT-5.2 and fixed throughout training, it's likely that $P_2$ contains two of the original three demonstrations and $P_0$ contains only the system instruction with no examples.
Assign the training data partition $\mathcal{D}^{(k)}$ : each curriculum stage uses a designated subset of the Natural Questions training data. This prevents the model from seeing the same questions at different curriculum stages, which could create confounding effects (e.g., the model performs better at 0-shot on questions it previously trained on at 3-shot because it memorized them).
Inner RL loop: for $t = 1$ to $T$ steps, for each query $q \in \mathcal{D}^{(k)}$ :

a. Freeze the current policy as the old policy: $\pi_{\theta_{\text{old}}} \leftarrow \pi_\theta$ . This is the policy that will be used for sampling (on-policy data collection), and the importance ratios in the GRPO loss will compare the updated policy to this snapshot.

b. Sample $N = 8$ trajectories from the old policy: each trajectory $\tau_i$ is generated autoregressively by $\pi_{\theta_{\text{old}}}$ , conditioned on the prompt $P_k$ and the query $q$ . During generation:
- The model generates tokens until it produces an <answer> tag with content or hits the maximum response length (2048 tokens).
- When the model generates a <search>query</search> pair, the training system intercepts the query string, sends it to the Serper API, retrieves the top 3 documents, and inserts them into the generation stream wrapped in <information> tags. The model then continues generating from this augmented context.
- The model can issue up to 6 search queries per trajectory (the paper mentions "up to 6 search turns per query" as part of the implementation details). If the model exceeds this, it is presumably truncated.
- Sampling uses temperature 1.0, which is relatively high—this encourages diversity across the 8 trajectories, which is beneficial for exploration and for the group-based advantage normalization (more diverse rewards produce more informative advantages).
c. Compute rewards: for each trajectory $\tau_i$ :
- Extract the final answer: parse the text between the last <answer> and </answer> tags (or the first, depending on the implementation; the paper does not specify handling of multiple answer tags, but the format penalty penalizes unbalanced tags, encouraging exactly one answer block).
- Compute $\text{reward}_{\text{acc}}$ : 1.0 if the extracted answer exactly matches the ground truth, else 0.0.
- Compute $\text{reward}_{\text{format}}$ : 1.0 minus the sum of violation penalties from Table 2. Check for: presence of <answer> tag, balanced <answer> tags, presence of <thinking> tag, balanced <thinking> tags, presence of <search> usage, non-empty answer content.
- Combine: $r_\phi(q, \tau_i) = 0.8 \cdot \text{reward}_{\text{acc}} + 0.2 \cdot \text{reward}_{\text{format}}$ .
d. Compute normalized advantages: for each trajectory $\tau_i$ : $A_i = \frac{R(\tau_i) - \text{mean}(\{R(\tau_1), \dots, R(\tau_8)\})}{\text{std}(\{R(\tau_1), \dots, R(\tau_8)\})}$ This normalizes rewards within the group of 8 trajectories for the same query. A trajectory with reward above the group mean gets a positive advantage; below the mean gets a negative advantage. The magnitude is measured in standard deviations, so a trajectory that is one standard deviation above the mean gets advantage +1.0 regardless of the absolute reward scale.

e. Compute token-level importance ratios: for each position $t$ in each trajectory $\tau_i$ : $r_{i,t}(\theta) = \frac{\pi_\theta(\tau_{i,t} \mid q, \tau_{i,<t})}{\pi_{\theta_{\text{old}}}(\tau_{i,t} \mid q, \tau_{i,<t})}$ This ratio is 1.0 for tokens where the updated policy $\pi_\theta$ assigns the same probability as the old policy $\pi_{\theta_{\text{old}}}$ ; greater than 1.0 where the updated policy would make the token more likely; less than 1.0 where less likely.

f. Apply loss masking: for positions $t$ where $\tau_{i,t}$ was inserted by the tool (inside <information> tags), set the loss contribution to 0. This is typically done by multiplying the per-token loss by a mask $m_t \in \{0, 1\}$ where $m_t = 0$ for tool-generated tokens and $m_t = 1$ for model-generated tokens. The masked GRPO loss becomes: $\mathcal{L}_{\text{GRPO}}(\theta) = \frac{1}{\sum_i \sum_t m_t} \sum_{i=1}^8 \sum_{t=1}^{|\tau_i|} m_t \cdot \text{CLIP}(r_{i,t}(\theta), A_i, \epsilon) - \beta \cdot D_{\text{KL}}[\pi_\theta \parallel \pi_{\text{ref}}]$ The normalization $\sum_i \sum_t m_t$ is the total number of model-generated tokens across all 8 trajectories, ensuring the loss is on a per-token basis that is comparable across queries with different amounts of search.

g. Update the policy: $\pi_\theta \leftarrow \pi_\theta - \eta \cdot \nabla_\theta \mathcal{L}_{\text{GRPO}}$ , where $\eta$ is the learning rate (set to $1 \times 10^{-6}$ for all model sizes). The gradient is computed over the batch of 8 trajectories per query, accumulated across a batch of 64 queries (the paper mentions "batch size of 64"). With 8 trajectories per query, this means 512 total trajectories per gradient step.

h. The KL penalty is computed and subtracted as part of the loss. The coefficient $\beta = 0.001$ is small, so the KL term primarily prevents catastrophic divergence rather than strongly constraining the policy.
After $T$ steps at curriculum stage $k$ , the training loop moves to stage $k-1$ : the prompt is reduced (e.g., from 3-shot to 2-shot), a new data partition $\mathcal{D}^{(k-1)}$ is used, and the inner RL loop restarts from the current policy $\pi_\theta$ (not from scratch—the model retains all learning from previous stages).

Output: the trained policy $\pi_\theta$ , which can now generate tool-augmented reasoning chains in a zero-shot setting (no demonstrations in the prompt, only the system instruction).

Implementation details that matter for reproducibility:

Hardware: 4 NVIDIA A100 GPUs (80GB each), suggesting a total of 320GB GPU memory. The models (3B and 7B parameters) are loaded in bfloat16 precision, which uses 2 bytes per parameter—roughly 6GB for the 3B model and 14GB for the 7B model, leaving substantial memory for optimizer states, activations, and batch processing.
Memory optimization: Fully Sharded Data Parallel (FSDP) training with gradient checkpointing. FSDP shards model parameters, optimizer states, and gradients across GPUs, reducing per-GPU memory usage. Gradient checkpointing trades compute for memory by recomputing activations during the backward pass rather than storing them.
Prompt and response lengths: maximum prompt length is 5000 tokens (to accommodate the few-shot demonstrations, the system instruction, and the query), and maximum response length is 2048 tokens (allowing up to 6 search turns with reasoning and answers). If a trajectory exceeds 2048 tokens before producing an <answer>, it is truncated and likely receives no accuracy reward and a format penalty for missing the answer tag.
Training data: Natural Questions (NQ) dataset, loaded via FlashRAG with preprocessed question-answer pairs. The dataset contains real Google search queries paired with Wikipedia passages containing answers. The paper randomly samples questions for training (the exact number of training steps $T$ is not specified, but with a batch size of 64, each step processes 64 distinct queries, each with 8 trajectories).
Few-shot example generation: three questions are randomly sampled from "the web" (not from NQ to avoid data leakage?) and GPT-5.2 generates complete tool-use trajectories formatted according to the template. These three examples are the only human/model-generated demonstrations used in the entire training process—no labeled tool traces, no SFT data, no additional annotation.
Retrieval: BM25 retriever (a sparse, term-frequency-based retrieval method) that returns the top-3 documents per search query. The documents come from a corpus that is not specified in the paper but is presumably the standard Wikipedia dump used in many retrieval-augmented QA benchmarks. The choice of BM25 over dense retrieval (e.g., DPR) is pragmatic: BM25 requires no training and is fast, though it may miss semantically relevant documents that use different vocabulary. The Serper API is mentioned as the tool integrated for "live results from Google Search," suggesting that for evaluation, live Google Search results are used, but for training, a static BM25 index is used for reproducibility and speed.
Learning rate: $1 \times 10^{-6}$ for all model sizes. This is relatively low for RL fine-tuning, suggesting the authors prioritize stability over convergence speed—tool-use learning is fragile, and large updates could destroy the model's instruction-following capabilities.
KL penalty coefficient: $\beta = 0.001$ . This is standard for GRPO-based RL fine-tuning; it provides enough regularization to prevent reward hacking without overly constraining the policy from learning new behaviors.
Temperature for sampling: 1.0. This is the default for exploratory sampling—it produces diverse trajectories without being so high that outputs become nonsensical.
Number of trajectories per query: $N = 8$ . This balances advantage estimation quality (more trajectories = more reliable group statistics) with computational cost (each trajectory requires a full autoregressive generation through the LLM plus potentially multiple search engine calls).
Maximum search turns: 6 per trajectory. This prevents infinite search loops and caps the cost per trajectory. If the model attempts more than 6 searches, the generation is likely truncated.
Gradient clipping: the paper uses the CLIP function from GRPO with $\epsilon$ (not explicitly stated, but standard PPO uses $\epsilon = 0.2$ ), which clips importance ratios to $[1-\epsilon, 1+\epsilon]$ multiplied by the advantage. This prevents any single token update from being more than $\epsilon \cdot |A_i|$ in magnitude relative to the old policy.

Why this specific training procedure over alternatives:

Why 8 trajectories per query instead of more or fewer: 8 is a common choice in GRPO implementations (the original DeepSeekMath paper uses 64 for math reasoning, but tool-use trajectories are much longer due to search results, making larger groups memory-prohibitive). More trajectories would give better advantage estimates (lower variance in the group mean and standard deviation) but cost proportionally more compute and GPU memory. Fewer trajectories would be cheaper but produce noisier advantages, slowing learning.
Why batch size 64: this is a standard RL batch size that balances gradient noise with training speed. With 64 queries × 8 trajectories = 512 trajectories per step, and each trajectory up to 2048 tokens, each training step processes up to roughly 1 million tokens—substantial but feasible on 4 A100s with FSDP and gradient checkpointing.
Why maximum response length 2048 with up to 6 search turns: this gives the model room for multi-step reasoning without being excessively long. Each search turn includes a <search> query, the returned documents (which can be lengthy), reasoning in <thinking> tags, and potentially another search. Six turns at ~300 tokens per turn (including search results) is roughly 1800 tokens, leaving room for the final answer.
Why train on NQ and evaluate on other datasets: Natural Questions is a large, diverse QA dataset with real user queries and Wikipedia-sourced answers. Training on NQ and evaluating on TriviaQA, HotpotQA, 2Wiki, Musique, and Bamboogle tests out-of-domain generalization—the model must learn general tool-use strategies from NQ's relatively straightforward single-hop questions and apply them to more complex multi-hop reasoning tasks without ever seeing those task formats during training.

Summary of Design Choices and Their Justifications

In-context demonstrations instead of SFT: avoids the cost of annotating tool-use trajectories and the distribution mismatch between SFT-imitated behavior and RL-optimized behavior. The demonstrations shape the initial exploration distribution (making well-structured outputs more likely) without being learned as fixed parameters.
Progressive reduction curriculum (3→2→0 shots): forces the model to internalize tool-use patterns rather than depending on in-context copying. The specific choice of 3→2→0 over 3→2→1→0 was validated by ablation (Figure 2) showing that an intermediate 1-shot stage causes premature stopping of multi-turn reasoning—the model converges to shallow strategies when the scaffolding is removed too gradually.
GRPO with group-based advantage normalization: eliminates the need for a separate critic network, reducing memory and avoiding the challenge of training a value function that can evaluate partial tool-use trajectories (where value depends on future search results that the model cannot predict).
Loss masking for tool-returned tokens: prevents the model from receiving gradients for text it didn't generate, which would create spurious correlations between search result tokens and rewards. This is essential because tool responses are environment observations, not model actions.
Composite reward with 80% accuracy and 20% format: provides a dense shaping signal (format) that guides early exploration before the sparse accuracy reward becomes achievable. The specific penalty values in Table 2 prioritize answer presence (0.5) over structural correctness (0.1–0.2), reflecting that producing an answer—even a wrong one—is more useful than producing perfectly formatted empty output.
BM25 retrieval with top-3 documents: provides a fast, reproducible retrieval baseline. Live Google Search via the Serper API is used for final evaluation but would be too slow and non-reproducible for training iterations (results change over time).
Fixed few-shot demonstrations generated by GPT-5.2: uses a strong external model once to create demonstrations, then relies entirely on the student model's own learning. The demonstrations are not updated during training (they are static), making the approach "supervision-light" (one-time cost of generating 3 examples) rather than "supervision-free."
Instruction-tuned base models (Qwen2.5-Instruct) rather than base models: the paper explicitly states that instruct variants are chosen "due to their strong instruction-following capabilities, which enable faster and more stable convergence during RL training." Base models without instruction tuning would likely struggle even more with the initial exploration, as they would lack the ability to understand and follow the system instruction about XML tags and tool use.

4. Key Insights and Innovations

Innovation 1: The Prompt Is a Cheaper Supervisor Than the Weights

The dominant assumption in tool-use training—inherited from the broader RLHF and reasoning literature—is that a model needs to internalize successful behavior patterns before RL can optimize them effectively. This internalization happens through SFT: thousands of demonstration trajectories are learned as fixed parameter updates, baking in the "right way" to use tools before any reward-driven exploration begins. The cold-start pipeline (SFT → RL) is so standard that it appears in virtually every competitive tool-use training method the paper evaluates against: O2-Searcher, ReTool, and implicitly Search-R1 and ParallelSearch, which build on instruction-tuned models that have already internalized formatting conventions through their own SFT phases.

ICRL challenges this at the conceptual level by asking: what if the demonstrations don't need to be in the weights at all? What if they can live in the context window during RL exploration, providing exactly the same guidance but without the permanent commitment of SFT? This is not an implementation trick—it is a fundamentally different theory of where guidance should reside. The weight-based theory says: bake the knowledge into the model so it can't forget it. The prompt-based theory says: keep the knowledge external so the model can be weaned off it, and let RL decide what's worth retaining.

The consequences of this reframing are non-obvious and go beyond data efficiency. When demonstrations are in the weights (SFT), the model learns a distribution over trajectories that approximates the demonstrations. When those same demonstrations are in the prompt (ICRL), the model learns a conditional policy that produces structured output when cued by examples, but the parameters themselves encode only what RL reinforces—the decisions that actually lead to correct answers. This means ICRL's learned policy is qualitatively different from an SFT-initialized policy: it has not been trained to reproduce the specific reasoning patterns, search queries, or turn counts from the demonstrations, only to produce outputs that the reward function approves of. The demonstrations shape the support of the initial exploration distribution (making well-formed trajectories possible) but do not dictate the mode of the converged policy (which is determined by the reward signal).

This distinction explains a pattern in the results that would otherwise be puzzling. ICRL outperforms O2-Searcher (which uses cold-start SFT) on four of five datasets despite using the same base model and training data (Table 4). If SFT were merely providing initial guidance that RL could later overwrite, the two methods should converge to similar performance. The fact that ICRL achieves substantially higher final performance (+2.9 average EM, with +12.9 on TriviaQA) suggests that the SFT initialization actually constrained subsequent RL optimization—the model inherited biases from the SFT trajectories that RL could not fully overcome. This is a specific, empirically-supported argument against the cold-start paradigm that goes deeper than "SFT data is expensive."

The comparison is even sharper in the math domain (Table 7): ICRL slightly underperforms ReTool on AIME2024 (64.1% vs. 67.0%) but surpasses it on AIME2025 (51.7% vs. 49.3%). The datasets differ primarily in difficulty and recency (AIME2025 questions are novel), and the pattern suggests that ICRL's prompt-based guidance produces a policy that generalizes better to out-of-distribution problem structures than ReTool's SFT-based initialization. This is consistent with the "constraint" interpretation: SFT trains the model to imitate specific code-writing patterns from the training distribution, while ICRL lets RL discover patterns that work for the actual reward signal.

This is a fundamental reframing, not an incremental improvement. It changes the question from "how do we get enough high-quality SFT data?" to "how do we design prompts and curricula that guide exploration effectively?"—a question that opens up an entirely different design space centered on curriculum pacing, demonstration selection, and the timing of scaffold removal rather than data annotation pipelines.

Innovation 2: The Curriculum Is the Algorithm—and Its Pacing Determines Reasoning Depth

Most RL training recipes treat the prompt as fixed infrastructure: you design it once, you use it throughout training, and the learning dynamics come from the reward signal and the optimizer. ICRL's progressive reduction curriculum—starting with 3-shot demonstrations, dropping to 2-shot, then to 0-shot—rejects this separation. The curriculum is the learning mechanism, not just its container. What the model learns (deep multi-turn reasoning vs. shallow fast-answering) is determined less by the reward function and more by how quickly the scaffolding is removed.

This is a diagnostic contribution, not just a design choice. The ablation in Figure 2 provides the clearest evidence: a four-stage curriculum (3→2→1→0 shots) performs dramatically worse than a three-stage curriculum (3→2→0 shots), with TriviaQA accuracy dropping from 75.4 to 20.8 and 2Wiki from 53.6 to 26.8. Both curricula use the same total training compute, the same reward function, the same base model, and the same data. The only difference is whether there's an intermediate stage with exactly one demonstration.

What's happening is not just "too much scaffolding delays learning"—it's that the 1-shot stage teaches the model a qualitatively different strategy. With one demonstration still present, the model can continue to rely on in-context copying for the basic structure of its output (issue a search, read results, answer). It never faces the pressure to internalize how many search turns are typically needed or how to decide when to stop searching. The single demonstration implicitly communicates that a small number of turns (typically 1-2 in a single example) is sufficient, and the model converges to a policy that answers as quickly as possible. When the demonstration is finally removed in the 0-shot stage, the model has already settled into a local optimum of shallow reasoning, and the sparse reward signal is too weak to push it toward deeper multi-turn strategies—because on many questions, the shallow strategy sometimes works, providing just enough positive reward to reinforce it.

Figure 2b quantifies the behavioral consequence: the four-stage model finishes over 80% of queries within two search turns, while the three-stage model distributes its turns more broadly across 1-6 searches. The three-stage curriculum forces a sharper transition. When moving from 2-shot to 0-shot, the model must simultaneously generate its own structure (no examples to copy) and determine how many searches to perform. This joint pressure apparently encourages the model to learn a more robust heuristic for search depth—one that depends on the query's complexity rather than defaulting to a small fixed number.

Prior work on curricula in RL (e.g., domain randomization, progressive environment complexity) typically focuses on gradually increasing task difficulty. ICRL inverts this: it gradually decreases support, forcing the model to become more autonomous rather than more capable. This is a distinct curriculum paradigm—scaffold fading rather than difficulty scaling—and the paper's demonstration that the fading schedule determines reasoning depth is a conceptual contribution with implications beyond tool use. Any setting where models learn from a combination of in-context guidance and RL feedback (code generation, multi-agent coordination, long-horizon planning) likely faces the same sensitivity: remove the scaffolding too gradually, and the model never develops autonomous depth; remove it too quickly, and the model collapses before learning.

The three-stage vs. four-stage result elevates curriculum design from a hyperparameter optimization problem to a first-class research question. It implies that different tasks may need different fading schedules depending on the complexity of the behavior being internalized. Multi-hop reasoning, which requires maintaining state across multiple search-results cycles, apparently benefits from a sharp transition that forces the model to handle the full complexity at once rather than a gradual reduction that lets it settle into a shallow attractor.

This is a fundamental insight with broad applicability, not an incremental finding. It reframes ICRL's contribution from "here's a way to avoid SFT" to "the schedule at which you remove in-context guidance is a lever that controls what kind of reasoning the model learns—and getting it wrong doesn't just slow training, it changes the asymptotic behavior."

Innovation 3: Exploration Guidance and Policy Optimization Are Separable Axes—and the Cold-Start Pipeline Conflates Them

The standard SFT+RL pipeline treats initialization and optimization as sequential phases: first define the policy's starting point (SFT), then optimize from there (RL). This sequential structure conflates two conceptually distinct functions:

Exploration guidance: making the initial policy's output distribution cover useful regions of the action space (well-formed tool calls, reasonable search queries, structured reasoning).
Objective-driven refinement: maximizing the actual task reward (answer accuracy) by adjusting which of those well-formed behaviors the model prefers.

SFT does both at once—it provides exploration guidance (the model starts with reasonable tool-use behavior) and shapes the model's preferences (it learns to imitate specific trajectories, which implicitly encodes preferences over query formulations, turn counts, and reasoning styles). When RL begins, it can only adjust preferences within the neighborhood established by SFT; the exploration guidance and the preference shaping are entangled in a way that depends on the specific SFT demonstrations used.

ICRL separates these functions cleanly. The few-shot demonstrations provide pure exploration guidance: they make well-structured outputs probable during sampling, but they never enter the training objective. The RL reward provides pure preference shaping: it determines which of the explored behaviors are reinforced, without any bias toward imitating the demonstrations per se. The demonstrations could be suboptimal (using unnecessarily verbose reasoning, searching for information the model already knows), and the RL phase can correct this because the demonstrations only influenced what was sampled, not what was rewarded.

This separation has practical consequences that the paper's results support. In Table 3, ICRL outperforms Rejection Sampling (which uses SFT-trained models to generate positive examples) by 15.6 average EM on Qwen2.5-3B (40.16 vs. 24.60) and by 15.2 on Qwen2.5-7B (49.12 vs. 33.94). Rejection Sampling is essentially an attempt to improve the SFT data quality—use the best outputs from an SFT-trained model as additional training examples. But it's still operating in the conflation regime: the SFT phase establishes both exploration preferences and task preferences, and no amount of filtering can separate them. ICRL's larger improvement over baselines that use SFT (O2-Searcher, Search-R1, ParallelSearch) compared to baselines that don't (Direct, CoT) suggests that the separation of exploration and optimization is not just data-efficient—it produces better final policies.

The conceptual move here is subtle but important. Prior work implicitly assumed that exploration guidance is a property of the model's parameters (via SFT) or not provided at all (RL from scratch). ICRL demonstrates that exploration guidance is a property of the generation context and can be manipulated independently of the model's learned policy. This opens a design dimension that the cold-start paradigm made invisible: you can change how the model explores without changing what it has learned, simply by modifying the prompt during rollout. The training prompt and the inference prompt can be different; the curriculum exploits this by making the rollout prompt a moving target that gradually converges to the inference prompt.

This is a reframing contribution rather than a metric gain. It doesn't propose a new algorithm so much as a new decomposition of the learning problem that suggests alternative architectures for tool-use training (e.g., dynamically selecting demonstrations based on query difficulty, using different demonstrations for different exploration phases, combining multiple demonstration sets to cover diverse tool-use patterns—all without SFT). The paper's results validate that this decomposition is productively exploitable, but the broader implication is that any training pipeline that separates exploration context from policy parameters is worth investigating, not just for tool use but for any RL task with a large combinatorial action space.

Innovation 4: Verifier-Free Multi-Turn Tool Use Through Outcome-Only Rewards

A quiet but significant aspect of ICRL is what it does not require: step-level supervision, process rewards, or any intermediate feedback on the quality of search queries or reasoning steps. The reward function is purely outcome-based—did the final answer match the ground truth?—plus format compliance. There is no critic evaluating whether a search query was "good," no value function estimating the expected future reward from a partial reasoning chain, and no dense reward shaping based on retrieval quality or reasoning coherence.

This is notable because multi-turn tool use presents a challenging credit assignment problem. When a model executes three searches before answering, and the answer is wrong, which search query was at fault? Was the first query poorly formulated? Did the model fail to read the second set of results carefully? Should it have searched a fourth time? Outcome-only rewards provide no direct signal about these intermediate decisions—the model must learn, purely from the correlation between its actions and final outcomes, how to formulate queries and when to stop searching.

The fact that ICRL succeeds with such sparse rewards (and actually increases its number of valid tool calls during the 0-shot training phase, per Figure 3) suggests something counterintuitive: the exploration guidance from in-context demonstrations is sufficient to make outcome-only learning work, even for multi-step tasks with combinatorial action spaces. The demonstrations don't just teach the format—they initialize the model's credit assignment pathways by showing it what kinds of actions tend to precede correct answers. The model doesn't need to discover from scratch that "searching for the president's name, then searching for their inauguration date" is a good decomposition of the question in Table 5; the demonstrations show that pattern, and RL reinforces it when it leads to correct answers.

This contrasts with approaches like Search-R1 and ReTool, which typically use process-based rewards or intermediate supervision to handle the credit assignment problem. The fact that ICRL matches or exceeds these methods without any intermediate rewards is evidence that the exploration guidance function of demonstrations is more powerful than previously recognized—it doesn't just help with format, it helps with the structure of multi-step reasoning by providing templates for how to decompose problems and sequence tool calls.

The implication for future work is that investment in better exploration guidance (smarter demonstration selection, query-adaptive prompting, demonstration diversity) may yield higher returns than investment in denser reward signals (process reward models, step-level verification). This inverts the prevailing intuition in the RL-for-reasoning literature, which has focused heavily on reward design (process rewards in DeepSeek-R1, outcome verification in STaR/ReST) as the primary lever for improving multi-step reasoning. ICRL suggests that for tool use specifically—where the action space includes structured interactions with an external environment—the exploration distribution is the binding constraint, not the reward signal.

This is an empirical finding with conceptual implications, not a theoretical advance. It establishes that outcome-only rewards are viable for multi-turn tool use provided the exploration distribution is adequately shaped, which expands the design space for tool-use training systems (you don't need to build a process reward model if you can design good rollout prompts). But it also raises a question the paper doesn't answer: what are the limits? At what task complexity does outcome-only credit assignment break down even with good exploration guidance? The hardest benchmarks (AIME math problems with code execution) show ICRL roughly matching SFT+RL baselines rather than substantially exceeding them, hinting that for sufficiently complex reasoning, the credit assignment problem may eventually require denser signals regardless of exploration quality.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary training corpus is the Natural Questions (NQ) dataset (Kwiatkowski et al., 2019), loaded via FlashRAG (Jin et al., 2025b) with preprocessed question-answer pairs and gold-standard answers. NQ contains real user queries from Google Search, each paired with Wikipedia passages containing the correct answer. For evaluation, the paper uses five QA benchmarks: TriviaQA (Joshi et al., 2017), HotpotQA (Yang et al., 2018), 2Wiki (Ho et al., 2020), Musique (Trivedi et al., 2022), and Bamboogle (Press et al., 2023). Up to 500 questions are randomly sampled from each benchmark for evaluation efficiency. Importantly, NQ is excluded from evaluation to avoid data leakage, and the selected benchmarks span both in-domain general QA tasks (TriviaQA, HotpotQA) and out-of-domain multi-hop QA tasks (2Wiki, Musique, Bamboogle) to test generalization. For math reasoning experiments, the models are evaluated on AIME2024 and AIME2025.
Base models. The primary experiments use the Qwen2.5 model family (Yang et al., 2024a), specifically Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and Qwen2.5-14B-Instruct. Additional experiments use Qwen3-8B (Yang et al., 2025) for the math reasoning generalization tests. The instruct variants are chosen over base models "due to their strong instruction-following capabilities, which enable faster and more stable convergence during RL training." All models are loaded in bfloat16 precision. The authors argue that Qwen2.5 models are "widely adopted for question answering and reasoning tasks" and representative of current instruction-tuned model capabilities.
Metrics. The primary metric is exact match (EM) accuracy (%), computed by comparing the model's predicted answer (extracted from between <answer> and </answer> tags) against the ground truth answer using exact string matching. For the five QA benchmarks, results are reported per-dataset and as an average EM across all five datasets. For math reasoning (AIME), accuracy is reported per-benchmark. The paper does not report confidence intervals, statistical significance tests, or standard deviations for any result.
Baselines. The paper compares ICRL against methods in three categories. Direct prompting methods: Direct (standard prompting), CoT (Chain-of-Thought prompting; Wei et al., 2022). Retrieval-based methods: IRCoT (Interleaving Retrieval Chain-of-Thought; Trivedi et al., 2023), Search-o1 (Li et al., 2025a), RAG (Retrieval-Augmented Generation; Lewis et al., 2020). Fine-tuning and RL methods: SFT (supervised fine-tuning; Chung et al., 2024), R1-base and R1-instruct (RL without search; Guo et al., 2025), Reject Sampling (Ahn et al., 2024), Search-R1 (Jin et al., 2025a), ZeroSearch (Sun et al., 2025), ParallelSearch (Zhao et al., 2025), and O2-Searcher (Mei et al., 2025). For math reasoning, the baseline is ReTool (Feng et al., 2025). Note that several baselines (O2-Searcher, ReTool) use cold-start SFT before RL, while ICRL uses no SFT. Some baselines appear only for specific model sizes (e.g., ParallelSearch and R1-base only for Qwen2.5-7B; R1-instruct appears for both 3B and 7B but with different configurations).
Generation budget / compute accounting. The paper does not report generation budgets (number of rollouts, total tokens generated, or FLOPs) for either training or inference. There is no direct comparison of computational cost between ICRL and baselines—the comparison is purely accuracy-based. However, the paper controls for tool access uniformly: "for fairness, each query retrieves the top 3 documents required for search-based reasoning" and the Serper API is integrated across all models. For training, each query generates 8 rollout trajectories (temperature 1.0) per GRPO step with a batch size of 64, producing 512 trajectories per gradient step. The maximum response length is 2048 tokens, allowing up to 6 search turns per trajectory. Training is conducted on 4 NVIDIA A100 GPUs (80GB each), but total training time, number of steps, or FLOPs are not reported.
Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported. The paper does not describe any held-out validation procedure within the training data, any multiple-seed training runs, or any variance estimates for the main results. Results in Tables 3, 4, 6, and 7 are reported as single-number EM accuracy values for each dataset with no error bars, confidence intervals, or standard deviations. The difficulty of reproducing RL training runs (cost, variance) makes this a notable omission—without multiple seeds, it is impossible to assess whether ICRL's advantages over baselines are statistically reliable or within the noise of training stochasticity.

Main Quantitative Results

ICRL Achieves State-of-the-Art Performance Across QA Benchmarks Without SFT

Table 3 presents the central result. On Qwen2.5-3B, ICRL achieves an average EM of 40.16 across the five QA datasets, outperforming the strongest baseline (Search-R1, 31.10) by +8.94 absolute percentage points. The gains are concentrated on multi-hop reasoning datasets: +7.3 on 2Wiki (39.2 vs. 31.9 for Search-R1), +9.7 on Musique (20.0 vs. 10.3), +7.2 on Bamboogle (33.6 vs. 26.4). On single-hop datasets, gains are smaller relative to the strongest retrieval-based baselines: TriviaQA at 72.6 (vs. 57.4 for ZeroSearch, +15.2) and HotpotQA at 35.4 (vs. 32.4 for Search-R1, +3.0).

On Qwen2.5-7B, ICRL achieves an average EM of 49.12, outperforming the strongest baseline (ParallelSearch, 41.78) by +7.34. ICRL achieves the best result on four of five datasets: TriviaQA (75.4 vs. 65.2 for ZeroSearch, +10.2), 2Wiki (53.6 vs. 42.4 for ParallelSearch, +11.2), Musique (26.0 vs. 19.7 for ParallelSearch, +6.3), and Bamboogle (48.0 vs. 41.1 for ParallelSearch, +6.9). On HotpotQA, ICRL (42.6) is essentially tied with ParallelSearch (42.9), with a difference of −0.3.

A notable pattern in Table 3 is the relative weakness of methods that lack either retrieval or RL. Direct prompting achieves 14.50 (3B) and 19.84 (7B) average EM; CoT is worse at 1.52 (3B) and 12.84 (7B). Standard RAG (23.04 on 3B, 27.70 on 7B) underperforms Search-R1 and ICRL by large margins, suggesting that fixed retrieval pipelines without learned search behavior are insufficient for multi-hop QA. Methods that use RL without search demonstrations (R1-instruct: 23.68 on 3B, 28.62 on 7B) substantially underperform search-augmented RL methods (Search-R1: 31.10 on 3B, 38.16 on 7B), confirming that tool augmentation and RL are complementary.

ICRL Outperforms Cold-Start SFT Methods Despite Using No Supervised Data

Table 4 provides the direct comparison between ICRL and O2-Searcher on Qwen2.5-3B. O2-Searcher uses cold-start SFT before RL (marked with ✓ in the SFT column); ICRL uses no SFT (marked ✗). Despite this, ICRL achieves a higher average EM of 40.16 vs. 37.26 (+2.9 advantage). ICRL outperforms O2-Searcher on four of five datasets: TriviaQA (72.6 vs. 59.7, +12.9), 2Wiki (39.2 vs. 37.4, +1.8), Musique (20.0 vs. 16.0, +4.0), and Bamboogle (33.6 vs. 34.4, −0.8). On HotpotQA, O2-Searcher slightly outperforms ICRL (38.8 vs. 35.4, −3.4).

This comparison is the most direct evidence for the paper's central claim—that in-context demonstrations during RL rollouts can replace an entire SFT phase while achieving better final performance. However, note that O2-Searcher's results are reported only for Qwen2.5-3B, not for 7B or 14B, which limits the generality of the comparison.

ICRL Scales to Larger Models Without Degradation

Table 6 reports results for Qwen2.5-14B with ICRL compared to Direct and CoT baselines (no other RL-based baselines are reported at this scale). ICRL achieves an average EM of 51.84, surpassing CoT (31.16) by +20.7 and Direct (24.80) by +27.0. Individual dataset results: TriviaQA 75.0, HotpotQA 43.2, 2Wiki 61.8, Musique 25.6, Bamboogle 53.6.

The results show continued improvement from 7B to 14B, consistent with the scaling trend from 3B to 7B. However, the gap between 7B (49.12) and 14B (51.84) is relatively small (+2.72 average EM), which could indicate diminishing returns, insufficient training optimization for the larger model, or saturation on certain benchmarks (TriviaQA is essentially identical: 75.4 at 7B vs. 75.0 at 14B; gains come primarily from 2Wiki: 53.6 → 61.8).

ICRL Generalizes to Code Execution for Math Reasoning

Table 7 reports results on AIME2024 and AIME2025 using Qwen3-8B, comparing ICRL (no SFT) against ReTool (SFT+RL pipeline). On AIME2024, ICRL achieves 64.1% vs. ReTool's 67.0% (−2.9). On AIME2025, ICRL achieves 51.7% vs. ReTool's 49.3% (+2.4).

The crossed pattern—ICRL underperforming on AIME2024 but outperforming on AIME2025—suggests that the SFT-based approach may have an advantage on problems closer to its training distribution (AIME2024 is older and more likely to have representation in training data), while ICRL's prompt-guided exploration produces a policy that generalizes better to novel problem structures (AIME2025 questions are more recent and less likely to appear in any training data). However, this interpretation is speculative without knowing what training data ReTool used.

Training Dynamics Show Progressive Internalization of Tool-Use Behavior

Figure 3 presents training curves for the Qwen2.5-7B model across the three curriculum stages (3-shot, 2-shot, 0-shot), tracking three metrics: response length, reward, and number of valid search calls.

Response length (Figure 3a): During the 3-shot and 2-shot stages, response length is "relatively stable," reflecting consistent output structure guided by the demonstrations. When transitioning to 0-shot, "the response length initially drops due to the removal of in-context examples but gradually increases again, indicating that the model is learning to independently compose longer and more structured outputs."

Reward (Figure 3b): The reward "remains relatively steady throughout training." This is notable because the reward is sparse (binary accuracy + format penalties)—the fact that it doesn't sharply increase or decrease suggests that the model maintains format compliance throughout the curriculum, and accuracy improvements are gradual enough not to create sharp reward transitions. The paper does not show separate accuracy and format reward curves, which would clarify whether the steady reward masks compensating changes (e.g., format improving while accuracy plateaus, or vice versa).

Number of valid search calls (Figure 3c): This is the most informative metric for the paper's claims about learning. During the 0-shot phase, "the number of valid tool calls increases." The paper interprets this as evidence that "ICRL successfully encourages the model to internalize tool-use behavior, even without dense or step-level supervision." The increase in search calls during 0-shot training means the model is learning that more searching leads to better answers (higher accuracy reward), contradicting what would happen if the model were simply optimizing for format compliance (which penalizes missing search usage with only 0.1 and would be satisfied by a single search).

The paper does not provide corresponding training curves for the 3B model, the 14B model, or for the math reasoning experiments. Without these, it is unclear whether the observed dynamics (response length dip at 0-shot transition, increasing valid search calls) are consistent across scales and domains or specific to the 7B web search setting.

Ablation Studies and Robustness Checks

Curriculum design (3→2→0 vs. 3→2→1→0): Figure 2 compares two curricula for reducing rollout demonstrations using Qwen2.5-7B. The three-stage schedule (3→2→0) substantially outperforms the four-stage schedule (3→2→1→0) across all five QA datasets. On TriviaQA: 75.4 vs. 20.8 (−54.6). On 2Wiki: 53.6 vs. 26.8 (−26.8). On Musique: 26.0 vs. 9.0 (−17.0). On Bamboogle: 48.0 vs. 14.4 (−33.6). On HotpotQA: 42.6 vs. 17.8 (−24.8). Figure 2b explains the mechanism: the four-stage curriculum causes premature stopping—over 80% of queries finish within two search turns, while the three-stage model distributes its search turns across 1–6, with a substantial fraction using 3+ searches. The paper interprets this as the intermediate 1-shot stage encouraging "premature stopping and weakening multi-turn reasoning," and the simpler 3→2→0 curriculum "maintains stronger performance by allowing the model to explore longer reasoning paths during training."

This is the paper's only reported ablation study. Several important ablations are missing:

No ablation on the number of initial demonstrations (e.g., 2→1→0, 4→3→2→0). The paper only compares 3→2→0 vs. 3→2→1→0, leaving open whether 3 initial shots is optimal.
No ablation on α (the accuracy-format balance) in the composite reward (Equation 7). The chosen value of 0.8 is not compared against alternatives (e.g., 0.5, 0.95, 1.0).
No ablation on the format penalty values (Table 2). The specific penalties (0.5 for missing answer tag, 0.2 for unbalanced answer tags, etc.) are not justified or compared against alternatives.
No ablation on the number of trajectories per query (N=8 for GRPO advantage normalization). It is unclear whether more trajectories (improved advantage estimation) or fewer (faster training) would change results.
No ablation on KL penalty coefficient β (fixed at 0.001).
No ablation on loss masking—the paper does not compare training with vs. without masking of tool-returned tokens, which would validate the claim that loss masking is essential.
No ablation on demonstration quality or source—the 3 demonstrations are generated once by GPT-5.2 and fixed. Would human-written demonstrations, demonstrations sampled from a different model, or dynamically selected demonstrations change results?

Generalization across tool domains (web search → code execution): Table 7 serves as a cross-domain generalization test, showing that ICRL works for code execution in math reasoning, not just web search for QA. However, this is not a controlled ablation—it uses a different base model (Qwen3-8B vs. Qwen2.5-7B), different training data (math problems vs. NQ), and different tool (Python interpreter vs. search engine). It demonstrates that the framework can generalize, but it does not isolate which components (curriculum, composite reward, loss masking) are necessary for that generalization.

Model scaling: Tables 3, 6, and the training dynamics (Figure 3, noted as "Qwen-7B" despite the x-axis not showing model size) provide evidence that ICRL scales from 3B to 7B to 14B, with average EM improving from 40.16 → 49.12 → 51.84 (though the 7B→14B gain is modest). However, no results are reported for base model sizes beyond 14B, and the 14B results lack the strong RL baselines (Search-R1, ParallelSearch, ZeroSearch) that appear for 7B, making it impossible to assess whether ICRL's relative advantage over baselines persists or diminishes at larger scales.

The paper contains no negative results in the traditional sense—no experiment where ICRL failed, no configuration that produced worse-than-baseline performance (except implicitly: the four-stage curriculum performs worse than three-stage, and direct RL without demonstrations would presumably fail, though this is stated as motivation rather than demonstrated experimentally). The ReST^EM negative result from the reference example paper (where RL-based revision training degraded performance) has no analog here. The absence of reported failure modes makes it difficult to assess the robustness of ICRL to implementation choices.

Critical Assessment

The experiments in this paper demonstrate that ICRL can train effective tool-use policies without SFT, but the strength of evidence for the paper's broader claims varies substantially across different claims and is limited by several experimental design choices.

Claim: ICRL achieves state-of-the-art performance and outperforms all baselines by large margins (up to +8.94 on 3B, +7.34 on 7B).

This claim is supported with significant caveats about baseline completeness and fairness. The +8.94 improvement on Qwen2.5-3B (Table 3) is measured against Search-R1 as the strongest baseline. However, several baselines are missing: ParallelSearch is reported only for 7B, not 3B; O2-Searcher is reported only for 3B, not 7B or 14B; ReTool appears only for math reasoning, not QA. This incomplete matrix means the "strongest baseline" varies by model size, and we cannot assess whether ICRL would outperform all baselines if they were all evaluated at the same scale.

More concerning is the missing compute-matched comparison. The paper provides no information about the relative computational cost of ICRL versus baselines. ICRL's training procedure requires generating 8 trajectories per query per GRPO step, each potentially containing up to 6 search engine calls and up to 2048 output tokens. If Search-R1 or ParallelSearch used fewer trajectories, shorter rollouts, or fewer search calls during training, then ICRL's accuracy advantage might simply reflect more training computation. Without a FLOPs-matched or wall-clock-time-matched comparison, the claim of "superior performance" conflates algorithmic efficiency with computational budget.

Additionally, the lack of error bars or multiple seeds is a significant weakness for RL results. GRPO training is known to be sensitive to random seeds, batch composition, and training dynamics. The main results report single-number EM accuracies with no variance estimates. It is possible that running ICRL with a different random seed would produce an average EM of 38 instead of 40.16 (3B) or 46 instead of 49.12 (7B), which would dramatically change the comparison to baselines. Given that the claimed improvements (+7.34 to +8.94) are moderate in absolute terms and some individual dataset margins are small (e.g., HotpotQA on 7B: ICRL 42.6 vs. ParallelSearch 42.9, essentially tied), statistical noise could account for some of the reported advantage.

Claim: ICRL eliminates the need for SFT and labeled data while matching or exceeding SFT+RL methods.

This claim is supported for the specific comparison to O2-Searcher on 3B (Table 4), where ICRL achieves 40.16 vs. 37.26 average EM without SFT. However, the comparison is limited to a single model size (3B) and a single SFT+RL baseline. The paper does not compare ICRL to a version of Search-R1, ParallelSearch, or ZeroSearch that uses SFT initialization (these baselines may or may not use SFT—the paper does not specify whether they follow the cold-start paradigm, though Search-R1 is described in the text as using RL without search, suggesting it does not require tool-use SFT). The math reasoning comparison (Table 7) shows ICRL slightly underperforming ReTool on AIME2024 (64.1% vs. 67.0%), which weakens the universality of the "matching or exceeding" claim—it holds on one benchmark (AIME2025) but not the other.

The paper also does not quantify the cost of the SFT data it avoids. How many supervised trajectories does O2-Searcher use? How many does ReTool use? If these methods use only a few hundred demonstrations, then the claim of "eliminating the need for labeled data" is less practically significant (generating 3 demonstrations with GPT-5.2 is cheap, but so is generating a few hundred). If they use tens of thousands, the efficiency gain is substantial. Without these numbers, the reader cannot assess the magnitude of the data efficiency advantage.

Claim: The curriculum design (specifically 3→2→0 over 3→2→1→0) is crucial for learning deep multi-turn reasoning.

This claim is strongly supported by the ablation in Figure 2. The performance difference between the two curricula is dramatic—75.4 vs. 20.8 on TriviaQA is not a marginal improvement; it is a qualitative difference in model behavior. Figure 2b provides a clear mechanistic explanation: the four-stage model learns to stop searching prematurely. This ablation is the cleanest and most convincing result in the paper.

However, the ablation raises a question the paper does not address: is 3→2→0 optimal, or would an even sharper transition (3→0) work better? What about 4→2→0? The paper compares only two schedules, both starting from 3-shot. Without exploring a broader space of curricula, the claim that "the three-stage curriculum is best" overstates what the experiment actually demonstrates—which is only that the four-stage curriculum with an intermediate 1-shot step is worse than skipping it. The more general insight (curriculum pacing determines reasoning depth) is supported, but the specific recommendation (use 3→2→0) is based on a single comparison.

Claim: ICRL generalizes across tool domains (web search and code execution).

This claim is supported with thin evidence. Table 7 shows ICRL working for code execution on math problems with Qwen3-8B, but this is a single cross-domain experiment using a different base model family. There is no ablation showing that the in-context curriculum mechanism is what enables cross-domain transfer—it could be that Qwen3-8B's base capabilities are stronger and would perform well with any RL approach. A proper demonstration of cross-domain generalization would train ICRL on web search and evaluate on code execution (or vice versa), showing that the learned tool-use meta-skill transfers. The current experiment trains ICRL separately on math problems with code execution demonstrations—the same framework is applied, but no transfer is demonstrated.

Missing experiments that would substantially strengthen the paper:

FLOPs-matched or generation-budget-matched comparison to baselines. Without controlling for computational cost, accuracy comparisons are incomplete.
Multiple training seeds for ICRL and at least the strongest baseline to assess variance. RL training is noisy; single-number results without error bars are difficult to interpret.
Ablation on the number of initial demonstrations (2-shot, 4-shot, 5-shot start). The paper only explores 3→2→0 vs. 3→2→1→0.
Ablation on α (accuracy-format balance). The choice of 0.8 is unexamined; if α=1.0 works nearly as well, the format reward would be unnecessary, which would change the interpretation of why ICRL works.
Ablation on loss masking. The paper claims this is critical for tool-use RL, but never demonstrates what happens without it.
Direct comparison to an "ICRL but with SFT instead of in-context demonstrations" condition. This would isolate the effect of the curriculum vs. having the same number of demonstrations baked into the model weights via SFT.
Evaluation on a benchmark that requires more than 6 search turns, to test whether the maximum-turn cap artificially constrains multi-hop reasoning and whether the 3→2→0 model continues searching deeper if allowed.
Latency or wall-clock time measurements. The 3→2→0 model in Figure 2b uses more search turns than the 4-stage model, which likely means higher latency per query. Is the accuracy improvement worth the additional inference time? Without this, the claim of "superior performance" ignores the latency-accuracy tradeoff.

Conditions under which the claims hold:

The claim that ICRL outperforms baselines holds for web search QA on the five evaluated benchmarks with Qwen2.5-Instruct models. It has not been demonstrated for other model families, other QA benchmarks, or other tool types (except the single code execution experiment).
The claim that the curriculum eliminates the need for SFT holds for the Natural Questions training distribution, which contains real user queries with Wikipedia-sourced answers. It is unclear whether the approach would work for training data that requires more complex tool interactions (e.g., tool chains, API calls with structured parameters, database queries) that might be harder to demonstrate in 3 fixed examples.
The claim that the 3→2→0 curriculum is superior to the 4-stage alternative holds when using 3 initial demonstrations and evaluating on multi-hop QA. It is unknown whether the finding generalizes to different numbers of initial demonstrations or different task types.
All performance claims are point estimates without variance information, meaning we cannot assess whether observed differences are statistically significant or within the noise of RL training stochasticity. This is the most important caveat for interpreting the main results table.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Unaccounted For, Making the Headline Efficiency Gains an Upper Bound

The assumption or constraint. The ICRL framework depends fundamentally on the ability to construct rollout prompts containing few-shot demonstrations of successful tool use. These demonstrations are generated once by GPT-5.2—three complete tool-use trajectories formatted according to the template in Table 1. The paper acknowledges this dependency implicitly through its methodology (Section 2.3: "we randomly sampled three questions from the web and used GPT-5.2 to generate few-shot examples") but does not frame it as a cost or limitation. More critically, the paper provides no analysis of what happens if these demonstrations are suboptimal, unrepresentative, or unavailable. A practitioner deploying ICRL to a new tool domain (e.g., a proprietary API, a domain-specific database, a multi-agent coordination setting) would need to either generate high-quality demonstrations themselves (requiring access to a strong model like GPT-5.2 and expertise in the target tool) or accept degraded performance from weaker demonstrations.

The consequence. The headline claim—that ICRL "eliminates the need for SFT and labeled data"—is accurate only in a narrow sense. ICRL does eliminate the need for thousands of supervised trajectories. But it replaces them with a different dependency: a small number of high-quality demonstrations from a frontier model that is presumably much stronger than the model being trained. If the demonstrations are generated by GPT-5.2 (which the paper uses), then ICRL is essentially distilling GPT-5.2's tool-use capabilities into a smaller model via in-context RL. This is a form of supervision that the paper's framing ("supervision-light") somewhat understates—the 3 demonstrations are still supervised data, just far fewer examples than traditional SFT. The cost of generating them is a one-time API call, but the availability of a model capable of producing them is a real constraint. If such a model doesn't exist for the target tool (e.g., a new API with novel interaction patterns), ICRL's bootstrap breaks.

A deeper consequence concerns demonstration quality as a hidden bottleneck. The paper does not ablate demonstration quality—all experiments use the same three GPT-5.2-generated examples. If these demonstrations contain subtle errors (e.g., an unnecessarily verbose search strategy, a suboptimal decomposition of multi-hop questions, a tendency to search when internal knowledge would suffice), the model may inherit those biases during early exploration. Unlike SFT, where such biases would be baked into the weights and potentially corrected during RL, ICRL's demonstrations influence exploration at every curriculum stage until they are removed entirely. The paper's finding that the four-stage curriculum causes premature stopping (Figure 2) hints that demonstrations shape behavior in persistent ways—if the demonstrations themselves encourage shallow reasoning, the model may never escape that attractor regardless of curriculum design.

What evidence exists in the paper. The paper provides no ablation on demonstration quality, number of initial demonstrations beyond the 3-shot starting point, or source of demonstrations (GPT-5.2 vs. other models vs. human-written). The only experiment varying demonstrations is the curriculum ablation (Figure 2), which changes how many of the same three demonstrations appear at each stage, not which demonstrations are used. The sensitivity to demonstration design is thus unmeasured. The paper also does not report what would happen if ICRL were trained with demonstrations from a weaker model (e.g., Qwen2.5-7B itself generating its own few-shot examples) or with demonstrations that contain deliberate errors. Without these experiments, a practitioner cannot assess how carefully the demonstrations need to be constructed.

Mitigation status. The paper does not address this limitation at all. There is no discussion of demonstration sensitivity, no suggestion for how to construct demonstrations for new tool domains, and no acknowledgment that the approach depends on access to a strong model for demonstration generation. Section 5 (Conclusion) frames ICRL as a "scalable and data-efficient alternative to traditional SFT+RL pipelines" without qualifying the requirement for high-quality demonstrations. A natural mitigation—training on dynamically selected demonstrations or allowing the model to generate and refine its own demonstrations during training—is not explored.

6.2 Single Model Family and Benchmark Suite Limits Generality; No Evidence for Cross-Model-Family Transfer

The assumption or constraint. All ICRL experiments use the Qwen2.5-Instruct model family (3B, 7B, 14B) for web search QA, with a single extension to Qwen3-8B for code execution math reasoning (Table 7). The paper states that the Qwen2.5 models are "widely adopted for question answering and reasoning tasks" and that the instruct variants provide "strong instruction-following capabilities, which enable faster and more stable convergence during RL training" (Section 3.1). This is a deliberate choice but also a significant scope limitation: the entire validation of ICRL rests on models from a single organization with a specific instruction-tuning recipe. The paper does not test Llama, Mistral, Gemma, DeepSeek, or any other model family—even though these models differ substantially in their in-context learning capabilities, instruction-following fidelity, and base reasoning skills, all of which could interact with ICRL's reliance on few-shot demonstrations during exploration.

The consequence. The core mechanism of ICRL—using in-context demonstrations to guide RL exploration—depends critically on the base model's ability to attend to and generalize from those demonstrations. Qwen2.5-Instruct models are specifically optimized for instruction following and in-context learning. A model with weaker in-context capabilities (e.g., a base model without instruction tuning, or a model from a family that emphasizes different capabilities) might not benefit from the demonstrations to the same degree. The paper explicitly acknowledges this indirectly by choosing instruct variants over base models: "We choose the instruct variants over base models due to their strong instruction-following capabilities, which enable faster and more stable convergence during RL training." This choice is well-justified, but it means the paper demonstrates ICRL's effectiveness only on models that are already good at the very capability (in-context learning) that ICRL exploits. A practitioner with a different base model cannot know whether ICRL would work without replicating the experiments.

The evaluation benchmark limitation compounds this issue. All QA experiments use five English-language factoid QA datasets (TriviaQA, HotpotQA, 2Wiki, Musique, Bamboogle) that share structural properties: short-answer questions with Wikipedia-sourced answers, retrievable via BM25 from an English corpus. The paper does not evaluate on non-English QA, on open-ended generation tasks, on multi-modal tool use (e.g., image retrieval), or on tool-use scenarios that require structured parameter passing rather than free-text search queries. The math reasoning extension (Table 7) broadens the domain to symbolic reasoning but still uses a constrained tool (Python execution with standard input/output). A practitioner deploying ICRL for a qualitatively different tool paradigm (e.g., SQL query generation, REST API calls with JSON parameters, multi-agent message passing) cannot infer likely performance from the paper's results.

What evidence exists in the paper. The evidence for generality is thin and indirect. The main results (Table 3) show consistent gains from 3B to 7B within the Qwen2.5 family, and the scaling to 14B (Table 6) shows continued improvement. The math reasoning result (Table 7) shows the same framework working on Qwen3-8B with a different tool, providing some evidence of cross-domain applicability. But there is no experiment varying the base model family, no experiment on non-English data, and no experiment with tools other than search and code execution. The paper treats these as demonstrations of generality ("ICRL also generalizes across domains, including web search and code execution, demonstrating its flexibility and effectiveness"), but the evidence supports only that the framework can be applied to two settings—it does not demonstrate that ICRL consistently works across diverse base models or tool types.

Mitigation status. The paper does not acknowledge this as a limitation. There is no discussion of the dependence on instruct-tuned models, no caveat about the Qwen2.5 family's specific properties, and no call for cross-model-family validation in future work. The Conclusion (Section 5) presents ICRL as a general framework without qualification. A reader unfamiliar with the variance across LLM families might reasonably infer that ICRL would work equally well on any instruction-tuned model, which the paper has not established.

6.3 No Compute-Matched Comparison Makes Accuracy Gains Uninterpretable as Efficiency Gains

The assumption or constraint. The paper reports accuracy improvements over baselines (e.g., +8.94 average EM on 3B, +7.34 on 7B) but provides no information about the relative computational cost of ICRL training versus baseline training. The paper does not report total FLOPs, GPU-hours, number of training steps, total tokens generated, or wall-clock time for ICRL or any baseline. The only hardware information is that training uses 4 NVIDIA A100 GPUs (80GB each) with a batch size of 64 and 8 trajectories per query (Section 3.1). Standard RL methods for tool use vary widely in their computational requirements—Search-R1, ZeroSearch, and ParallelSearch each have different training recipes with potentially different numbers of rollouts per query, different response length caps, and different total training durations. Without controlling for compute, a reader cannot distinguish between "ICRL is a more efficient algorithm" and "ICRL used more training compute."

The consequence. This is a fundamental interpretability problem for the paper's central claim. The headline numbers in Table 3 show ICRL outperforming Search-R1 by +8.94 on 3B and ParallelSearch by +7.34 on 7B. But if ICRL's training required 2× or 3× more GPU-hours than these baselines (due to 8 trajectories per query at temperature 1.0 with full tool execution, compared to potentially fewer or shorter rollouts for baselines), the accuracy advantage might simply reflect a larger effective training budget. This is not a hypothetical concern: GRPO with 8 rollouts per query already multiplies the per-query generation cost by 8 compared to methods that use 1 or 4 rollouts per query, and the tool execution itself (6 search calls per trajectory × 8 trajectories × 64 queries per batch = up to 3,072 search engine calls per training step, plus BM25 retrieval) adds nontrivial overhead. Without compute accounting, the paper's claim of "data-efficient" (no SFT data needed) and "scalable" cannot be evaluated against the actual resource requirements.

The inference-time cost is similarly unreported. The trained ICRL model uses multiple search turns (Figure 2b shows a significant fraction of queries requiring 3-6 searches for the 3→2→0 model), while the four-stage model answers most questions in 1-2 turns. Higher accuracy comes with higher inference latency—the 3→2→0 model on TriviaQA (75.4 EM) might average 3-4 search calls per query, making it potentially 2-3× slower at inference than a model that answers in 1-2 searches. The paper does not report average search turns per query for ICRL at evaluation time (only during training in Figure 3c and the ablation in Figure 2b), so the latency-accuracy Pareto frontier is unknown.

What evidence exists in the paper. The paper provides no compute-matched comparisons, no FLOP counts, no training time measurements, and no inference latency measurements for ICRL or any baseline. The training curves in Figure 3 show metrics over the course of training steps, but the x-axis is steps, not FLOPs or wall-clock time, and the total number of steps is not reported. The GRPO algorithm uses 8 trajectories per query (stated in Section 3.1), but the paper does not report how many trajectories per query the baselines use during their RL phases, making it impossible to compare sample efficiency. The only cost-related detail is that models are loaded in bfloat16 and trained on 4 A100s with FSDP and gradient checkpointing—this tells a reader that training is feasible, not what it costs relative to alternatives.

Mitigation status. The paper does not acknowledge this limitation. There is no mention of compute matching, FLOP accounting, or the need for cost-controlled comparisons in the discussion of results or in the Conclusion. The absence of compute information makes it impossible to assess whether ICRL is genuinely more efficient than baselines or simply uses more computation to achieve better accuracy, which is a critical distinction for practitioners deciding whether to adopt the method.

6.4 No Statistical Significance or Variance Reporting Undermines the Reliability of Claimed Improvements

The assumption or constraint. Every quantitative result in the paper is reported as a single-number EM accuracy with no error bars, confidence intervals, standard deviations, or indication of statistical significance. Tables 3, 4, 6, and 7 present accuracies to one decimal place (e.g., 40.16, 49.12) for individual datasets and averages. Figure 2 presents bar charts with no error bars. Figure 3 presents training curves (presumably averaged over training steps or rollouts, but the aggregation method is unspecified) with no variance shading. The paper does not mention running multiple training seeds, performing cross-validation, or computing any measure of uncertainty for any result.

The consequence. RL training is fundamentally noisy. GRPO relies on sampling 8 trajectories per query at temperature 1.0, which introduces substantial stochasticity in the reward signals and advantage estimates. Different random seeds can produce meaningfully different final policies due to differences in which exploratory trajectories happen to succeed early in training, creating path-dependence in the learned behavior. Without variance information, a reader cannot assess whether the reported improvements—particularly the moderate ones—are reliable or within the noise of training stochasticity. For example, the difference between ICRL and ParallelSearch on HotpotQA for Qwen2.5-7B is 42.6 vs. 42.9 (Table 3), a 0.3 percentage point difference that is almost certainly within the noise of a single training run. The average improvement of +7.34 on 7B is driven primarily by large differences on TriviaQA (+10.2) and 2Wiki (+11.2), but if those differences are partially attributable to seed variance, the true advantage could be substantially smaller—or larger. Without error estimates, the paper's quantitative claims are point estimates from single training runs, which is insufficient for establishing reliable superiority over baselines in a field where ±2-3 percentage point variance across seeds is common for RL training.

The test set sizes compound this problem. The paper evaluates on "up to 500 questions randomly sampled from each dataset" (Section 3.1). With 500 test questions, a 2-3 percentage point difference corresponds to 10-15 questions. If the true accuracy difference between ICRL and a baseline is small, the observed difference could be explained by which specific 500 questions were sampled. The paper does not report whether test set samples are fixed across methods or resampled, and does not use bootstrapping or any other technique to estimate the variance due to finite test sets. For the difficulty bin analyses that would decompose performance further (analogous to the reference paper's five difficulty quintiles), the sample size per bin would be even smaller, making variance estimation even more critical—but the paper does not perform such decomposition.

What evidence exists in the paper. The limitation is evident from the complete absence of any variance metric throughout the experimental section. The training curves in Figure 3 show smooth trends, which suggests some form of averaging (likely across batches within an epoch or across rollouts), but the averaging window and smoothing method are not specified. The paper does not claim to have run multiple seeds, does not acknowledge the sensitivity of GRPO training to stochasticity, and does not discuss uncertainty in any capacity. This is inconsistent with standard practice in empirical ML, where RL results are typically reported with multiple seeds and standard deviations, particularly when claiming superiority over a range of baselines.

Mitigation status. The paper does not address this at all. There is no mention of seed variance, no plan for multiple training runs, and no discussion of the reliability of the numerical results. This is a significant methodological weakness that affects the interpretability of every quantitative claim in the paper. For a practitioner deciding whether to invest resources in replicating ICRL, the absence of variance information means the reported gains come with unknown reliability—the true expected performance of ICRL relative to baselines could be substantially better or worse than the point estimates suggest.

6.5 Hard Problems and Out-of-Distribution Tool Use Scenarios Remain Unaddressed

The assumption or constraint. The paper evaluates ICRL on a specific class of QA benchmarks (factoid questions with retrievable answers) and a specific math reasoning benchmark (competition problems solvable with code execution). The results in Table 3 show that ICRL's performance varies dramatically across datasets: on Qwen2.5-7B, TriviaQA reaches 75.4 EM while Musique reaches only 26.0 EM—a gap of nearly 50 percentage points. This variation reflects underlying differences in problem difficulty and reasoning complexity, but the paper does not analyze which questions ICRL succeeds or fails on, what properties make a question hard for ICRL, or whether there exists a difficulty threshold beyond which ICRL provides no benefit over simpler methods. This is in stark contrast to the reference paper (on compute-optimal test-time scaling), which decomposed all results by five difficulty quintiles and explicitly identified the hardest bin as a regime where test-time compute provides near-zero improvement regardless of budget.

The consequence. A practitioner deploying ICRL cannot predict whether it will help on their specific problem distribution. The paper demonstrates strong performance on multi-hop QA benchmarks (2Wiki, Bamboogle, Musique) relative to baselines, but the absolute numbers tell a different story: 53.6 on 2Wiki means ICRL fails on 46.4% of questions; 26.0 on Musique means it fails on 74.0% of questions. These are not solved benchmarks. Without understanding why ICRL fails on the questions it gets wrong, it is impossible to know whether those failures are due to retrieval quality (BM25 misses relevant documents), reasoning complexity (the model cannot compose information across 3+ hops), or fundamental capability limits of the base Qwen2.5 models. If the failures are due to retrieval quality, then switching to a dense retriever might help. If they are due to reasoning limits, then larger base models or different training strategies would be needed. If they are due to the 6-turn search cap, simply allowing more searches might help. The paper provides no failure analysis, no examples of incorrect answers (beyond the single correct example in Table 5), and no difficulty-stratified results.

The math reasoning results (Table 7) further illustrate the bounds: ICRL achieves 51.7% on AIME2025 on Qwen3-8B, meaning it still fails on nearly half of competition math problems despite tool access and RL training. The paper does not analyze whether ICRL failures on math problems stem from incorrect code generation, misinterpretation of problem statements, or inability to map mathematical reasoning to executable code. Without such analysis, a practitioner targeting a problem domain harder than AIME (e.g., research-level mathematics, formal theorem proving) has no guidance on whether ICRL would help or what adaptations would be needed.

What evidence exists in the paper. The paper provides one qualitative example (Table 5) showing a successful multi-hop reasoning trajectory, and aggregate EM scores per dataset. There is no error analysis, no difficulty-based performance breakdown, no examples of failure modes, and no discussion of what types of questions ICRL consistently gets wrong. The training curves (Figure 3) show increasing valid search calls during the 0-shot phase, but this only indicates that the model is searching more—it does not show whether those additional searches lead to correct answers on hard questions or are wasted on questions the model fundamentally cannot answer.

The paper also does not evaluate on benchmarks designed to stress-test tool-use capabilities, such as adversarial retrieval settings (where relevant documents are deliberately mixed with distractors), questions requiring tool chains (where the model must use one tool's output as another tool's input), or questions requiring tool selection (where multiple tools are available and the model must choose which to use). The tool-use setting is simplified: a single tool (search or code execution) with a fixed interface and a 6-turn limit. Real-world tool use often involves multiple tools, conditional tool selection, and error recovery when a tool returns unexpected or empty results. The paper provides no evidence about ICRL's behavior in these more realistic scenarios.

Mitigation status. The paper does not acknowledge this as a limitation. The Conclusion (Section 5) presents ICRL as achieving "strong performance across a range of QA and reasoning benchmarks" without discussing the substantial failure rates on the hardest benchmarks or the unknown behavior on problem types not represented in the evaluation suite. There is no call for failure analysis, difficulty-stratified evaluation, or extension to more complex tool-use scenarios. For a practitioner whose problem distribution includes questions harder than the average Musique or AIME problem, the paper provides no basis for estimating likely performance.

6.6 The 6-Turn Search Cap and Fixed 2048-Token Response Limit Artificially Bound Multi-Turn Reasoning Depth

The assumption or constraint. The paper imposes a hard limit of 6 search turns per trajectory and a maximum response length of 2048 tokens (Section 3.1: "allowing up to 6 search turns per query"). The prompt can be up to 5000 tokens (to accommodate the few-shot demonstrations). These limits are practical engineering choices driven by GPU memory constraints and the need to keep training iterations tractable. However, they also define an artificial ceiling on the complexity of tool-use behavior the model can learn. If a question genuinely requires 8 or 10 search-and-reason cycles (e.g., complex multi-hop questions requiring verification of intermediate facts, or compositional queries where each sub-answer requires its own retrieval), the model is physically prevented from learning such strategies—trajectories that would require more than 6 searches are truncated before completion.

The consequence. The 6-turn cap interacts with the curriculum design findings in a concerning way. The paper's central ablation (Figure 2) shows that the 3→2→0 curriculum produces models that "distribute search turns across 1–6, with a substantial fraction using 3+ searches" (paraphrased from the paper's discussion), while the 3→2→1→0 curriculum produces models where "over 80% of queries finish within two search turns." The paper interprets the 3→2→0 model's broader search distribution as evidence of learning "deep multi-turn reasoning." But this interpretation is constrained by the 6-turn ceiling: we cannot know whether the 3→2→0 model would search even more (7, 8, 10 turns) if allowed, or whether it has converged to using 4-6 turns because that's what the reward signal supports. More critically, if some questions in Musique or 2Wiki genuinely require more than 6 searches for reliable answering, then the 6-turn cap imposes a hard ceiling on ICRL's maximum possible accuracy on those datasets—no amount of training can overcome a physical limit on the number of tool interactions. The paper's reported accuracy of 26.0 on Musique and 53.6 on 2Wiki (for 7B) should thus be understood as potentially hardware-limited, not a measure of the model's true capability ceiling.

A secondary consequence concerns inference latency unpredictability. Figure 2b shows the 3→2→0 model using 1-6 search turns with a broad distribution. This means inference latency varies substantially across queries: a question answered in 1 search turn is fast; a question requiring 6 search turns is potentially 6× slower (plus reasoning time). The paper reports no latency statistics—mean, median, tail latency (99th percentile)—making it impossible to assess whether ICRL-trained models are suitable for latency-sensitive applications. If the distribution of search turns has a long tail (e.g., 10% of queries using 5-6 turns), the worst-case latency could be unacceptable for interactive applications even if the accuracy is excellent.

What evidence exists in the paper. The 6-turn limit is stated in the implementation details (Section 3.1) but is never discussed as a constraint that could affect results. Figure 2b shows the distribution of search turns for the trained models, with the 3→2→0 model's distribution appearing to have non-trivial mass at 5 and 6 turns—exactly the region where the cap binds. This suggests that some queries might benefit from additional searches beyond 6 but are prevented from doing so. The paper does not report what fraction of trajectories hit the 6-turn cap during training or evaluation, which would directly measure how often the cap is binding. The 2048-token response limit is similarly unanalyzed: we don't know what fraction of trajectories are truncated by the token limit rather than by the model choosing to answer.

Mitigation status. The paper does not acknowledge the search-turn cap or response-length limit as limitations. There is no experiment varying the cap (e.g., training with 4, 6, 8, or 10 maximum turns to measure how performance scales with allowed search depth), no analysis of truncated trajectories, and no discussion of how the cap might interact with the curriculum design findings. The Conclusion presents ICRL's multi-turn reasoning capabilities as a strength of the method without noting that those capabilities are bounded by an engineering constraint rather than by the learning algorithm itself. For a practitioner considering deploying ICRL on problems that require deep multi-step research (e.g., literature reviews, complex legal or medical reasoning, multi-document synthesis), the paper provides no evidence about how performance degrades or plateaus as the required search depth increases beyond the studied range of 1–6 turns.

7. Implications and Future Directions

How This Work Changes the Landscape

ICRL introduces a methodological reframing with practical efficiency consequences rather than a paradigm shift. It does not propose a fundamentally new learning algorithm—it uses GRPO, which is standard—nor does it introduce a new model architecture. Instead, it demonstrates that the separation between "providing initial guidance" (traditionally done via SFT) and "optimizing for rewards" (done via RL) can be implemented entirely through the rollout prompt rather than through model weights, and that this separation produces better final policies than the standard cold-start pipeline that entangles them. This is a reframing of where supervision lives during training, not whether supervision is needed, and the practical consequence is that a practitioner can train effective tool-use models with three GPT-generated demonstrations instead of thousands of SFT trajectories.

The magnitude of this reframing should be sized carefully. It is not a demonstration that RL alone works for tool use—the paper explicitly acknowledges that RL from scratch fails, and ICRL's central mechanism is providing demonstrations during exploration to solve exactly this failure. It is also not a demonstration that SFT is unnecessary in general—the demonstrations are still a form of supervision, just provided in-context rather than through weight updates. What changes is the cost structure of that supervision: generating three demonstrations costs a few API calls to GPT-5.2; generating thousands of SFT trajectories for each new tool domain costs orders of magnitude more. The reframing matters most for practitioners who need to train tool-use models across many domains, tools, or model versions, where the per-domain SFT cost becomes a bottleneck.

The paper also provides a diagnostic contribution about curriculum pacing that extends beyond tool use. The finding that a four-stage curriculum (3→2→1→0 shots) causes catastrophic performance degradation compared to a three-stage curriculum (3→2→0 shots)—with TriviaQA dropping from 75.4 to 20.8 EM—is not just a hyperparameter tuning result. It is evidence of a qualitative attractor in learning dynamics: the intermediate 1-shot stage teaches the model to stop searching prematurely, and once this shallow-strategy attractor is established, the sparse reward signal cannot push the model toward deeper multi-turn reasoning. This finding generalizes to any setting where models learn from a combination of in-context demonstrations and sparse outcome rewards: the schedule at which demonstrations are removed determines what kind of behavior the model converges to, not just how fast it converges. Curriculum design in scaffold-fading settings is therefore a first-class research problem, not a secondary optimization detail.

Reconciling prior contradictions. The paper implicitly resolves a tension between two bodies of evidence. On one side, work like Search-R1, ZeroSearch, and ParallelSearch demonstrates that RL can train effective tool-use policies, but these methods either rely on existing instruction-tuned models with implicit formatting knowledge or use cold-start SFT before RL. On the other side, the field widely recognizes that RL from scratch on tool-use tasks fails due to the exploration problem. ICRL's contribution shows that these findings are not contradictory—RL works for tool use if and only if the initial exploration distribution is adequately shaped—and it provides a specific mechanism (in-context demonstrations with progressive removal) for doing that shaping without SFT. This reframes the research question from "RL or SFT+RL?" to "what is the cheapest way to provide adequate exploration guidance?," opening a design space that includes not just in-context demonstrations but potentially curriculum learning, demonstration retrieval, or learned exploration policies.

Research directions that become more attractive:

Prompt-based guidance for RL exploration—ICRL shows that in-context demonstrations can substitute for SFT for tool use. This immediately suggests investigating whether the same approach works for other tasks with exploration bottlenecks: code generation with library APIs, multi-agent coordination protocols, structured data extraction, or formal theorem proving with tactic selection. Each of these has a similar structure: a large combinatorial action space where random exploration fails, but where a small number of demonstrations can cover the basic action patterns.
Curriculum pacing theory for demonstration reduction—the dramatic difference between 3→2→0 and 3→2→1→0 curricula suggests that pacing is not just a nuisance parameter but a fundamental determinant of learned behavior. This opens a research program around optimal demonstration reduction schedules as a function of task complexity, reward sparsity, and model capacity.
Dynamic and query-adaptive demonstration selection—ICRL uses fixed demonstrations throughout each curriculum stage. A natural extension is to select demonstrations dynamically based on the training query (retrieve the most relevant examples) or on the model's current competence (provide more support for query types where the model is struggling, less where it has mastered the behavior). This connects tool-use training to the retrieval-augmented generation and active curriculum learning literatures.

Research directions that become less attractive:

Cold-start SFT for tool-use training in settings where a few high-quality demonstrations can be generated cheaply. ICRL shows that for web search QA and code execution math reasoning, the SFT phase can be replaced with a small number of in-context demonstrations. For a practitioner with access to GPT-5.2 (or any strong model) but without thousands of SFT trajectories, investing in cold-start SFT pipelines is now harder to justify—ICRL provides comparable or better performance at a fraction of the data preparation cost. The cold-start paradigm may remain necessary for tool-use settings where even generating three high-quality demonstrations is difficult (e.g., tools with complex structured interfaces where the frontier model doesn't already know how to use them), but for text-based tools where the demonstration format follows a natural language pattern, ICRL's results make the SFT-first approach look unnecessarily expensive.
Process reward models for tool use, at least as a first investment. ICRL achieves strong multi-turn tool-use behavior (up to 6 searches per query, per Figure 2b) using only outcome-based rewards (answer accuracy + format compliance). While process rewards might further improve performance—particularly on the hardest benchmarks where Musique accuracy is only 26.0 at 7B—ICRL's results suggest that exploration guidance is the binding constraint, not reward density. A practitioner deciding where to invest engineering effort should prioritize better demonstrations and curriculum design over building step-level verifiers, at least until the exploration bottleneck is saturated.

Follow-Up Research This Work Enables

Optimal demonstration reduction schedules as a function of task reasoning depth. The paper's central ablation (Figure 2) compares exactly two curricula—3→2→0 and 3→2→1→0—and finds a dramatic difference. But this raises a larger question: for a task requiring K-hop reasoning, what is the optimal number of stages and the optimal number of initial demonstrations? A systematic study could train ICRL with curricula ranging from 2→0 (minimal scaffolding, sharp transition) to 5→4→3→2→1→0 (maximal scaffolding, gradual transition) on a stratified benchmark like Musique where ground-truth reasoning depth is known (1-hop, 2-hop, 3-hop, 4-hop questions). The prediction from ICRL's results is that tasks requiring deeper reasoning benefit from fewer intermediate stages (to avoid the shallow-strategy attractor), while simpler tasks may be insensitive to curriculum design. Measuring per-hop accuracy as a function of curriculum would directly test this and produce actionable guidance for practitioners.

Cross-model-family validation of the in-context RL mechanism. All ICRL experiments use Qwen2.5-Instruct models, which are specifically optimized for instruction following and in-context learning. The core mechanism—in-context demonstrations guide exploration during RL—depends on the base model's ability to attend to and generalize from those demonstrations. A necessary stress test is to replicate ICRL on model families with different in-context learning capabilities: Llama-3.1-Instruct, Mistral-Large-Instruct, Gemma-2-Instruct, and DeepSeek-V2-Instruct. If ICRL produces similar gains across all families, the mechanism is robust and a practitioner can adopt it without concern for base model choice. If ICRL works well on Qwen2.5 but degrades substantially on, say, Gemma-2 (which has different instruction-tuning properties), then the method is family-dependent and practitioners need guidance on which model properties enable successful in-context RL. The experiment should control for total training compute and report per-family gains relative to the same RL-without-SFT baseline.

Query-adaptive demonstration retrieval during RL rollouts. ICRL uses the same three fixed demonstrations for every training query at each curriculum stage. This is simple but potentially suboptimal: a demonstration showing multi-hop reasoning over political figures is less relevant when the training query asks about a scientific concept. A natural extension is to maintain a pool of diverse demonstrations and, for each training query, retrieve the K most relevant demonstrations (by embedding similarity between the query and the demonstration questions) to include in the rollout prompt. This would test whether demonstration relevance matters more than demonstration quality, and whether dynamic selection enables faster learning or better generalization than fixed demonstrations. The experiment should compare three conditions: ICRL with fixed demonstrations (replication), ICRL with dynamically retrieved demonstrations from a pool of 50+ GPT-generated examples, and ICRL with randomly selected demonstrations (to isolate the effect of relevance from the effect of diversity). If dynamic retrieval improves performance—particularly on out-of-domain benchmarks like 2Wiki and Bamboogle where query patterns differ from the NQ training distribution—it would suggest that the exploration guidance function of demonstrations benefits from being query-adaptive.

Failure mode characterization on the hardest QA questions. ICRL achieves only 26.0 EM on Musique and 53.6 on 2Wiki at 7B (Table 3). These low absolute numbers—despite being state-of-the-art—mean the model fails on a large fraction of questions, and the paper provides no analysis of why. A necessary follow-up is to categorize 100 randomly sampled failure cases from Musique into error types: retrieval failure (search returns irrelevant documents despite well-formulated queries), query formulation failure (model asks the wrong question to the search engine), integration failure (model retrieves relevant information but fails to compose it into a correct answer), premature stopping (model answers before gathering all necessary information), and hallucination (model generates an answer inconsistent with retrieved information). This categorization would reveal whether the primary bottleneck is retrieval quality (in which case switching from BM25 to dense retrieval or increasing top-K would help), reasoning capability (in which case larger base models or different training objectives are needed), or the 6-turn search cap (in which case simply allowing more searches would improve performance on the hardest questions). The experiment should also measure per-hop accuracy on Musique (which has ground-truth decomposition depth) to determine whether ICRL's failures are concentrated at greater reasoning depths.

Scaling the number of initial demonstrations and the phase-out granularity. The paper fixes the starting point at 3-shot and compares only 3→2→0 vs. 3→2→1→0. This leaves open the entire space of curricula: does 4→2→0 outperform 3→2→0? Does 5→3→0? Is there a threshold beyond which more initial demonstrations provide diminishing returns? And on the phase-out side: does 3→0 (single-stage transition) work? Does 3→2.5→2→1.5→1→0.5→0 (continuous reduction, implemented by randomly dropping demonstrations with increasing probability) outperform discrete-stage curricula? This is a large ablation space, but the paper's demonstration that curriculum design matters (Figure 2) makes exploring it valuable. A systematic sweep over initial demonstration count (1 through 10) and phase-out schedule (varying both the number of intermediate stages and the sharpness of the transition) on a single model size (7B) and benchmark suite would produce the first empirical characterization of scaffold-fading dynamics in tool-use RL. The key metric is not just final accuracy but the shape of the learning curve at each transition—does the model exhibit a temporary performance dip when demonstrations are reduced (suggesting dependence on the scaffolding), and how does the magnitude of that dip relate to the sharpness of the reduction?

Adversarial demonstration quality experiments to bound ICRL's sensitivity. The paper uses demonstrations generated by GPT-5.2, presumably a strong frontier model. How sensitive is ICRL to demonstration quality? A systematic degradation experiment could train ICRL with: (1) the original GPT-5.2 demonstrations (upper bound), (2) demonstrations generated by the base Qwen2.5-7B-Instruct model itself before any tool-use training (lower quality, plausibly available without a frontier model), (3) human-written demonstrations from crowd workers with minimal guidance (realistic for new tool domains where no model knows how to use the tool), and (4) demonstrations that contain deliberate errors—such as an unnecessary search query, a poorly formatted thinking step, or a suboptimal query formulation (to test whether ICRL can overcome bad demonstrations through RL). If ICRL performance degrades gracefully with demonstration quality (condition 2 performs moderately worse than condition 1; condition 3 performs comparably to condition 2), then the method is robust and the bootstrap requirement is practical. If performance collapses with even mildly suboptimal demonstrations, then ICRL's practical applicability is limited to settings where a frontier model can generate the demonstrations—exactly the kind of dependency the paper frames as avoidable. This experiment would also clarify whether the exploration guidance function is about demonstration correctness (the model needs to see perfect examples) or demonstration format/structure (the model just needs to see the right output shape, and RL can correct content errors).

Practical Applications and Downstream Use Cases

Multi-domain tool-use deployment without per-domain SFT pipelines. An organization that needs to deploy tool-augmented LLMs across multiple domains—say, a legal research assistant that searches case law databases and a medical QA system that queries PubMed—currently faces a per-domain SFT burden: each domain requires collecting hundreds or thousands of tool-use trajectories, formatting them, and running SFT before RL. ICRL reduces the per-domain setup cost to generating 3 high-quality demonstrations per domain (a few GPT-5.2 API calls) and running the same RL curriculum. The paper's results suggest this produces competitive or better performance than SFT-based approaches: ICRL outperforms O2-Searcher (which uses cold-start SFT) by +2.9 average EM on Qwen2.5-3B (Table 4) and matches ReTool (SFT+RL) on AIME math reasoning within a few percentage points (Table 7). For an organization targeting 5-10 tool domains, the data preparation savings could be substantial—replacing thousands of SFT trajectories with 30 demonstrations total—while maintaining state-of-the-art accuracy.

Self-improving search-augmented QA systems with modest compute budgets. ICRL's training recipe is feasible on 4 A100 GPUs (80GB each) for 3B-7B models, which is within the compute budget of well-resourced academic labs and mid-size industry teams. The trained model can then be deployed with the same search tool used during training (Serper API or an internal search index). The paper's results on Qwen2.5-7B—75.4 EM on TriviaQA, 53.6 on 2Wiki, 48.0 on Bamboogle—represent strong performance on knowledge-intensive QA without requiring a larger model or closed-source API access at inference time. For applications where answer accuracy on fact-based questions is critical and latency is acceptable (the model uses 1-6 search turns per query, per Figure 2b), ICRL provides a recipe for training a competitive system from publicly available models and data, with the only external dependency being the 3 demonstrations (generated once) and the search API (used during both training and inference). A team could replicate the entire pipeline—from downloading Qwen2.5-7B-Instruct to deploying a trained search-augmented model—with a few days of GPU time and minimal data preparation.

Bootstrapping tool-use capabilities in new models without human annotation. When a new instruction-tuned model is released (e.g., a future Llama-4-Instruct or Gemma-3-Instruct), ICRL provides a way to add tool-use capabilities without waiting for the community to produce SFT datasets for that specific model. A practitioner can take the same 3 GPT-5.2-generated demonstrations used in the paper, load them into the rollout template, and run ICRL training on NQ (or any QA dataset with ground-truth answers) to produce a tool-augmented version of the new model. The paper's results show this works for Qwen2.5-3B, 7B, and 14B (Tables 3 and 6) with consistent improvements over baselines, and for Qwen3-8B on code execution (Table 7), suggesting the approach transfers across model scales and generations within the Qwen family. Whether it transfers to entirely different model families is untested (see follow-up research direction above), but for organizations building on the Qwen ecosystem, ICRL provides a turnkey recipe for enabling search-augmented reasoning on new model releases with zero additional annotation cost beyond the initial 3 demonstrations.

When to Prefer This Method

The paper positions ICRL explicitly as an alternative to cold-start SFT+RL pipelines, and the experimental design supports a direct comparison. Based on the paper's evidence, the decision rule is:

Prefer ICRL over cold-start SFT+RL when: (1) you are training a relatively small instruction-tuned model (3B-14B parameters) within the Qwen2.5 family or a model with comparable instruction-following capabilities, (2) the tool interface involves natural language queries with structured text responses (search, code execution, and plausibly API documentation retrieval or database natural-language queries), (3) a frontier model is available to generate 3 high-quality demonstrations of the desired tool-use behavior (a one-time cost), and (4) your evaluation metric is final answer accuracy on knowledge-intensive QA or similar tasks where exact match against ground truth provides a binary reward signal. Under these conditions, ICRL provides accuracy comparable to or better than SFT+RL (Tables 4 and 7) with dramatically lower data preparation cost.
Prefer cold-start SFT+RL when: (1) the tool interface is highly structured (e.g., SQL with strict syntax that few-shot demonstrations cannot reliably convey, REST APIs with complex parameter schemas) and the model needs to learn precise formatting conventions that in-context demonstrations may not adequately cover, (2) a frontier model is not available to generate high-quality demonstrations for the target tool (for truly novel tools or proprietary internal APIs), (3) the task requires reasoning chains substantially longer than 6 tool interaction turns—the paper's 6-turn cap bounds ICRL's demonstrated capability, and longer chains may require the more stable initialization that SFT provides, or (4) inference latency is a primary constraint and the trained model must answer in 1-2 tool calls consistently—ICRL's 3→2→0 curriculum produces models that use 1-6 turns broadly (Figure 2b), and if a 6× latency variance is unacceptable, the behaviorally more constrained output of an SFT-trained model may be preferable despite lower average accuracy.
The paper does not provide evidence to choose between ICRL and RL-only methods that don't use SFT but also don't use in-context demonstrations (e.g., R1-base, R1-instruct). These baselines underperform ICRL substantially in Table 3 (R1-instruct: 23.68 vs. ICRL: 40.16 on 3B; 28.62 vs. 49.12 on 7B), suggesting that the in-context demonstrations provide benefits beyond what RL alone achieves even with strong base models. But the paper does not compare ICRL to these methods at matched compute budgets, so the comparison is accuracy-only. A practitioner choosing between ICRL and a simpler RL-only approach should weigh the accuracy advantage (+15-20 average EM) against the (unknown) additional training cost of the curriculum and demonstration infrastructure.