ZeroSearch: Incentivize the Search Capability of LLMs without Searching

🎯 Pitch

A 7B LLM simulating search results during RL training matches the performance of a model trained on real Google Search—and a 14B simulator surpasses it—by progressively feeding the trainee noisier documents in a curriculum. The key is that document quality must be deliberately degraded during training, not just simulated, to force the model to learn robust reasoning without ever calling a live search API.

1. Executive Summary

This paper introduces ZEROSEARCH, a novel reinforcement learning framework that trains LLMs to use real search engines without interacting with them during training, instead substituting a lightweight fine-tuned LLM as a simulated retrieval module. Using Qwen-2.5 and LLaMA-3.2 models on seven QA benchmarks—including NQ, TriviaQA, and HotpotQA—the framework employs a curriculum rollout mechanism that progressively degrades the quality of simulated documents (from useful to noisy) to elicit increasingly robust search and reasoning behaviors. ZEROSEARCH outperforms methods trained with real search engines—a 7B simulation LLM achieves parity with Google Search, while a 14B simulation LLM surpasses it—all while incurring zero API costs, establishing that a well-trained local simulator can serve as an effective substitute for live retrieval only when document quality is deliberately controlled through curriculum-based training.

2. Context and Motivation

The Core Problem: RL Search Training is Bottlenecked by Real Search Engine Dependencies

The fundamental challenge this paper tackles is deceptively simple: how do you train an LLM to use a search engine effectively without actually calling a search engine during training? This matters because reinforcement learning (RL) has emerged as a dominant paradigm for teaching LLMs to perform multi-step reasoning tasks—including iterative information retrieval—but RL training requires thousands of rollout trajectories where the model interacts with its environment to learn from trial and error. When that environment is a live commercial search API, the financial and practical costs become prohibitive at scale.

The paper frames this as a scalability bottleneck that prevents the widespread adoption of RL-based search training. Consider the arithmetic: a typical RL training run with a batch size of 64, five rollouts per prompt, and 200 training steps generates approximately 64,000 search queries. At commercial API rates (the paper estimates Google Web Search via SerpAPI at roughly $586.70 for this volume; Section D, Table 8), this is manageable for a single experiment. But research isn't a single experiment—it's hyperparameter sweeps, ablation studies, model size comparisons, and algorithm variants. Each experiment multiplies the cost. If you want to scale batch sizes, increase training steps, or run larger models that require more rollouts, the API bill grows linearly with no ceiling. Worse, if you adopt algorithms like REINFORCE or GRPO that require multiple repeated rollouts per prompt (Section 5.4), the cost multiplies further. The result is that only well-resourced labs can afford to experiment in this space, and even they face practical limits on how much training they can do.

This gap is significant for several practical reasons the paper highlights implicitly throughout:

Democratizing research: If search-engine access is a prerequisite for training capable retrieval-augmented LLMs, progress is gated by budget. A simulation-based approach removes this gate.
Scalability: Real search APIs have rate limits, latency, and concurrent-request caps that constrain RL training throughput. A GPU-hosted simulation LLM scales horizontally with hardware—add more GPUs, generate more simulated documents in parallel (Section 1, paragraph 3: "increasing the number of GPUs significantly accelerates the generation throughput of the simulation LLM").
Reproducibility: Real search engine results change over time as the web is updated, indexed differently, or rate-limited differently. Two runs of the same experiment conducted weeks apart may encounter different document distributions. A simulated search engine provides a frozen, controllable environment.

Why Prior Approaches Fall Short

The paper identifies specific limitations in three categories of prior work, each of which fails to fully address the scalability-reliability tension in search-based RL training.

Prompt-based RAG methods require manual engineering and strong models. Early work on retrieval-augmented generation relied on prompting strategies: guide the LLM through query generation, decomposition, and multi-turn retrieval using carefully designed instructions (Section 2.1, citing Yu et al., 2022; Press et al., 2023; Yoran et al., 2023). While effective, these approaches have two weaknesses. First, they depend on intricate prompt engineering that is brittle across domains and requires human expertise to tune. Second, they implicitly assume the underlying model is already capable of reasoning about when and what to search—a capability that smaller or weaker models may lack entirely. The paper notes that these methods "place high demands on the model's reasoning capabilities" (Section 1, paragraph 2), making them more a test of a model's pre-existing skill than a training method for acquiring that skill.

Supervised fine-tuning (SFT) methods improve smaller models but rely on expensive trajectory data. To address the dependency on strong base models, subsequent work (Self-RAG by Asai et al., 2024; RetroLLM by Li et al., 2024; Adaptive-RAG by Jeong et al., 2024) proposed fine-tuning smaller LLMs on search-and-reasoning trajectories. The problem is that generating those trajectories typically requires a capable model interacting with a real search engine to produce high-quality demonstrations. The data pipeline inherits the same API cost problem that RL faces, just shifted to the data generation phase. Moreover, SFT alone teaches the model to imitate retrieval behavior but doesn't optimize for the end goal of correct answers through reinforcement—it's teaching form without necessarily optimizing function.

Test-time scaling methods (MCTS, tree search) push the computational burden to inference. Recent work (Jiang et al., 2024; RAG-Star by Jiang et al., 2024; AirRAG by Feng et al., 2025) explores Monte Carlo Tree Search and related techniques that expand the search space dynamically during inference. While these approaches show promise, the paper identifies a critical limitation: they "incur significant computational overhead, posing challenges for practical deployment" (Section 1, paragraph 2). The cost is not just in API calls but in the wall-clock time required for iterative search-space exploration—each query may spawn dozens or hundreds of sub-queries. For real-time applications, this latency is unacceptable. These methods also don't train the model to be a better searcher; they just brute-force the search at test time.

RL methods using real search engines face document quality instability. The recent wave of RL-based search training—Search-R1 (Jin et al., 2025), R1-Searcher (Song et al., 2025), ReSearch (Chen et al., 2026), DeepResearcher (Zheng et al., 2025), and WebThinker (Li et al., 2026)—directly tackles the problem by training policy models through interaction with live search engines. This is the most directly relevant prior work, and the paper positions ZEROSEARCH as an evolution that addresses its two fundamental weaknesses (Section 1, paragraph 3):

Uncontrolled Document Quality: When you query a real search engine during RL training, you have no control over what comes back. The top-5 results might be perfect—containing the answer directly—or might be completely irrelevant, outdated, or contradictory. This introduces noise and instability into the RL training signal. In early training, when the policy model is still learning how to formulate queries and parse results, getting bad documents can prevent it from ever learning the correct association between good query formulation and good information retrieval. In later training, getting too-good documents can make the task trivial—the model doesn't need to reason or refine its search strategy, it just reads the answer off the results. The paper shows this instability in Figure 3 (Appendix A), where the reward curve for Search-R1 (using a real search engine) exhibits substantial fluctuation compared to the smoother ZEROSEARCH curve. The authors attribute this directly to uncontrollable document quality: "In the early stages of training, low-quality documents may prevent the policy model from developing a correct understanding of the task. In later stages, if the documents are too high-quality, the policy model may not be sufficiently challenged to continue improving" (Appendix A).
Prohibitively High API Costs: The authors quantify this in Section D, Table 8: a training run with approximately 64,000 search queries costs ~ $587 in API fees versus$ 0 for API costs (plus GPU rental) for the simulated approach. This gap widens dramatically with scale. The paper is explicit: "the API cost increases substantially with larger batch sizes or more training steps, making the method less scalable due to prohibitively high expenses" (Section D).

RL methods using static corpora miss the complexity of real-world interaction. Methods like Search-R1, R1-Searcher, and ReSearch that use local Wikipedia dumps as their retrieval corpus avoid API costs but introduce a different problem: they don't simulate the messiness of real web search. Real search engines return documents with varied provenance, quality, style, and relevance—and training on clean, pre-filtered Wikipedia articles doesn't prepare the model for this reality. The paper explicitly notes that these methods "fail to capture the complexities of real-world interaction" (Section 2.2). When the model is deployed and encounters actual search results with their noise, redundancy, and inconsistent formatting, it may struggle because it was trained in a sanitized environment.

Conflicting Demands: Quality Control vs. Realism

The paper's central motivational insight is that prior approaches face an unsolved tension between two desirable properties of a training environment:

Realism: The training environment should approximate deployment conditions. A model trained on curated Wikipedia excerpts will underperform when faced with messy web results. This pushes toward using real search engines.
Controllability: The training environment should allow systematic difficulty adjustment. A model that only sees high-quality results never learns to distinguish signal from noise, while a model that only sees noise may never learn the task. This pushes toward simulated environments.

Real search engines maximize realism but sacrifice controllability. Static corpora offer some controllability (through corpus selection) but sacrifice realism. The paper positions ZEROSEARCH as a framework that achieves both: it simulates real search results (realism) while allowing explicit, fine-grained control over document quality through prompt design and curriculum scheduling (controllability). This is accomplished through a single mechanism—the lightweight fine-tuned simulation LLM—that serves both goals simultaneously.

How This Paper Positions Itself

The paper positions ZEROSEARCH not as a fundamentally new training algorithm but as an environment design contribution—it's about what the policy model interacts with during RL, not how the RL update is computed. The key insight is that LLMs, having been trained on vast corpora that include web pages, Wikipedia articles, news, and reference materials, already possess enough world knowledge to generate plausible search engine results given a query. The paper cites Yu et al. (2022)'s observation that "LLMs are strong context generators" (Section 1, paragraph 3) as precedent for using LLMs to produce documents rather than retrieve them. What ZEROSEARCH adds is: (1) lightweight SFT to close the style gap between raw LLM generation and actual search engine output, (2) explicit training of the simulator to distinguish and produce both useful and noisy documents on command, and (3) a curriculum schedule that progressively increases noise to match the policy model's growing competence.

The relationship to prior work is one of complementary replacement. ZEROSEARCH does not claim to improve upon the RL algorithms themselves—it works with REINFORCE, PPO, and GRPO (Section 3.6, Table 5). It does not claim to improve upon the policy model architecture. It replaces one component—the retrieval environment—with a learned simulator, and shows that this substitution not only eliminates API costs but enables a training curriculum that would be impossible with a real search engine (since real engines don't accept instructions to "return noisy results"). The paper explicitly frames this as an alternative to Search-R1's real-engine approach: "This setting allows for a direct and equitable comparison between the real search engine and our simulated search engine" (Section 4.2).

The paper also positions itself within the broader trend of using LLMs as world simulators—replacing expensive or inaccessible real-world systems with learned generative models. Just as prior work has used LLMs to simulate user behavior, dialogue partners, or tool outputs, ZEROSEARCH uses an LLM to simulate a search engine. The paper suggests this paradigm could extend beyond search: if a fine-tuned LLM can simulate Google results well enough to train effective search policies, what other real-world systems could be similarly simulated for RL training?

The Hidden Assumption About LLM Knowledge Coverage

An important, unstated assumption underlies the entire approach: the simulation LLM must have sufficient world knowledge to generate relevant documents for the queries the policy model will issue. If the simulation LLM doesn't know the answer to a query, it cannot generate a useful document in response—and the policy model receives a noise-only signal that may not reflect what a real search engine would return. This assumption is partially validated by the paper's choice to include the input question and ground-truth answer in the simulation LLM's prompt during document generation (Table 2): "The user is trying to answer the question: [question] whose answer is [ground truth]." This means the simulation LLM receives oracle knowledge—it knows the correct answer—and simply needs to produce documents that either contain or don't contain that answer. This is simultaneously a strength (it guarantees that useful documents contain correct information) and a limitation (it assumes the simulator has access to ground-truth answers during training, which may not be available for all tasks).

The paper addresses this implicitly through model scaling results: a 3B simulation LLM already enables effective training, a 7B achieves Google parity, and a 14B surpasses it (Table 4). This scaling trend suggests that larger pretrained models have broader knowledge coverage, making them better simulators even without oracle answer access—though the current experiments all use the oracle-enhanced prompt.

Summary of the Gap

To synthesize: the paper addresses the problem that RL-based training of search-capable LLMs is gated by the cost and uncontrollability of real search engine access. Prior work either avoided RL entirely (prompt-based SFT), accepted high API costs and training instability (real-engine RL), or sacrificed realism for cost savings (static-corpus RL). ZEROSEARCH proposes a fourth path: replace the real search engine with a fine-tuned LLM that can generate controllable-quality documents, eliminating API costs while enabling a curriculum that progressively increases retrieval difficulty. The core bet is that an LLM's parametric knowledge, when refined through lightweight domain-specific fine-tuning, is sufficient to serve as a training-time proxy for a real search engine—and that the controllability gained through simulation actually yields better-trained policies than training on real, uncontrolled search results.

3. Technical Approach

3.1 Reader Orientation

ZEROSEARCH is a training framework that teaches a language model to use a search engine by having it practice against a simulated search engine—another, smaller language model fine-tuned to generate search-result-like documents—rather than calling the real Google API during training. The framework solves the problem that reinforcement learning for search is bottlenecked by two constraints: the financial cost of hundreds of thousands of live API calls, and the fact that real search results have uncontrollable quality, making it impossible to structure a smooth learning curriculum from easy to hard retrieval scenarios. The "shape" of the solution is a three-stage pipeline: first, fine-tune a small LLM to become a controllable document generator that can produce either useful or noisy search results on command; second, run standard RL training where the policy model's search queries go to this simulator instead of a real API; third, progressively increase the fraction of noisy documents throughout training so the policy model first learns the basics of querying and then adapts to increasingly difficult retrieval.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components connected in a training loop:

Simulation LLM ( $\pi_\psi$ ) — a frozen, fine-tuned language model (default: Qwen-2.5-14B-Instruct) that receives search queries generated by the policy model and returns either useful or noisy documents, controlled by a keyword in its input prompt. This replaces the real search engine during training.
Policy Model ( $\pi_\theta$ ) — the LLM being trained (e.g., Qwen-2.5-7B-Base). It receives a question, iteratively reasons and issues search queries within structured tags, receives simulated documents, and ultimately produces an answer. Its parameters are updated by RL.
Rollout Module — the orchestration logic that manages multi-turn interaction between the policy model and the simulation LLM, generates multiple response trajectories per prompt, applies the curriculum noise schedule, masks document tokens for loss computation, and computes rewards.
Reward Function and RL Optimizer — a rule-based F1 scoring mechanism that evaluates answer correctness, plus a choice of RL algorithm (REINFORCE, PPO, or GRPO) that updates $\pi_\theta$ based on the reward signal.

Information flows as follows: a training question enters → the policy model generates a multi-turn trajectory containing reasoning, search queries, and a final answer → each search query is intercepted and sent to the simulation LLM instead of a real API → the simulation LLM generates documents with quality controlled by the curriculum schedule → the completed trajectory is scored by the F1 reward function → the RL optimizer updates the policy model using the reward, applying loss masking to document tokens. At evaluation time, the simulation LLM is removed and the trained policy model interacts with the real Google Web Search API.

3.3 Roadmap for the Deep Dive

First, the formal RL objective (Equation in Section 3.1), which defines what is being optimized and introduces the key notation distinguishing the trainable policy model from the frozen simulation LLM.
Second, the training template (Section 3.2), because the structured tag-based interaction format is what enables the policy model to learn search as a discrete, interleaved reasoning-and-retrieval skill rather than a monolithic generation task—and the template's design choices shape everything downstream.
Third, the simulation LLM training (Section 3.3), since the simulator is the paper's core contribution and understanding how it learns to generate controllable-quality documents is essential for understanding the curriculum mechanism.
Fourth, the curriculum rollout strategy (Section 3.4), which is the training-time procedure that progressively degrades document quality according to an exponential schedule—this is the mechanism that enables smooth learning and is what separates ZEROSEARCH from simply swapping a real API for a static LLM.
Fifth, the reward design and loss masking (Sections 3.5 and 3.6), which covers the concrete supervision signal and the critical implementation detail of preventing document tokens from contaminating policy gradients.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training methodology paper whose core idea is that a fine-tuned LLM can serve as a drop-in replacement for a real search engine during RL training, and that the controllability gained through simulation—specifically, the ability to schedule document quality from easy to hard—produces better-trained policies than training on real, uncontrolled search results, while eliminating API costs entirely.

Reinforcement Learning Without a Search Engine: The Formal Objective

The paper begins by formalizing what it means to do reinforcement learning when the search engine is simulated by another LLM rather than being an external tool. The optimization objective is stated as:

$\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot|x; \pi_\psi)} \left[ r_\phi(x, y) \right] - \beta \, D_{KL} \left( \pi_\theta(y | x; \pi_\psi) \, \| \, \pi_{\text{ref}}(y | x; \pi_\psi) \right)$

where:

$\pi_\theta$ is the policy model being trained (the LLM that learns to search and answer),
$\pi_\psi$ is the simulation LLM that replaces the search engine—its parameters are frozen throughout training,
$\pi_{\text{ref}}$ is a reference policy (typically a frozen copy of the initial policy model) used for KL regularization,
$x \sim \mathcal{D}$ is a question sampled from the training distribution,
$y \sim \pi_\theta(\cdot|x; \pi_\psi)$ is a multi-turn trajectory generated by the policy model interacting with the simulation LLM,
$r_\phi(x, y)$ is the reward function that scores the final answer against the ground truth,
$\beta$ is a hyperparameter controlling the strength of KL regularization,
$D_{KL}(\cdot\|\cdot)$ is the Kullback-Leibler divergence between the current policy and the reference policy.

What this equation computes: The objective maximizes the expected reward of trajectories generated by the policy model $\pi_\theta$ when it interacts with the frozen simulation LLM $\pi_\psi$ , minus a penalty proportional to how far the policy has diverged from the reference model. The expectation is over the training question distribution and the policy's own stochastic generation. The KL term prevents the policy from drifting too far from its initial behavior, which is a standard stabilization technique in RL fine-tuning of LLMs. The critical notational detail is that the policy's generation distribution is conditioned on $\pi_\psi$ : $\pi_\theta(\cdot|x; \pi_\psi)$ means the model generates tokens autoregressively, but when it produces a <search> query, the subsequent <information> block is filled by the simulation LLM rather than by the policy model's own generation—those document tokens come from $\pi_\psi$ , not $\pi_\theta$ .

Why this form: The paper explicitly frames this as a standard RL objective adapted for a multi-agent setup where one agent (the simulator) is frozen. The key difference from a formulation where the search engine is external is that the simulation LLM is a learned function with its own distribution— $\pi_\psi$ is a generative model, not a deterministic lookup. This means the policy model is trained against a stochastic document distribution, which the paper argues is closer to the variability of real search results than a static corpus would be, while still being controllable. The KL penalty follows the standard PPO/GRPO formulation (Schulman et al., 2017; Shao et al., 2024) and is included for compatibility with those algorithms, though the paper primarily uses REINFORCE, which does not require the KL term in its simplest form. The notation $\pi_\theta(\cdot|x; \pi_\psi)$ is non-standard in the RL literature but serves an important purpose: it makes explicit that the policy's output during training is not purely its own generation but is interleaved with simulator outputs, which has consequences for gradient computation (see the loss masking discussion below).

Training Template: Structured Multi-Turn Interaction

The paper does not use supervised fine-tuning to teach the model how to search. Instead, following Search-R1 (Jin et al., 2025), it provides a multi-turn interaction template that structures each trajectory into three semantically distinct segments, enforced through XML-style tags. The full template is reproduced in Table 1:

Answer the given question. You must conduct reasoning inside  thinking and  response first
every time you get new information. After reasoning, if you find you lack some knowledge,
you can call a search engine by <search> query </search>, and it will return the top
searched results between <information> and </information>. You can search as many
times as you want. If you find no further external knowledge needed, you can directly provide
the answer inside <answer> and </answer> without detailed illustrations. For example,
<answer> Beijing </answer>. Question:

The question is appended at the end. During both training and inference, the policy model generates text that follows this structured format. The three stages are:

Reasoning stage (inside <thinking>...</thinking>): The model articulates its internal reasoning. This is where it decides what it knows and what it needs to look up. The paper notes that this reasoning is regenerated "first every time you get new information"—meaning after each search result comes back, the model re-thinks before deciding whether to search again or answer.
Search stage (inside <search>...</search>): If the model determines it needs external information, it issues a search query. During training, this query is intercepted and fed to the simulation LLM $\pi_\psi$ . During evaluation, it goes to the real Google Web Search API via SerpAPI. The template allows an arbitrary number of search turns: "You can search as many times as you want."
Answer stage (inside <answer>...</answer>): Once the model decides it has sufficient information, it produces its final answer without additional reasoning or search. The example in the template (<answer> Beijing </answer>) demonstrates that the answer should be concise—just the answer string—rather than a verbose explanation.

Why this structured format: The explicit separation of reasoning, searching, and answering serves several purposes. First, it creates a clear decision boundary for the RL agent: the model must learn when to search versus when to answer based on its current knowledge state. Second, it makes the search queries extractable for rerouting to the simulator—the rollout module can parse <search> tags and substitute simulated documents. Third, it means the reward function (Section 3.5) can focus solely on the content inside <answer> tags without needing to parse or evaluate the reasoning quality. Fourth, and critically, the paper observes that the model "consistently produces well-formed responses without explicit supervision" on format (Section 3.5)—the template alone is sufficient to teach the structure, and no auxiliary reward for format adherence is needed. This is a notable empirical finding: reinforcement learning on answer correctness alone generalizes to learning the correct interaction protocol.

An important detail: the template explicitly includes the instruction "first every time you get new information," which means the model is expected to produce a thinking block, then optionally a search block, then another thinking block after receiving results, then optionally another search block, and so on, until it finally produces an answer. This creates a natural multi-turn structure where each turn consists of thinking → [search] → (results) → thinking → [search] → ... → answer. The case studies in Table 9 show this pattern: the model reasons, searches, receives documents, reasons again based on the new information, searches again if needed, and finally answers.

The paper does not describe any special tokenization or embedding of these tags—they appear to be treated as regular text tokens generated by the policy model, with the <information> block being the only part generated externally (by the simulator). This means the model must learn to terminate its own generation with </search> and then the rollout module injects the simulated documents before allowing the model to continue generating.

Search Simulation Tuning: Building a Controllable Document Generator

This section describes how the paper takes a general-purpose LLM and transforms it into a simulation LLM $\pi_\psi$ that can produce search-engine-like documents with explicitly controllable quality. This is the core technical contribution of ZEROSEARCH and involves three sub-steps: data collection, quality labeling, and supervised fine-tuning.

Data collection via real search engine interaction. The paper first generates interaction trajectories by prompting an LLM (presumably a strong model, though the specific model is not stated in Section 3.3—the default simulator backbone is Qwen-2.5-14B-Instruct per Section 4.3) to engage with the real Google Web Search API in a multi-turn manner. The LLM issues queries, receives real search results, and continues until it produces a final answer, following the same structured template format described above. From these trajectories, the paper extracts query-document pairs: each time the model issued a <search> query and received <information> from the real API, that query and its corresponding documents are recorded as a training example. The number of trajectories and the specific data collection prompt are not described in detail in the main text; the paper states only that this is done "in a multi-turn manner until a final answer is reached."

Quality labeling by LLM-as-judge. For each extracted query-document pair, the paper uses an LLM as a judge to independently assess whether the document contains sufficient information to answer the corresponding query. The exact prompt for this judgment is not provided, but the binary decision rule is straightforward: if the judge determines the document provides enough information to answer the query, it is labeled useful; otherwise, it is labeled noisy. This creates a training dataset of (query, document) pairs with binary quality labels, where the documents themselves come from real search engine results and thus exhibit natural web text style, length, and formatting.

Why fine-tuning is necessary: bridging the style gap. The paper notes that a straightforward alternative—directly prompting an off-the-shelf LLM to generate documents—produces a "noticeable style gap compared to outputs from real search engines" (Section 3.3). Real search snippets have characteristic features: they are typically 20-40 words, contain partial sentences, include metadata like dates or source names, and have a telegraphic quality that differs from the fluent, complete-sentence style of standard LLM generation. The lightweight SFT described next is designed to close this gap by training the LLM on actual search engine outputs, making its generations stylistically indistinguishable from real search results. This is critical because if the policy model learns to exploit stylistic artifacts of simulator-generated documents (e.g., always looking for well-formed paragraphs), it may fail when deployed against real search results with different surface features.

Supervised fine-tuning procedure. The fine-tuning uses a structured prompt template, shown in Table 2:

You are the Google search engine.
Given a query, you need to generate five [useful / noisy] documents for the query.
The user is trying to answer the question: [question] whose answer is [ground truth].
Each document should contain about 30 words, and these documents should contain [useful /
noisy] information.
Query: [query]
[Useful / Noisy] Output:

Several design choices in this prompt deserve careful analysis:

Simulator persona: The simulation LLM is explicitly instructed "You are the Google search engine." This is not just cosmetic—it shifts the model's generation distribution toward the document styles it has seen in its pretraining data that are associated with search results, snippets, and web extracts.
Keyword-based quality control: The adjective [useful / noisy] appears three times in the prompt: once in the instruction ("generate five [useful / noisy] documents"), once in the content specification ("these documents should contain [useful / noisy] information"), and once in the output header ("[Useful / Noisy] Output:"). This repetition ensures the model attends strongly to the quality signal. The paper states that "the distinction between useful and noisy outputs can be effectively controlled by adjusting a few words in the prompt" (Section 3.3)—meaning that simply swapping "useful" for "noisy" in the template causes the model to generate documents of corresponding quality, without any architectural change or separate model.
Oracle knowledge injection: The simulation LLM receives the input question and its ground-truth answer in the prompt: "The user is trying to answer the question: [question] whose answer is [ground truth]." This is a critical design choice. When the simulator is asked to generate useful documents, it knows the correct answer and can produce documents that contain it. When asked to generate noisy documents, it knows the correct answer and can deliberately produce documents that do not contain it—or contain misleading, tangential, or irrelevant information. This oracle access is what enables precise control over document quality: the simulator can guarantee that useful documents are genuinely helpful and noisy documents are genuinely unhelpful, creating a clean training signal for the policy model.
Document count and length: The prompt specifies generating exactly five documents per query, each approximately 30 words. This matches the evaluation setting where exactly five results are returned per search (Section 4.3). The 30-word target produces documents of realistec length: real search snippets are typically 1-2 sentences, not paragraphs.
Broadening knowledge coverage: Including the question and answer in the prompt serves an additional purpose beyond quality control: it "broaden[s] the knowledge boundary of the simulation LLM" (Section 3.3). A smaller LLM like the 3B variant may not have memorized every fact, but providing the answer in-context allows it to generate documents about that fact regardless of its parametric knowledge. This mitigates the knowledge coverage problem discussed in Section 2: the simulator doesn't need to know everything; it needs to be able to generate plausible documents when given the facts.

Training hyperparameters. The paper specifies in Appendix F that the simulation LLM undergoes "lightweight SFT" using Qwen-2.5-3B-Instruct, Qwen-2.5-7B-Instruct, and Qwen-2.5-14B-Instruct as backbones, with a learning rate of $1 \times 10^{-6}$ for all sizes. The batch size, number of epochs, optimizer, and other training details are not specified in the main text or appendix, but the "lightweight" characterization and the fact that training completes in approximately 12 hours even for the 14B model (Section D, Table 8) suggests this is a relatively small-scale fine-tuning—likely a few thousand query-document pairs at most, given that 64,000 total queries are used across the entire RL training run and only a fraction of those would be needed for simulator training.

What the fine-tuned simulator can do. After SFT, the simulation LLM has two capabilities that an off-the-shelf LLM lacks: (1) it generates documents in a style that matches real search engine outputs, reducing the domain gap between training and deployment; (2) it accepts a binary control signal (useful vs. noisy) that determines whether the generated documents contain the information needed to answer the query. This second capability is what enables the curriculum rollout mechanism—without it, the simulator could only produce documents of uncontrolled quality, and the curriculum would not be possible.

Concrete examples of simulator outputs. Table 10 in Appendix E shows what the 14B simulator produces. For the query "Tour de France 2018 second place" (correct answer: Tom Dumoulin), the useful output includes a document stating "2nd – Tom Dumoulin (NED / Team Sunweb)" alongside other relevant race information, while the noisy output includes documents about prize money amounts and historical records from unrelated years—documents that are topically related to the Tour de France but do not answer the specific question. For the query "sequel to The Alienist Caleb Carr" (correct answer: The Angel of Darkness), the useful output includes "The sequel to the acclaimed national bestseller The Alienist, Caleb Carr's The Angel of Darkness is a breathtaking thriller set in 1897 New York," while the noisy output discusses the TV adaptation, the book's setting, and general discussion about the author—all related to The Alienist but none identifying the sequel. This demonstrates the simulator's ability to produce documents that are on-topic but either contain or omit the specific sought-after fact.

Rollout with Curriculum Search Simulation: Progressive Difficulty Scheduling

This section describes the mechanism by which document quality is progressively degraded during RL training. This is the innovation that transforms simulator-based training from a simple cost-saving measure into a better training procedure than using a real search engine.

The problem that curriculum solves. If the policy model always receives high-quality documents (where the answer is directly present in the results), it learns to simply extract and regurgitate the answer without developing genuine search skills—it becomes a reading comprehension model rather than a retrieval strategist. If it always receives low-quality documents, it may never learn the association between good queries and good results, and training may fail to converge. The curriculum approach addresses this by starting with mostly useful documents (allowing the model to learn the basic mechanics of issuing queries, parsing results, and formatting answers) and progressively increasing the proportion of noisy documents (forcing the model to refine its query strategy, combine information across multiple searches, and distinguish between relevant and irrelevant results).

The noise probability schedule. The curriculum is parameterized by a single scalar $p_i$ that determines the probability of generating noisy (low-quality) documents at training step $i$ . The schedule follows an exponential interpolation between a start probability $p_s$ and an end probability $p_e$ :

$p_i = p_s + \frac{b^{i/m} - 1}{b - 1}(p_e - p_s)$

where:

$p_i$ is the probability of generating noisy documents at the current training step $i$ ,
$p_s$ is the initial noise probability at step 0,
$p_e$ is the final noise probability at step $m$ ,
$i$ is the current training step (ranging from 0 to $m$ ),
$m$ is the total number of training steps,
$b$ is the exponential base (default value: 4).

What this equation computes: The noise probability at each training step is a convex combination of $p_s$ and $p_e$ , but the weighting is not linear—it follows an exponential function that stays close to $p_s$ in early training and accelerates toward $p_e$ in later training. Specifically, the term $(b^{i/m} - 1)/(b - 1)$ maps the linear progress fraction $i/m$ to a nonlinear weight on [0, 1]. When $i = 0$ , the fraction is 0 and $p_i = p_s$ . When $i = m$ , the fraction is 1 and $p_i = p_e$ . For intermediate steps, because $b = 4 > 1$ , the exponential $4^{i/m}$ grows slowly at first and then rapidly—for example, when $i/m = 0.5$ (halfway through training), $4^{0.5} = 2$ , yielding a weight of $(2-1)/(4-1) = 1/3$ , meaning $p_i$ is only one-third of the way from $p_s$ to $p_e$ even though half the training is done. At $i/m = 0.75$ , $4^{0.75} \approx 2.83$ , giving a weight of $(2.83-1)/3 \approx 0.61$ . This means the model spends the majority of training seeing relatively easy retrieval scenarios, with difficulty ramping up significantly only in the final quarter of training.

Why exponential scheduling (base 4): The paper does not explicitly justify the choice of $b = 4$ , but the motivation for exponential scheduling is implicit in the training dynamics. The policy model needs substantial exposure to well-structured search scenarios early on to learn the format and the basic query-answer mapping. Once these fundamentals are established (which likely requires many gradient updates due to the complexity of multi-turn generation), the model can begin handling noise. An exponential schedule concentrates the noisy phase toward the end of training, which means the model spends most of its parameter updates on clean scenarios and only experiences significant noise when its policy is already reasonably well-formed. This prevents the noise from derailing early learning while still providing a strong challenge signal in later stages. A linear schedule would introduce noise too quickly, potentially preventing the model from ever learning the core search task. A step function (switching from clean to noisy at some threshold) would create a sharp distribution shift that could destabilize training. The exponential schedule provides a smooth transition.

Per-step implementation. At each rollout step during training, when the policy model issues a search query, the rollout module samples a random number and compares it to $p_i$ . If the random number is less than $p_i$ , the simulator is given the "noisy" prompt variant; otherwise, it receives the "useful" prompt variant. The simulator then generates 5 documents of the specified quality, which are inserted into the trajectory between <information> tags. Importantly, this means that within a single trajectory containing multiple search turns, some searches may return useful documents and others may return noisy documents—the per-step noise probability creates heterogeneous trajectories where the model must sometimes deal with good and bad results intermixed. This is more realistic than binary all-useful or all-noisy trajectories and forces the model to evaluate each retrieved document's quality independently.

Model-specific noise parameters. The paper uses different $p_s$ and $p_e$ values for different policy model families and sizes, as specified in Appendix F:

For Qwen-2.5-7B (Base and Instruct): $p_s = 0$ , $p_e = 0.75$ . The model starts with only useful documents and ends with 75% probability of noise—meaning in the final training stages, only 25% of searches return useful documents. This is a aggressive curriculum that forces the 7B model to develop strong noise-robust search strategies.
For Qwen-2.5-3B (Base and Instruct): $p_s = 0$ , $p_e = 0.25$ . The 3B model, being smaller and potentially less capable of handling high noise, is trained with a milder curriculum reaching only 25% noise probability. This suggests the paper tuned noise levels to model capacity—smaller models need cleaner signals to learn effectively.
For LLaMA-3.2-3B (Base and Instruct): $p_s = 0.25$ , $p_e = 0.5$ . Interestingly, the LLaMA models start with 25% noise already, unlike the Qwen models which start at 0%. This may reflect differences in the base models' instruction-following ability or the fact that LLaMA models might overfit to clean signals if not exposed to some noise from the start. The ending noise probability of 50% is intermediate between the Qwen-3B and Qwen-7B settings.

These per-model choices are not ablated in the paper (no experiment varies $p_s$ and $p_e$ for a fixed model to show sensitivity), but the strong performance across all configurations suggests that the curriculum mechanism is robust to the specific parameter values as long as they provide a reasonable easy-to-hard progression.

Why a curriculum-based approach over alternatives. The paper contrasts the curriculum approach against a "random setup, where the probability of generating noisy documents is fixed at 0.5 throughout training" (Section 5.5, Table 6). This ablation is the key evidence that the curriculum, not just the simulation, is responsible for improved training. Two alternative designs are implicitly rejected:

Always-useful simulation: If the simulator always produced high-quality documents (equivalent to $p_s = p_e = 0$ ), the training would be a pure supervised signal where the model learns to extract answers from clean text. This might produce a model that performs well on simple questions but fails when real search results are noisy or ambiguous. The curriculum avoids this by eventually introducing noise.
Always-noisy simulation: If the simulator always produced low-quality documents ( $p_s = p_e = 1$ ), the training signal might be too weak for the model to ever learn the task. The curriculum avoids this by starting with clean signals.
Fixed-mix simulation (the random baseline): A constant 50% noise probability provides diversity but doesn't structure the learning process. The model must simultaneously learn the search format, query formulation, answer extraction, and noise discrimination—a harder multi-task learning problem. The curriculum decomposes this into sequential sub-problems: learn format and basic retrieval first, then learn noise robustness later. The experimental results (Table 6) confirm this decomposition produces better final performance.

Reward Design: F1-Based Scoring to Prevent Reward Hacking

The reward function is the sole source of supervision in the RL training pipeline. There is no dense reward for intermediate reasoning quality, search query quality, or format adherence—only the final answer is evaluated against the ground truth. This makes the design of the reward function critical: if it can be exploited, the policy model will find and exploit the exploit rather than learning to search.

The problem with exact match. The paper reports that during preliminary experiments, using Exact Match (EM) as the reward metric "often led to reward hacking: the policy model tended to produce excessively long answers to increase the chance of including the correct answer" (Section 3.5). This is a classic reward misspecification problem: if the reward is 1 whenever the prediction string contains the ground truth as a substring (or is identical after normalization), the optimal policy generates long lists of candidate answers, hoping one matches. For example, if the true answer is "Tom Dumoulin," the model might output "Tom Dumoulin, Geraint Thomas, Chris Froome, Romain Bardet, Jakob Fuglsang" to guarantee a match regardless of which rider actually came second. This behavior—generating many possible answers to cover the correct one—is reward-maximizing under EM but does not reflect genuine retrieval or reasoning.

The F1-based reward. To address this, the paper adopts an F1 score computed over word overlap between the prediction and ground truth:

$r_\phi(x, y) = \frac{2 \times I_N}{P_N + R_N}$

where:

$I_N$ is the number of overlapping words between the prediction string and the ground-truth answer string,
$P_N$ is the total number of words in the prediction,
$R_N$ is the total number of words in the ground truth.

What this equation computes: The F1 score is the harmonic mean of precision and recall at the word level. Precision is $I_N / P_N$ (what fraction of the prediction's words are correct), recall is $I_N / R_N$ (what fraction of the ground truth's words are captured), and F1 combines them as $2 \times \text{Precision} \times \text{Recall} / (\text{Precision} + \text{Recall})$ , which simplifies to the formula above. For a perfect match where every word in the prediction and ground truth aligns, $I_N = P_N = R_N$ and $r_\phi = 1$ . For the reward-hacking example above ("Tom Dumoulin, Geraint Thomas, Chris Froome, Romain Bardet, Jakob Fuglsang"), $I_N$ would be at most 2 (if "Tom Dumoulin" is the ground truth), $P_N$ would be around 10-12 (depending on tokenization), and $R_N$ would be 2, yielding $r_\phi = 4 / (12 + 2) = 0.29$ —much lower than the $r_\phi = 1$ for the concise correct answer. This penalizes verbosity and rewards concise, accurate answers.

Why F1 over alternatives. The choice of F1 over EM is specifically motivated by the reward hacking observation. An alternative like BLEU or ROUGE would also penalize verbosity but might be too lenient on partial matches, encouraging the model to produce answers that are somewhat related but not exactly correct. F1 at the word level provides a natural balance: it rewards the model for including all ground-truth words while penalizing inclusion of extraneous words. The paper mentions no additional reward terms—specifically, "We do not incorporate an additional reward for output format, as we observe that the model consistently produces well-formed responses without explicit supervision" (Section 3.5). This is a notable empirical result: the structured template alone is sufficient to teach the model to format its outputs correctly, and the F1 reward on answer content provides all the necessary learning signal.

What the reward does NOT capture. The F1 reward evaluates only the final <answer> content. It provides no signal about: whether the model's reasoning was logical, whether its search queries were well-formulated, whether it searched the right number of times, or whether it used the retrieved information appropriately. All of these intermediate skills must be learned indirectly through the reward signal propagating back through the multi-turn trajectory via the RL algorithm's credit assignment mechanism (e.g., REINFORCE's Monte Carlo return). This is the same "outcome supervision" paradigm used in DeepSeek-R1 (Guo et al., 2025) and Search-R1 (Jin et al., 2025): the model discovers effective intermediate strategies by trial and error, guided only by whether the final answer is correct.

Training Algorithm and Document Token Loss Masking

The paper's approach is compatible with multiple RL algorithms, and the training procedure includes a critical implementation detail—loss masking for document tokens—that prevents externally generated content from corrupting the policy gradient.

Algorithm compatibility. The paper experiments with three algorithms: REINFORCE (Williams, 1992), Proximal Policy Optimization (PPO; Schulman et al., 2017), and Group Relative Policy Optimization (GRPO; Shao et al., 2024). REINFORCE is the default and achieves the best performance (Table 5, discussed in Section 5.4). The training configuration for each algorithm is specified in Appendix F:

REINFORCE: Learning rate $1 \times 10^{-6}$ , 5 responses sampled per prompt. This is the simplest policy gradient algorithm: the policy is updated in the direction of advantage-weighted log-probabilities, where the advantage is the reward minus a baseline (typically the mean reward across the 5 samples).
GRPO: Learning rate $1 \times 10^{-6}$ , 5 responses sampled per prompt. GRPO computes advantages relative to the group mean, eliminating the need for a separate value function.
PPO: Policy learning rate $1 \times 10^{-6}$ , value model learning rate $1 \times 10^{-5}$ , Generalized Advantage Estimation (GAE) with $\lambda = 1$ and $\gamma = 1$ . PPO trains a separate value model to estimate expected returns and uses clipped surrogate objectives for stable updates. The GAE parameters $\lambda = 1$ and $\gamma = 1$ mean that advantages are computed as Monte Carlo returns without temporal-difference bootstrapping (since $\lambda = 1$ uses all n-step returns with exponential weighting that reduces to the full Monte Carlo return) and without discounting (since $\gamma = 1$ weights all future rewards equally). This is appropriate for episodic tasks with a single terminal reward.

The document token contamination problem. The rollout trajectory in ZEROSEARCH contains two types of tokens: tokens generated by the policy model $\pi_\theta$ (reasoning, search queries, answer) and tokens generated by the frozen simulation LLM $\pi_\psi$ (the document content between <information> tags). If gradients are computed with respect to all tokens in the trajectory, the policy model's parameters would be updated based on tokens it did not generate—the document tokens. This introduces noise because the policy model has no control over the simulator's output; the simulator may generate slightly different documents for the same query due to sampling, and these differences would create spurious variance in the policy gradient that is unrelated to the policy's own actions. Moreover, the simulator's distribution $\pi_\psi$ is frozen and not being optimized, so gradients through document tokens would effectively be trying to make the policy model's hidden state "undo" or "explain away" randomness introduced by the simulator—a clearly undesirable learning signal.

The loss masking solution. The paper introduces a simple mechanism: "we introduce a loss masking mechanism for document tokens, ensuring that gradients are only computed with respect to the model's own outputs" (Section 3.6). In practice, this means that when computing the policy gradient loss, the log-probability terms for tokens inside <information>...</information> blocks are multiplied by zero (masked out), so they contribute nothing to the gradient. Only the policy model's own generated tokens contribute to the parameter update. This is analogous to masking out padding tokens in standard sequence-to-sequence training, but applied selectively to externally injected content.

Why this matters. The ablation in Section C (Appendix C, Figure 4c and Table 7) demonstrates that removing loss masking causes a "substantial drop in the reward" during training and a significant decline in final model performance. For LLaMA-3.2-3B-Base, the average EM score across seven benchmarks drops from 36.07 to 34.53 when loss masking is removed (Table 7). The reward curves in Figure 4c show that training without masking is noisier and converges to a lower reward. The paper attributes this to "training instability" caused by including externally generated tokens in the loss. This is a crucial implementation detail that would be easy to overlook: because the RL algorithms operate on token-level log-probabilities, failing to mask document tokens means the policy model receives gradient updates for "generating" text it never actually produced, creating a mismatch between the action space being optimized and the actions the model actually controls.

Training infrastructure. The paper deploys the simulation LLM ( $\pi_\psi$ ) on 4 H20 GPUs and conducts policy model ( $\pi_\theta$ ) training on another 4 H20 GPUs (Section 4.3, Appendix F). The simulation server is described as having "inconsistent" GPU utilization—high during rollout generation, low during policy updates—which the paper acknowledges as a source of resource waste. The suggestion to "share a single simulation server across multiple RL training tasks" (Section D) is a practical optimization for deployment but is not evaluated in the paper.

Why This Architecture Over Alternatives

The paper addresses the "why this design" question implicitly through its ablation studies, but several comparative design choices are worth making explicit:

Why a frozen simulator rather than joint training? The simulation LLM $\pi_\psi$ is fine-tuned once and then frozen during RL training. An alternative would be to jointly train both models—update $\pi_\psi$ to become a better adversary as the policy model improves, creating a self-play dynamic. The paper does not explore this, and the frozen design has two advantages: (1) stability—the policy model's learning target doesn't shift under it, and (2) computational efficiency—the simulator can run on separate hardware and its outputs can potentially be cached or precomputed.

Why use oracle answers in the simulator prompt? The simulation LLM receives the ground-truth answer during document generation (Table 2). An alternative would be to have the simulator generate documents based solely on its parametric knowledge, without oracle access. The paper's choice means the simulator can always produce correct useful documents regardless of its own knowledge limitations—this makes the training signal reliable. The downside is that the method requires ground-truth answers for all training questions, which limits applicability to tasks where answers are known. The paper does not explore whether oracle-free simulation (relying on the LLM's own large pretraining knowledge) would work—the scaling results in Table 4, where larger simulators perform better, suggest that parametric knowledge coverage is indeed a factor, but the oracle prompt makes the simulator's job easier at all sizes.

Why REINFORCE over PPO/GRPO as the default? Table 5 shows REINFORCE achieves the best average performance (34.47) compared to GRPO (33.17) and PPO (32.67) on Qwen-2.5-3B-Base. The paper states this is "consistent with the observations in Search-R1" (Section 5.4), suggesting that simpler policy gradient methods may be more stable for this particular multi-turn, tool-use training setup. PPO and GRPO introduce additional complexity (value function estimation, clipped objectives, group-based advantages) that may not be beneficial when the reward signal is sparse and the action space includes tool interactions. The paper does not deeply analyze why REINFORCE outperforms here—it treats algorithm choice as an empirical finding rather than a theoretical claim.

Why the F1 reward rather than a learned reward model? The paper uses a simple rule-based F1 score rather than training a learned reward model (e.g., an outcome-supervised classifier that predicts answer correctness). Rule-based rewards are deterministic, require no training, and cannot be gamed through reward model over-optimization. The trade-off is that F1 provides a sparse, surface-form signal that doesn't capture semantic equivalence—"London, England" and "London" would have imperfect F1 overlap despite being equivalent answers. The paper's evaluation metric is Exact Match (after normalization), which handles some of this through answer normalization, but the training reward is F1. This creates a slight mismatch between training objective and evaluation metric (train on F1, evaluate on EM), but the paper does not discuss this gap or its implications.

4. Key Insights and Innovations

Innovation 1: Simulated Search Engines as a Controllable Training Environment, Not Just a Cost-Saving Hack

The paper's most fundamental conceptual move is reframing the relationship between RL training and search engines. Prior work treated the search engine as a fixed external resource—something to be called, paid for, and tolerated in its unpredictability. Whether doing prompt-based RAG (Yu et al., 2022; Press et al., 2023), SFT on retrieval trajectories (Asai et al., 2024; Li et al., 2024), or RL with live APIs (Search-R1 by Jin et al., 2025; DeepResearcher by Zheng et al., 2025; WebThinker by Li et al., 2026), the field's default assumption was that training a model to search required access to actual search results—the messier and more realistic, the better. The search engine was infrastructure, not a design variable.

ZEROSEARCH's innovation is to treat the search engine as a trainable component of the RL environment whose behavior can and should be optimized for pedagogical purposes. This is a fundamental shift, not an incremental refinement: it moves the search simulator from a "cheaper approximation of reality" (the obvious, cost-motivated framing) to a "strictly better teacher than reality" (the more interesting, capability-motivated framing). The paper makes this case explicitly through the reward curves in Figure 2a-b and Appendix A, Figure 3: training against a real search engine produces noisier, less stable reward trajectories than training against the controlled simulator, precisely because reality doesn't provide a curriculum. Real search results are sometimes too easy (the answer is in the first snippet), sometimes too hard (all top-5 results are irrelevant), and the timing of these difficulty fluctuations is arbitrary with respect to the model's learning trajectory. The simulator, by contrast, can match difficulty to competence.

This reframing has implications beyond search. If a fine-tuned LLM can serve as a better training environment than the real system it simulates, the same principle applies to any RL setting where the environment is expensive, uncontrollable, or poorly structured for learning: tool-use APIs, database queries, code execution environments, even physical simulators. The paper doesn't make this generalization explicitly, but the conceptual machinery is there: a learned environment model that accepts a difficulty control signal enables curriculum learning in domains where real environments are "one-size-fits-all" in their difficulty. This positions ZEROSEARCH not just as an efficient training method for search-augmented LLMs, but as a case study in a broader principle—train against a controllable teacher, deploy against the real world—that could reshape how the field thinks about environment design for language model training.

The evidence that the simulator is better, not just cheaper, comes from two sources. First, the performance comparison in Table 3: ZEROSEARCH-base achieves 40.93% average EM versus Search-R1-base's 39.51% across seven benchmarks (Qwen-2.5-7B), and the gap widens for the 3B models (34.47% vs. 32.81%). Second, the cost analysis in Table 8: the simulation approach costs $17.70–$ 70.80 in GPU time versus $586.70 for real API calls for the same training run, while producing better or equal models. These two facts together—better performance at ~8–33× lower cost—make the case that controllability is not merely economical but pedagogically superior. If the simulator were just a cheap approximation, we'd expect some performance degradation in exchange for cost savings. The absence of that degradation (and, in many cases, improvement) is the paper's strongest argument for its central thesis.

Innovation 2: The Curriculum Rollout Mechanism as Difficulty-Conditioned Environment Design

The paper introduces an exponential curriculum schedule (Equation 1, Section 3.4) that progressively degrades simulator document quality throughout training, controlled by start and end noise probabilities $p_s$ and $p_e$ with an exponential base $b = 4$ . This is not a new algorithmic idea—curriculum learning dates back to Bengio et al. (2009)—but its application here reveals something non-obvious about the relationship between search training and document quality that prior work had missed.

Before ZEROSEARCH, the dominant assumption in RL-based search training was that document quality variation is noise to be averaged over. Search-R1 (Jin et al., 2025) and related methods simply called the real search API on every rollout and accepted whatever came back, on the implicit assumption that sufficient training samples would smooth out quality fluctuations. The paper's Appendix A makes explicit what was wrong with this assumption: "In the early stages of training, low-quality documents may prevent the policy model from developing a correct understanding of the task. In later stages, if the documents are too high-quality, the policy model may not be sufficiently challenged to continue improving its reasoning capability." This is a stage-dependent optimal difficulty argument: what constitutes a productive training signal changes as the model learns. Early training needs clean examples to establish the basic search-format-answer mapping; late training needs noisy examples to force discrimination and refinement. A fixed distribution (whether real or simulated) cannot satisfy both requirements simultaneously.

The curriculum mechanism is conceptually significant because it decomposes the search learning problem into sequential sub-problems that were previously entangled. The policy model doesn't have to simultaneously learn how to search (format, query formulation, multi-turn interaction) and when to distrust search results (noise discrimination, cross-referencing, deciding to search again). Instead, it learns the mechanics first against clean signals, then layers on skepticism and verification as noise increases. This decomposition explains why ZEROSEARCH's training curves (Figure 2a-b) are smoother than Search-R1's: the model isn't fighting random quality fluctuations while trying to learn the basics.

The ablation in Table 6 provides the critical evidence that the curriculum, specifically, is responsible for the gains. When document quality is randomly sampled at a fixed 50% noise probability throughout training (the "random" baseline), average performance drops from 34.47% to 32.59% for Qwen-2.5-3B-Base and from 36.07% to 34.84% for LLaMA-3.2-3B-Base. These gaps (~2 percentage points) represent a substantial fraction of the total gain over Search-R1, confirming that the temporal structure of noise, not just its presence, determines training effectiveness. The fact that the curriculum parameters differ across model families (Qwen-2.5-7B gets $p_s=0, p_e=0.75$ ; Qwen-2.5-3B gets $p_s=0, p_e=0.25$ ; LLaMA-3.2-3B gets $p_s=0.25, p_e=0.5$ —Appendix F) further reinforces that optimal difficulty scheduling is model-specific, not a one-size-fits-all recipe. Smaller or differently-architected models need different noise trajectories, suggesting that the curriculum hyperparameters encode something about model capacity that would be worth investigating systematically.

This innovation connects conceptually to the broader literature on compute-optimal training (Hoffmann et al., 2022) but applied to environment difficulty rather than model size. Just as Chinchilla scaling laws revealed that optimal pretraining allocates compute between model parameters and data tokens according to a specific ratio, ZEROSEARCH reveals that optimal RL training allocates experience between easy and hard retrieval scenarios according to a specific schedule. The exponential base $b = 4$ concentrates noisy experience in the final quarter of training, which is analogous to the Chinchilla finding that most pretraining compute should go to data rather than parameters—both are about front-loading the easier/foundational learning before introducing complexity.

Innovation 3: Demonstrating That LLM Parametric Knowledge Is Sufficient for Search Simulation When Augmented with Oracle-Enhanced Prompting

The paper makes a specific, empirically-backed claim about LLM capabilities that contradicts a reasonable prior assumption: you do not need a retrieval corpus, vector database, or external knowledge source to simulate a search engine for training purposes—an LLM's parametric knowledge, when amplified by oracle answer access in the prompt, is sufficient. This matters because it radically simplifies the deployment requirements for RL-based search training. Prior work using static corpora like Wikipedia (Search-R1, R1-Searcher by Song et al., 2025; ReSearch by Chen et al., 2026) implicitly assumed that simulation requires access to the same factual content a real search engine would index. ZEROSEARCH shows this assumption is unnecessarily restrictive.

The innovation is not that LLMs contain knowledge—that's well-established. It's that the quality of that knowledge, when channeled through a structured prompt that includes ground-truth answers, is sufficient to serve as a training signal for search behavior—and that this sufficiency scales with model size in a predictable way (Table 4). A 3B simulator already enables effective training (34.47% average EM on Qwen-2.5-3B-Base policies, Table 4, SFT-3B row); a 7B simulator achieves parity with Google Search (33.53% vs. 32.81%); a 14B simulator surpasses it (34.47% vs. 32.81%). This scaling relationship—performance improves monotonically with simulator size—is the paper's strongest evidence that parametric knowledge quality, not just the oracle prompt, drives the effect. If the oracle prompt were doing all the work, simulator size wouldn't matter, since all simulators receive the same ground-truth answer. The fact that larger simulators produce better-trained policies implies they generate more realistic, diverse, or contextually appropriate documents around that answer.

The conceptual implication extends beyond search. This result suggests a general principle for building RL training environments with LLMs: an LLM can simulate any tool or system whose outputs are text, provided it has (or can be given) enough context to ground those outputs. The oracle answer in the prompt serves as a form of "cheat sheet" that compensates for gaps in parametric knowledge—the simulator doesn't need to know every fact about the Tour de France; it just needs to know how to write a plausible search snippet about Tom Dumoulin's 2018 performance when told the answer is "Tom Dumoulin." This is a different capability than factual recall; it's closer to constrained text generation given a target fact, which LLMs are demonstrably good at.

However, the innovation comes with a sharp boundary that the paper does not fully explore: the method requires ground-truth answers for all training questions (Table 2 includes "whose answer is [ground truth]"). This limits applicability to tasks where answers are known in advance—question answering with labeled datasets, but not open-ended generation or tasks where the "answer" is multi-faceted or subjective. Whether the method would work without oracle answers, relying purely on the simulator's parametric knowledge, is an open question. The scaling results suggest that larger simulators have better knowledge coverage and might approach oracle-level quality without explicit answer injection, but the paper provides no direct evidence for this. The innovation is therefore partially bounded: it shows what's possible with oracle access, and leaves the harder problem (fully unsupervised simulation) to future work.

Innovation 4: Identifying and Solving the Document Token Gradient Contamination Problem

The paper identifies a subtle technical problem that arises specifically when RL is applied to multi-agent, interleaved-generation settings, and provides a simple, effective solution. The problem: in a trajectory containing both policy-generated tokens and simulator-generated document tokens, computing policy gradients over all tokens causes the policy model to receive gradient updates for text it did not produce. The solution: loss masking over document tokens ensures the policy is optimized only for its own actions. The ablation (Appendix C, Figure 4c and Table 7) demonstrates that removing masking causes a significant performance drop—LLaMA-3.2-3B-Base average EM falls from 36.07% to 34.53%.

At first glance, this appears to be an implementation detail rather than an intellectual contribution. But it qualifies as an innovation because it identifies and names a failure mode that will recur in any RL setting where a language model interacts with external text-generating systems. The problem is not specific to ZEROSEARCH; it would affect any setup where RL training involves tool calls that return text injected into the generation context. A policy model that calls a calculator and receives a numeric result, or calls a translation API and receives translated text, or calls a code interpreter and receives execution output, faces the same contamination problem. The paper's contribution is to isolate this issue, demonstrate its severity empirically, and provide the straightforward fix of loss masking—establishing a pattern that future work can adopt.

What makes this more than an engineering note is the diagnostic value of the ablation. The fact that contamination causes a 1.5-point EM drop (Table 7) and visibly noisier training (Figure 4c) reveals how much spurious variance externally generated tokens introduce into the policy gradient. This is a measurable quantity that future work can use to evaluate the quality of environment simulations: if the variance contributed by simulator tokens is high relative to the policy's own action variance, the training signal is being dominated by environment noise. Loss masking is the solution, but the diagnostic—comparing masked vs. unmasked training performance—provides a general tool for assessing whether an environment's outputs are contaminating the policy gradient.

The paper does not claim this as an original algorithmic contribution (loss masking is a standard technique in sequence-to-sequence training), but its application to the RL-for-tool-use setting, combined with the empirical demonstration of its necessity, establishes it as a required component of the training recipe that would not have been obvious a priori. Given that multi-agent and tool-augmented LLM training is a rapidly growing area, identifying and naming this pitfall serves as a service to the field—preventing others from rediscovering it through failed experiments.

Innovation 5: Outcome-Only Reward Is Sufficient for Learning Multi-Turn Search and Format Adherence

The paper makes a striking empirical observation: the policy model learns to produce correctly formatted, multi-turn search trajectories using only an F1 reward on the final answer content, with no auxiliary rewards for format, query quality, or search behavior. The training template (Table 1) specifies the expected output structure—thinking, searching, and answering within XML tags—but the reward function (Section 3.5) never checks whether the model followed this structure. The model could, in principle, ignore the tags entirely, output a direct answer without any search, and still receive the same F1 reward. That it instead learns the full interaction protocol is a discovery about what RL with outcome supervision can achieve, not a designed property of the system.

Why is this conceptually significant? Because it challenges a common assumption in the RL-for-LLM literature: that complex, multi-step behaviors require either dense intermediate rewards or supervised fine-tuning on demonstration trajectories to bootstrap. Search-R1 (Jin et al., 2025) and DeepSeek-R1 (Guo et al., 2025) used outcome-only rewards for reasoning, but reasoning is purely internal—the model generates a chain of thought and then an answer, all within its own generation. ZEROSEARCH extends this to a setting where the model must produce specific, structured outputs (search queries inside tags) that trigger external tool calls and receive external information—a fundamentally different interaction pattern. The model has to learn not just to reason, but to interact with an external system according to a protocol, purely from the downstream reward on answer correctness.

The case studies in Table 9 (Appendix E) show the result: the trained model produces clean, multi-turn trajectories where it reasons, issues targeted search queries, parses returned documents, decides whether to search again or answer, and formats its final answer correctly. None of this behavior was explicitly rewarded. The paper states that "We do not incorporate an additional reward for output format, as we observe that the model consistently produces well-formed responses without explicit supervision" (Section 3.5)—a finding delivered almost as an aside, but one with significant implications for how simple reward structures can be in RL-based tool-use training.

This connects to a broader question in RL research: how much behavioral complexity can emerge from sparse, outcome-level rewards when the policy model has strong language priors? The answer from ZEROSEARCH is "more than you might expect." The template provides the syntax of interaction (what tags to use), and the RL process discovers the semantics (when to search, what to search for, how to integrate results) purely through trial and error guided by answer correctness. This suggests that language models have strong inductive biases toward following structured interaction protocols when those protocols are specified in natural language in the prompt—the prompt acts as a kind of "action space specification" that the RL process then optimizes within. The practical implication is that future work on tool-use RL may not need to design complex reward functions with intermediate checks; a well-specified template plus outcome supervision may suffice for surprisingly complex behaviors.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on seven question-answering benchmarks, following the setup in Search-R1 (Jin et al., 2025): (1) Single-Hop QA: NQ (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and PopQA (Mallen et al., 2023); (2) Multi-Hop QA: HotpotQA (Yang et al., 2018), 2WikiMultiHopQA (Ho et al., 2020), MuSiQue (Trivedi et al., 2022), and Bamboogle (Press et al., 2023). For fine-tuning-based approaches, the training sets of NQ and HotpotQA are merged to create a unified training dataset; evaluation is conducted on all seven datasets to assess both in-domain (NQ, HotpotQA) and out-of-domain generalization.
Base model(s). Experiments use three model families: Qwen-2.5-7B (Base/Instruct), Qwen-2.5-3B (Base/Instruct) (Yang et al., 2024), and LLaMA-3.2-3B (Base/Instruct) (Grattafiori et al., 2024). Both base and instruction-tuned variants are evaluated to test whether ZEROSEARCH works without relying on pre-existing instruction-following capabilities—the paper explicitly compares reward curves for both in Appendix B, Figure 4a-b, showing that base models start with lower rewards but "demonstrate greater reward improvements and reach performance levels comparable to their instruction-tuned counterparts."
Metrics. The primary evaluation metric is Exact Match (EM) after normalization, following Search-R1: "A prediction is deemed correct if its normalized form exactly matches any of the normalized ground-truth answers" (Section 4.1). During training, the reward function uses F1 score over word overlap (Section 3.5) to prevent reward hacking—this creates a train-vs-eval objective mismatch (train on F1, evaluate on EM) that the paper does not discuss but is consistent with Search-R1's setup. All results in tables and figures are reported as EM percentages on the test splits.
Baselines. The paper compares against three categories: (1) Vanilla prompting methods: Direct Answer, Chain-of-Thought (CoT), and standard Retrieval-Augmented Generation (RAG). (2) Advanced RAG methods: RAgent (Li et al., 2025) and Search-o1 (Li et al., 2025), both of which iteratively search for relevant information. (3) RL tuning methods: R1 (trained to reason purely from internal knowledge, following DeepSeek-R1 by Guo et al., 2025) and Search-R1 (Jin et al., 2025), which trains the policy model to interact with a real search engine during inference. Among RL-based search baselines, the paper compares only with Search-R1 because "it avoids complex reward design, data selection, or elaborate training pipelines" (Section 4.2), enabling a direct comparison between real and simulated search engines. For prompt-based baselines, only Instruct models are used since Base models "typically struggle to follow task instructions" (Section 4.3); for RL-based methods, both Base and Instruct are evaluated. All methods use Google Web Search via SerpAPI as the search engine during evaluation with exactly five retrieved documents per search to ensure fair comparison.
Generation budget / compute accounting. The paper does not formalize a unified compute budget for comparing methods, as this is not a compute-scaling study. Instead, fairness is ensured through fixed evaluation conditions: all methods use the same search API with the same result count (5 documents per query) during inference. Training costs are reported separately in Section D (Table 8) as a financial comparison: approximately 64,000 search queries are generated over 200 training steps with batch size 64 and 5 rollouts per prompt, costing ~ $587 in API fees for the real-engine approach versus$ 17.70–$70.80 in GPU rental for the simulated approach. The paper explicitly notes that "the API cost increases substantially with larger batch sizes or more training steps" (Section D), making the cost comparison a function of training scale rather than a fixed budget.
Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing. The 200-step training run uses a fixed merged training set (NQ + HotpotQA), and evaluation is conducted once on the standard test splits of all seven datasets. The reward curves (Figures 2, 3, 4) show mean training reward over steps but do not include error bars or confidence intervals. Given test set sizes (NQ and TriviaQA are relatively large; Bamboogle and MuSiQue are smaller), the absence of variance estimates makes it difficult to assess whether observed differences—often in the 1-3 percentage point range—are statistically reliable. This is a notable methodological gap, particularly for the out-of-domain datasets where test set sizes may be small.

Main Quantitative Results

Overall Performance: ZEROSEARCH vs. All Baselines (Table 3)

The headline result from Table 3 is that ZEROSEARCH-base using Qwen-2.5-7B achieves an average EM of 40.93% across all seven benchmarks, outperforming all baseline methods including Search-R1-base (39.51%), which uses a real search engine during training. The gap is modest but consistent: ZEROSEARCH-base exceeds Search-R1-base on 5 of 7 datasets (TriviaQA: 66.40% vs. 61.40%; PopQA: 60.40% vs. 54.60%; HotpotQA: 32.00% vs. 31.20%; 2WikiMultiHopQA: 34.00% vs. 37.20%—this one favors Search-R1; MuSiQue: 18.00% vs. 18.20%—essentially tied; Bamboogle: 33.33% vs. 30.56%). For the instruction-tuned variant, ZEROSEARCH-inst achieves 39.08% versus Search-R1-inst at 38.17%, again a small but consistent advantage. These gaps are in the 1-3 percentage point range and lack statistical confidence intervals, so the claim of outperformance should be understood as directional rather than definitive.

The more striking pattern is the gap between any search-capable method and non-search baselines. Direct Answer achieves only 15.17% average with Qwen-2.5-7B; CoT improves to 17.37%; standard RAG reaches 25.51%. The RL-based search methods (Search-R1 and ZEROSEARCH) essentially double the performance of the strongest non-RL baseline (RAG), confirming that learned multi-turn search—whether trained against real or simulated engines—provides substantial gains over one-shot retrieval.

The paper also demonstrates cross-model-family and cross-scale generalizability (Table 3, lower sections). For Qwen-2.5-3B-Base, ZEROSEARCH achieves 34.47% average versus Search-R1's 32.81%—a 1.66-point gap. For LLaMA-3.2-3B-Base, ZEROSEARCH achieves 36.07% versus Search-R1's 34.21%—a 1.86-point gap. The instruction-tuned variants show similar patterns (31.70% vs. 28.13% for Qwen-2.5-3B-Inst; 30.98% vs. 30.48% for LLaMA-3.2-3B-Inst). Critically, performance scales with policy model size: Qwen-2.5-7B-Base (40.93%) substantially outperforms Qwen-2.5-3B-Base (34.47%), which in turn outperforms LLaMA-3.2-3B-Base (36.07%)—though the LLaMA-3B achieving higher than Qwen-3B suggests architecture and pretraining differences matter beyond raw parameter count.

A notable anomaly: ZEROSEARCH appears to sometimes underperform Search-R1 on specific datasets for instruction-tuned models, notably PopQA for Qwen-2.5-7B-Inst (48.80% vs. 51.60%), HotpotQA for LLaMA-3.2-3B-Inst (22.80% vs. 21.00%—ZEROSEARCH wins, but Search-R1 wins on 2Wiki at 20.40% vs. 21.40%), and Bamboogle for Qwen-2.5-7B-Inst (27.78% vs. 26.39%). There is no clear pattern to these reversals, and the paper does not analyze which types of questions benefit more from real versus simulated training.

Training Dynamics: Reward Curve Comparison (Figures 2a-b and Figure 3)

The paper presents reward curves comparing ZEROSEARCH and Search-R1 during training to support the claim that simulated search provides a more stable learning signal. Figure 2a (Qwen-2.5-3B-Base) shows ZEROSEARCH starting at approximately 0.35 reward versus Search-R1 at approximately 0.25, then both rising to converge around 0.45–0.50 by step 200, with ZEROSEARCH showing noticeably less variance—particularly in the 75-150 step range where Search-R1 exhibits larger downward spikes. Figure 2b (Qwen-2.5-3B-Inst) shows a similar pattern: ZEROSEARCH starts lower (~0.28) but rises more smoothly to approximately 0.48-0.50, while Search-R1 starts higher (~0.33) but fluctuates more, ending at approximately 0.45. The paper interprets this as evidence that "the curriculum rollout mechanism... helps the model gradually master search tool usage" (Section 5.1).

Figure 3 in Appendix A extends this comparison to LLaMA-3.2-3B and Qwen-2.5-7B models. For LLaMA-3.2-3B-Base (Figure 3a), ZEROSEARCH shows a monotonic reward increase from ~0.35 to ~0.50, while Search-R1 plateaus around step 75-125 before rising again—a pattern the paper attributes to document quality variability disrupting learning. For Qwen-2.5-7B-Base (Figure 3c), both curves are similar, with ZEROSEARCH slightly smoother. The paper's narrative is that simulation provides stability, but an alternative interpretation is that the real-engine curves are noisier simply because real search results vary—and that this variability, while creating training instability, might produce more robust policies if sufficient training steps were provided. The paper does not explore training for more steps to see whether Search-R1 eventually catches up or surpasses ZEROSEARCH when given enough samples to average over real-result noise.

Critically, the reward curves measure training reward, not evaluation performance—training reward is computed by the F1 scoring function on the training distribution, which may not correlate perfectly with EM on held-out test sets. The paper does not provide validation curves showing test-set performance during training, so we cannot assess whether the smoother ZEROSEARCH training curves translate to better generalization or simply reflect less overfitting to the training distribution.

Simulation LLM Scaling: How Simulator Size Affects Policy Performance (Table 4)

Table 4 presents one of the paper's most practically significant results: how the quality of the simulation engine affects the trained policy model's performance. Using Qwen-2.5-3B-Base as the fixed policy model:

SFT-14B simulator: 34.47% average EM
SFT-7B simulator: 33.53% average EM
SFT-3B simulator: 30.47% average EM
Prompt-14B simulator (no fine-tuning, only prompting): 31.50% average EM
Prompt-7B simulator: 31.39% average EM
Prompt-3B simulator: 29.44% average EM
Google Search (real engine): 32.81% average EM
Base Model (no RL training at all): 8.82% average EM

Several patterns demand attention. First, SFT-7B achieves parity with Google Search (33.53% vs. 32.81%), and SFT-14B surpasses it (34.47% vs. 32.81%). This is the paper's strongest evidence that simulation can match or exceed real-engine training. Second, fine-tuning provides a substantial boost over prompting alone: SFT-3B (30.47%) outperforms Prompt-3B (29.44%) by ~1 point, SFT-7B (33.53%) outperforms Prompt-7B (31.39%) by ~2 points, and SFT-14B (34.47%) outperforms Prompt-14B (31.50%) by ~3 points. The gap between fine-tuned and prompted simulators grows with model size, suggesting that larger models benefit more from style adaptation—their raw generation is further from search-engine style, so the SFT correction provides proportionally more value.

Third, performance improves monotonically with simulator size, both for prompted and fine-tuned variants. This supports the paper's implicit argument that larger models have broader and deeper parametric knowledge, enabling them to generate more realistic and diverse documents that better approximate real search results. However, the gains diminish: SFT-3B to SFT-7B provides a +3.06-point improvement, while SFT-7B to SFT-14B provides only +0.94 points. This suggests diminishing returns to simulator scale, though the paper does not test whether a 32B or 70B simulator would provide further gains.

Fourth, and concerning for the paper's thesis: Prompt-14B (31.50%) underperforms Google Search (32.81%), meaning that without fine-tuning, even a 14B model used purely through prompting cannot match real search engine training. This undermines the simpler version of the paper's claim—"LLMs can simulate search engines"—and supports the more nuanced version—"LLMs can simulate search engines if fine-tuned on real search outputs to close the style gap." The SFT step is thus not optional; it is load-bearing.

A limitation: Table 4 tests only Qwen-2.5-3B-Base as the policy model. It does not show whether the simulator scaling pattern holds for other policy model families (LLaMA) or sizes (7B policies). The paper implies that the 14B simulator is the default for all experiments (Section 4.3), but the simulator scaling results are shown only for the 3B policy. This leaves open the question of whether larger policy models would benefit equally from larger simulators, or whether the simulator's benefit saturates differently depending on policy capacity.

Interaction Turns Analysis: How Search Behavior Evolves During Training (Figure 2c)

Figure 2c shows both the training reward and the average number of interaction turns (search queries per trajectory) throughout training for LLaMA-3.2-3B-Base. The paper uses this to analyze the policy model's evolving search strategy:

Early phase (steps 0-50): Interaction turns drop sharply from approximately 2.2 to 1.8, while reward increases slowly from ~0.1 to ~0.3. The paper interprets this as the model initially producing "redundant interactions" because it "lacks knowledge of how to properly invoke the search engine," then quickly learning "the correct format and begins to eliminate unnecessary steps" (Section 5.3). This is consistent with the curriculum design: early training uses mostly useful documents ( $p_s = 0.25$ for LLaMA), so the model can extract answers efficiently without needing multiple searches.
Middle phase (steps 50-150): Both interaction turns and reward increase sharply—turns rise from ~1.8 to ~2.1, reward from ~0.3 to ~0.45. The paper attributes this to the model becoming "capable of effectively retrieving relevant documents and ultimately achieving correct answers, resulting in higher rewards" (Section 5.3). The increase in interaction turns during this phase is interesting: it suggests the model is learning to search more as it gets better, not less—likely because it's discovering that multiple targeted searches produce better results than a single broad query.
Late phase (steps 150-200): Both metrics stabilize, with reward around 0.45-0.48 and turns around 2.0-2.1. The paper notes that "although the reward appears stable in the later stages of training, the underlying task difficulty continues to rise due to the curriculum rollout mechanism. Therefore, the policy must continuously refine its strategy and improve its reasoning ability to maintain the high reward score" (Section 5.3). This is a crucial point: a flat reward curve in the presence of increasing noise actually indicates improving capability—the model is maintaining performance on a harder task.

This analysis provides behavioral evidence that the model is learning genuine search strategies, not just memorizing answers or exploiting surface patterns. However, it is limited to a single model (LLaMA-3.2-3B-Base) and does not compare interaction turn patterns across ZEROSEARCH and Search-R1 to show whether simulation-trained models develop systematically different search behaviors (e.g., more or fewer queries, different query lengths, different stopping criteria).

Algorithm Compatibility: REINFORCE vs. GRPO vs. PPO (Table 5)

Table 5 compares three RL algorithms within the ZEROSEARCH framework using Qwen-2.5-3B-Base as the policy model:

REINFORCE: 34.47% average EM (best)
GRPO: 33.17% average EM
PPO: 32.67% average EM

The differences are modest—a 1.8-point spread between best and worst—but consistent across most individual datasets. REINFORCE leads on NQ (43.00% vs. 39.40% and 38.20%), TriviaQA (61.60% vs. 55.80% and 58.60%), and HotpotQA (33.80% vs. 29.80% and 27.20%). GRPO shows an anomalous spike on Bamboogle (23.61% vs. 13.89% for REINFORCE and 16.67% for PPO), but this is a small dataset and the result may not be reliable. PPO slightly edges out on MuSiQue (13.80% vs. 13.00% and 12.40%).

The paper attributes REINFORCE's advantage to "training stability" (Section 5.4) and notes consistency with Search-R1's findings. The likely explanation: REINFORCE uses Monte Carlo returns directly without learned value functions (PPO) or group-based advantage normalization (GRPO), and in a setting where the environment (the simulator) introduces its own stochasticity, adding more learned components (value model in PPO) or more complex advantage estimation (group comparisons in GRPO) may introduce additional variance that hurts rather than helps. The paper also notes a practical implication: "both REINFORCE and GRPO involve repeated rollouts, which incur higher API costs when interacting with a real search engine. This further underscores the practicality of our simulated search setup" (Section 5.4). The cost savings from simulation are most pronounced for algorithms that require many rollouts—if you're doing 5 rollouts per prompt with REINFORCE, that's 5x the API calls.

Ablation Studies and Robustness Checks

Curriculum rollout vs. fixed 50% noise (Table 6, Section 5.5): The curriculum schedule with model-specific noise parameters produces consistently better policies than a random baseline where noise probability is fixed at 0.5 throughout training. For Qwen-2.5-3B-Base, curriculum achieves 34.47% average EM versus random's 32.59%—a 1.88-point gap. For LLaMA-3.2-3B-Base, curriculum achieves 36.07% versus random's 34.84%—a 1.23-point gap. The effect is consistent across most individual datasets: curriculum beats random on NQ (43.00% vs. 41.40%), TriviaQA (61.60% vs. 59.00%), HotpotQA (33.80% vs. 29.00%), and MuSiQue (13.00% vs. 10.60%) for Qwen. Two datasets show reversals: PopQA favors random for Qwen (44.20% vs. 41.40%) and LLaMA (49.60% vs. 48.40%), and 2Wiki favors random for LLaMA (36.00% vs. 35.60%). These reversals are small (<3 points) and may be noise, but they raise the possibility that for some question types, exposure to consistent 50% noise throughout training builds more robust strategies than the curriculum's late-stage noise concentration. The paper does not investigate this heterogeneity.
Loss masking on document tokens (Figure 4c, Table 7, Appendix C): Removing loss masking on document tokens causes a substantial performance drop: LLaMA-3.2-3B-Base average EM falls from 36.07% to 34.53%. Individual dataset results (Table 7) show consistent degradation: NQ drops from 43.40% to 41.60%, TriviaQA from 63.80% to 61.00%, PopQA from 48.40% to 46.60%, HotpotQA from 32.20% to 29.80%. MuSiQue (13.80% vs. 13.80%) and Bamboogle (15.28% vs. 15.28%) show no difference—likely because these are small test sets where variance dominates. The reward curve (Figure 4c) shows markedly lower and noisier training reward without masking, confirming that document token gradients introduce training instability. This ablation demonstrates a necessary implementation component, though it is not specific to ZEROSEARCH—any RL training with interleaved external text would face the same issue.
Base vs. instruction-tuned policy models (Figures 4a-b, Appendix B): Instruction-tuned models start with higher initial rewards (e.g., Figure 4a shows Qwen-2.5-3B-Inst at ~0.32 vs. Base at ~0.28) because they possess stronger instruction-following capabilities that allow more effective early search engine invocation. However, base models exhibit "greater reward improvements" (Appendix B) and converge to comparable or higher final rewards. For Qwen-2.5-3B (Figure 4a), Base reaches approximately 0.48 versus Instruct at approximately 0.46 by step 200. For Qwen-2.5-7B (Figure 4b), both converge to ~0.58-0.60. Final evaluation performance (Table 3) shows a mixed picture: Qwen-2.5-7B-Base (40.93%) outperforms Instruct (39.08%) by 1.85 points, but Qwen-2.5-3B-Base (34.47%) outperforms Instruct (31.70%) by 2.77 points, while LLaMA-3.2-3B-Base (36.07%) also substantially outperforms Instruct (30.98%) by 5.09 points. This is a surprising and potentially important result: it suggests that instruction-tuning may actually constrain the policy model's ability to discover optimal search strategies through RL, perhaps because instruction-tuned models have been optimized for human preference alignment rather than task completion and are less willing to explore non-standard interaction patterns. The paper does not investigate this mechanism, but the magnitude and consistency of the base-model advantage across three model families is noteworthy.
Simulation LLM: prompt-based vs. fine-tuned (Table 4, discussed above): SFT-based simulators consistently outperform prompt-based simulators at every model scale. The SFT advantage is 1.03 points at 3B, 2.14 points at 7B, and 2.97 points at 14B. The growing gap with model size suggests that larger models' raw outputs diverge more from search-engine style, making the SFT style correction increasingly valuable—or alternatively, that larger models benefit more from the structured training data because they have more capacity to learn the nuanced distinction between useful and noisy document generation.
Model-specific noise parameters (Appendix F): The paper uses different $p_s$ and $p_e$ values for different policy model families: Qwen-2.5-7B gets $p_s=0, p_e=0.75$ ; Qwen-2.5-3B gets $p_s=0, p_e=0.25$ ; LLaMA-3.2-3B gets $p_s=0.25, p_e=0.5$ . These choices are not ablated—the paper does not show what happens if you train Qwen-3B with the 7B noise schedule or vice versa. This is a significant missing experiment: if the optimal curriculum parameters are highly model-specific, then deploying ZEROSEARCH on a new model architecture would require expensive hyperparameter tuning. If they are robust, then the specific values matter less than the general easy-to-hard structure. The paper provides no evidence either way.
Reward function: F1 vs. Exact Match (Section 3.5): The paper reports that preliminary experiments using EM as the reward "often led to reward hacking: the policy model tended to produce excessively long answers to increase the chance of including the correct answer" (Section 3.5). This motivated the switch to F1. However, no ablation comparing F1-trained and EM-trained final performance is reported in the main text or appendix—the claim about reward hacking is anecdotal. An ablation showing EM-trained policies with lower evaluation EM (due to verbose outputs that normalize poorly) would substantiate this important design choice. Additionally, the F1 metric creates a training-evaluation mismatch since evaluation uses EM. The paper does not discuss whether models trained with F1 reward exhibit systematic differences in answer formatting (e.g., consistently producing multi-word answers when single words would suffice) that affect EM evaluation. This gap is particularly relevant for datasets like NQ where answers can vary in granularity.

Critical Assessment

Claim 1: ZEROSEARCH outperforms methods trained with real search engines.

What the experiments demonstrate: The experiments in Table 3 show that ZEROSEARCH-base using Qwen-2.5-7B achieves 40.93% average EM versus Search-R1-base at 39.51%—a 1.42-point advantage. The gap is directionally consistent across most model families and sizes but is narrow and lacks statistical characterization. With seven datasets of varying sizes (some as small as Bamboogle with likely only dozens of test questions) and no reported confidence intervals, we cannot assess whether a 1.42-point average gap is statistically significant or within sampling noise. Individual dataset gaps are sometimes negative (2WikiMultiHopQA for Qwen-2.5-7B: 34.00% vs. 37.20% favoring Search-R1; PopQA for Qwen-2.5-7B-Inst: 48.80% vs. 51.60% favoring Search-R1), suggesting that ZEROSEARCH does not uniformly dominate.

What was not tested: The paper compares against Search-R1 with matching training steps (200), batch size (64), and rollouts (5). However, Search-R1 was not designed with these exact hyperparameters—the paper is applying ZEROSEARCH's training configuration to the Search-R1 algorithm. This may disadvantage Search-R1, which might require different hyperparameters to perform optimally. More critically, Search-R1's original paper used different datasets, different model sizes, and potentially different reward designs. The comparison is fair in the sense that both methods get identical conditions, but it may not represent Search-R1 at its best. A stronger comparison would include hyperparameter tuning for Search-R1 or, at minimum, an ablation showing that the chosen configuration is near-optimal for the real-engine approach.

The training compute is not equalized: ZEROSEARCH requires additional GPU resources for the simulation LLM (4 H20 GPUs), while Search-R1 only requires the policy model's GPUs. The cost analysis in Table 8 accounts for this and shows ZEROSEARCH is still cheaper ( $17.70-$ 70.80 vs. $586.70), but the GPU cost estimate is based on AWS pricing and depends on instance type, region, and utilization assumptions. The paper's acknowledgment that simulation GPU utilization is "inconsistent—high during the rollout and low during policy update—resulting in periods of underutilization and resource waste" (Section D) means the true GPU cost is higher than the simple hourly rate computation suggests because GPUs sit idle during policy updates. For a single training run, this overhead may be acceptable; for a lab running dozens of experiments, the wasted GPU hours accumulate.

The verdict: The claim that ZEROSEARCH can match or slightly exceed real-engine training is supported directionally but not definitively given the absence of statistical testing and the narrow margins. The stronger and better-supported claim is that ZEROSEARCH achieves comparable performance to real-engine training at substantially lower financial cost—a claim that holds across all tested configurations. The paper would be more accurate to frame ZEROSEARCH as "competitive with" rather than "outperforming" real-engine methods.

Claim 2: A 7B simulation LLM achieves comparable performance to Google Search, while a 14B simulation LLM surpasses it.

What the experiments demonstrate: Table 4 provides clean evidence for this claim, but with a crucial scope limitation: the comparison is performed only for Qwen-2.5-3B-Base as the policy model. The 7B simulator (33.53%) effectively ties Google Search (32.81%)—a 0.72-point difference that is almost certainly within noise. The 14B simulator (34.47%) leads by 1.66 points. The scaling trend (3B → 7B → 14B simulators produce progressively better policies) is monotonic and plausible.

What was not tested: The paper does not show whether this scaling relationship holds for other policy models. Does a Qwen-2.5-7B policy trained with a 14B simulator also outperform one trained with Google? Does a LLaMA policy benefit equally? The paper's default configuration uses the 14B simulator for all experiments (Section 4.3), which implicitly assumes the finding generalizes, but Table 4 provides evidence for only one policy-simulator combination. Additionally, the paper does not test whether an even larger simulator (32B, 70B, or even a commercial API model) would provide further gains, or whether the benefit saturates after 14B.

The verdict: The claim is supported for the specific configuration tested but its generalizability is assumed rather than demonstrated. The statement that "a 7B retrieval module achieves comparable performance to the real search engine, while a 14B retrieval module even surpasses it" (Section 1, contributions) should include the qualifier "when training a Qwen-2.5-3B policy model."

Claim 3: The curriculum rollout mechanism progressively elicits reasoning ability, and the curriculum outperforms uniform noise.

What the experiments demonstrate: Table 6 shows that the curriculum setup outperforms a fixed 50% noise baseline by 1.88 points for Qwen-2.5-3B-Base and 1.23 points for LLaMA-3.2-3B-Base. This is clear evidence that the easy-to-hard schedule matters. Figure 2c provides behavioral evidence that interaction patterns evolve coherently during training. However, the ablation compares curriculum only against a single fixed-noise baseline (50%). The paper does not test whether a different fixed noise level—say 25% or 75%—would perform better than 50%, or whether a linear curriculum, step-function curriculum, or reverse curriculum (hard-to-easy) would work. The design space for the curriculum (exponential base 4, model-specific $p_s$ and $p_e$ ) is presented as-is without any sensitivity analysis—we learn only that the chosen parameters work, not why they work or how sensitive results are to their exact values.

What was not tested: The model-specific noise parameters ( $p_s$ and $p_e$ ) are different across model families (Appendix F), but no experiment shows whether these specific values are necessary or whether a single curriculum (e.g., $p_s=0, p_e=0.5$ for all models) would perform comparably. The fact that Qwen-3B gets a milder curriculum (ending at 25% noise) than Qwen-7B (ending at 75% noise) suggests the paper tuned these parameters, but the tuning process is not described and no sensitivity curves are provided. This makes it difficult for practitioners to adopt ZEROSEARCH on new model architectures without their own expensive hyperparameter search.

The verdict: The claim that curriculum helps is well-supported. The claim about why it helps (progressive elicitation of reasoning) is interpretive and not directly tested—the interaction turn analysis provides suggestive but not causal evidence. The specific curriculum design (exponential, base 4, model-specific parameters) is inadequately justified by the experiments.

Claim 4: ZEROSEARCH generalizes across model families, sizes, and types (base/instruct).

What the experiments demonstrate: Table 3 shows strong performance across Qwen-2.5-7B, Qwen-2.5-3B, and LLaMA-3.2-3B, for both Base and Instruct variants. The method works across all tested configurations. The base models often outperform instruct models (a finding discussed in the ablations above), which the paper correctly notes as evidence that "base models can effectively acquire search capabilities through reinforcement learning without the need for supervised fine-tuning as a warm-up" (Appendix B).

What was not tested: The generalization is demonstrated only across 3B and 7B scales within two model families. The paper does not test at the 1B scale (would the method still work with very limited base capabilities?) or at larger scales (13B, 70B). The claim of generalizability "across both base and instruction-tuned LLMs of various parameter sizes" (Section 1) is accurate as far as it goes but the "various" sizes are only 3B and 7B—a range that spans less than one order of magnitude in parameters.

The verdict: Generalization across the tested configurations is solidly demonstrated. The claim would be strengthened by testing at additional scales (particularly 1B, where base model capabilities may be insufficient to learn the structured interaction protocol from outcome-only reward) and additional model families (Mistral, Gemma, etc.).

Claim 5: ZEROSEARCH is compatible with a wide range of RL algorithms.

What the experiments demonstrate: Table 5 shows that REINFORCE, GRPO, and PPO all produce working policies, with REINFORCE performing best. This is straightforward evidence of algorithm compatibility. The paper does not claim that ZEROSEARCH improves these algorithms—only that it works with them.

What was not tested: The comparison uses a single model (Qwen-2.5-3B-Base) and a single set of algorithm-specific hyperparameters. The paper does not show whether GRPO or PPO might perform better with different learning rates, different numbers of rollouts, or algorithm-specific tuning. The relative ranking (REINFORCE > GRPO > PPO) might be sensitive to hyperparameter choices, and the paper does not provide evidence that the comparison is hyperparameter-fair.

The verdict: The claim of compatibility is well-supported—all three algorithms learn. The implicit claim that REINFORCE is the best choice for this setting is plausible but not rigorously established, since algorithm comparisons without hyperparameter tuning are inherently unfair.

Missing Experiments and Methodological Gaps

Several experiments would have strengthened the paper substantially:

Statistical significance testing or confidence intervals on Table 3 results, particularly given the narrow gaps between ZEROSEARCH and Search-R1.
Validation curves during training (test-set performance, not just training reward) to assess whether the smoother ZEROSEARCH training curves translate to better generalization or simply less overfitting.
Curriculum sensitivity analysis: training one model with multiple curriculum parameters (different $b$ values, different $p_s$ / $p_e$ combinations, linear vs. exponential scheduling) to characterize how sensitive results are to curriculum design choices.
Simulator scaling across policy models: Table 4's simulator comparison only for Qwen-2.5-3B—replicating it for a 7B policy and a LLaMA policy would show whether the benefit of larger simulators is universal.
Training for more steps: The 200-step training budget is fixed. If Search-R1's noisy reward curve reflects genuine sampling variance that averages out over more steps, a longer training run might close the gap. Similarly, if ZEROSEARCH's smoother curve reflects a curriculum that accelerates early learning but saturates, the advantage might shrink over longer training.
Query quality analysis: The paper focuses on answer accuracy but never evaluates whether the policy model learns to formulate better queries—a key aspect of genuine search capability. An ablation measuring query specificity, relevance, or diversity between ZEROSEARCH and Search-R1 would provide behavioral evidence for the claimed mechanism.
Oracle-free simulation: All experiments use ground-truth answers in the simulator prompt (Table 2). An ablation where the simulator generates documents without oracle access—relying purely on its parametric knowledge—would test the method's applicability to settings where labeled training data is unavailable.

The overall experimental picture is one where ZEROSEARCH works—it produces policies competitive with or slightly better than real-engine training at lower cost, and the curriculum mechanism contributes measurably to training stability and final performance. However, the narrow performance margins, absence of statistical rigor, and limited exploration of the design space mean that the paper's strongest claims ("surpasses," "significantly outperforms") should be understood as directional rather than definitive. The contribution is solid but benefits more from the practical value of eliminating API costs and enabling curriculum-based training than from clear, unambiguous performance superiority over prior methods.

6. Limitations and Trade-offs

6.1 Oracle Answer Access During Simulation: The Method Assumes Ground-Truth Labels Are Available for All Training Questions

The simulation LLM's prompt template includes the input question and its ground-truth answer: "The user is trying to answer the question: [question] whose answer is [ground truth]" (Table 2, Section 3.3). This oracle access is what enables the simulator to reliably generate useful documents that contain the correct answer and noisy documents that do not—the simulator doesn't need to know the answer from its parametric knowledge; it receives it explicitly. The paper acknowledges this only implicitly by including it in the template, never discussing the assumption as a limitation or exploring whether the method works without it.

The consequence is that ZEROSEARCH, as presented, is only applicable to training tasks where ground-truth answers are known in advance—labeled QA datasets, math problems with verifiable solutions, and similar settings. It cannot be applied to open-ended generation, dialogue, summarization, or any task where the "correct answer" is ambiguous, multi-faceted, or unknown during training. This sharply bounds the method's applicability: the paper's experiments are entirely on factoid QA datasets (NQ, TriviaQA, HotpotQA, etc.) where a short, verifiable ground-truth answer exists for every question. A practitioner wanting to train a search-capable model for, say, writing research reports or answering subjective questions would find the method inapplicable without major adaptation.

There is a secondary, subtler consequence: the oracle answer in the prompt may make the simulation task artificially easy. The simulator doesn't need to demonstrate genuine retrieval or knowledge access—it only needs to paraphrase or embed the given answer into document-like text. This means the simulator's outputs may be less diverse or realistic than real search results, which contain documents of varying relevance, partial matches, and related-but-not-exact information that the oracle-guided simulator never produces. A policy model trained on these oracle-generated documents may learn to expect that useful documents are reliably distinguishable from noisy ones by the simple presence or absence of the answer string—a brittle heuristic that fails on real search results where useful documents might phrase the answer differently or embed it in broader context.

What evidence exists in the paper: There is no ablation testing oracle-free simulation. All experiments use the oracle-enhanced template. Table 4 shows that larger simulators perform better (3B → 7B → 14B yields improving policy performance), which the paper interprets as evidence that parametric knowledge matters, but this scaling trend could also reflect larger models being better at generating stylistically realistic documents around the provided oracle answer rather than genuinely knowing more facts. The paper never disentangles these two explanations. The fact that fine-tuned simulators substantially outperform prompted ones (e.g., SFT-14B at 34.47% vs. Prompt-14B at 31.50% in Table 4) suggests that the SFT process—which trains on real search engine outputs—is doing significant work beyond what the oracle answer alone provides, but we cannot tell how much of the residual benefit comes from the answer itself.

Mitigation status: The paper does not address this limitation, propose an oracle-free variant, or discuss it as a constraint on applicability. The authors treat the oracle answer as a natural part of the simulation prompt design without acknowledging that it restricts the method to supervised settings. A straightforward mitigation—using the simulator's parametric knowledge without oracle access, relying only on the prompt-style fine-tuning to approximate search results—is suggested by the scaling trend in Table 4 but never evaluated. Future work could test whether a sufficiently large, fine-tuned simulator (perhaps 70B+) generates acceptably accurate useful documents from parametric knowledge alone, which would extend ZEROSEARCH to tasks without ground-truth labels.

6.2 The Difficulty Estimation and Curriculum Design Are Inadequately Characterized: No Sensitivity Analysis or Tuning Guidance

The paper's curriculum mechanism is controlled by four hyperparameters: start noise probability $p_s$ , end noise probability $p_e$ , total training steps $m$ , and exponential base $b$ (default 4) in Equation 1 (Section 3.4). The paper uses different $p_s$ and $p_e$ values for different model families—Qwen-2.5-7B gets $p_s=0, p_e=0.75$ while Qwen-2.5-3B gets $p_s=0, p_e=0.25$ and LLaMA-3.2-3B gets $p_s=0.25, p_e=0.5$ (Appendix F)—but provides no ablation or sensitivity analysis showing how performance varies with these choices. The exponential base $b=4$ is stated as the default with no justification or comparison to alternatives (linear schedule, step function, different bases).

The consequence is that a practitioner adopting ZEROSEARCH for a new model architecture or task faces an open-ended hyperparameter search with no guidance on where to start. The 3× difference in final noise probability between Qwen-3B (25%) and Qwen-7B (75%) suggests that optimal curriculum parameters are highly model-specific, yet the paper offers no principle for selecting them. A poorly chosen curriculum could produce worse results than the fixed 50% noise baseline (which Table 6 shows underperforms the tuned curriculum by 1.2–1.9 points), meaning the curriculum mechanism can hurt rather than help if naively applied. Worse, the curriculum is baked into the training process—you cannot observe its effect without running a full 200-step training run—making hyperparameter search expensive precisely when it's most needed (on a new, uncharacterized model).

The deeper issue is that we don't know whether the curriculum's benefit is robust or brittle. Is the easy-to-hard structure doing the work, with the specific parameter values mattering little as long as they're in a reasonable range? Or is precise tuning required, with small deviations producing large performance drops? The single ablation in Table 6 compares only against a fixed 50% baseline—it shows that the chosen curriculum beats one alternative, not that the chosen curriculum is near-optimal or that the curriculum principle is robust.

What evidence exists in the paper: Table 6 demonstrates that the chosen curriculum outperforms uniform 50% noise, but provides no information about sensitivity. The paper does not report experiments varying $b$ , testing linear schedules, or sweeping $p_s$ and $p_e$ . The per-model parameter differences are presented as configuration details in Appendix F without commentary on how they were chosen—were they tuned via grid search, guided by validation performance, or set heuristically? The answer is absent.

Mitigation status: Not addressed. The paper does not acknowledge this as a limitation or provide guidance for parameter selection. A sensitivity experiment—training one model with multiple $p_s$ / $p_e$ pairs and/or different $b$ values—would substantially increase confidence that the method transfers to new settings. Without it, the curriculum is a demonstrated-but-unexplained recipe rather than a principled technique.

6.3 Evaluation Is Limited to Factoid QA with a Single Search API Configuration; Generalization to Other Tasks, Search Engines, or Retrieval Settings Is Untested

All experiments evaluate on seven factoid QA datasets (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle) using Google Web Search via SerpAPI with exactly five retrieved documents per query (Section 4.1–4.3). The training data is merged from only NQ and HotpotQA (Section 4.3), meaning out-of-domain generalization is tested across closely related QA datasets—all involve short, factual questions with verifiable short answers. The paper does not test on code generation, multi-document summarization, open-ended research tasks, or conversational QA, nor does it test with different search APIs (Bing, DuckDuckGo), different retrieval depths (1, 3, 10 documents), or different result types (news, scholarly, image snippets).

The consequence is that we cannot assess whether ZEROSEARCH's learned search strategies are genuine information-seeking behaviors or task-specific heuristics adapted to the particular characteristics of factoid QA with Google snippet-style results. Several potential failure modes are untested:

Answer style mismatch: The paper trains with an F1 reward that penalizes verbosity (Section 3.5) and evaluates with Exact Match. On tasks requiring longer, structured, or multi-sentence answers, the concise-answer strategy the model learns may be inappropriate—the model might truncate important information to keep answers short.
Search engine shift: If the model learns patterns specific to Google snippets (e.g., expecting answers to appear in document titles or bolded text in search results), switching to a different search API with different formatting could degrade performance. The paper's entire evaluation uses the same API it simulated during training (implicitly, since the simulator was fine-tuned on Google results).
Result count sensitivity: The fixed five-document setting is arbitrary. A model trained to read five documents might underperform when given three (insufficient information) or ten (information overload), yet the paper provides no evidence of robustness to this parameter.
Temporal drift: Real search results change over time as the web is updated. A model trained against a snapshot of Google results (captured during simulator fine-tuning) may develop expectations about result quality or content that become stale.

Additionally, the out-of-domain generalization claim is weakened by the homogeneity of the test suite: all seven datasets are factoid QA. "Out-of-domain" here means "questions from a different QA dataset," not "a different task type." True out-of-domain generalization would include tasks the model was never trained to perform, which the paper does not test.

What evidence exists in the paper: Table 3 shows performance on seven QA datasets, with out-of-domain datasets (TriviaQA, PopQA, 2Wiki, MuSiQue, Bamboogle) showing generally strong results. However, performance on Bamboogle is notably lower and more variable than on in-domain datasets (e.g., Qwen-2.5-7B-Base ZEROSEARCH: 33.33% vs. 42.40% on NQ and 66.40% on TriviaQA), suggesting that the model's capabilities vary substantially with question characteristics. The paper does not analyze which types of questions succeed or fail, so we cannot characterize the generalization boundary.

Mitigation status: Not addressed. The paper does not discuss task diversity as a limitation or propose experiments on non-QA tasks. The strong performance on out-of-domain QA datasets is a positive signal, but it's a narrow slice of the generalization space. Testing on even one non-QA task (e.g., fact verification, entity linking, or multi-document synthesis) would establish whether ZEROSEARCH teaches genuine search skills or QA-specific heuristics.

6.4 Simulation GPU Costs Are Not Fully Amortized and the Financial Analysis May Understate Deployment Overhead

The paper positions ZEROSEARCH as a cost-saving alternative to real search APIs, with Section D (Table 8) reporting that a 200-step training run with ~64,000 queries costs $0 in API fees and$ 17.70– $70.80 in GPU rental for the simulation server, versus$ 586.70 for Google Search. The paper acknowledges that "GPU utilization of the simulation server is inconsistent—high during the rollout and low during policy update—resulting in periods of underutilization and resource waste" (Section D), but treats this as a minor inefficiency that can be addressed by "shar[ing] a single simulation server across multiple RL training tasks."

The consequence is that the cost comparison in Table 8 does not represent the true economic cost of deployment, particularly for research groups that own their GPUs rather than renting them, or for use cases where sharing a simulation server across tasks is infeasible. Several factors are omitted:

Capital cost amortization: The cost figures use AWS on-demand pricing, which is the appropriate comparison for cloud-based API calls. But many research labs own GPUs with fixed capital costs. The marginal cost of running an owned GPU is electricity (negligible), not the rental rate—but the opportunity cost is real (the GPU cannot be used for other experiments). The paper's GPU cost numbers are meaningful for cloud users but misleading for hardware owners, who would see the API savings more starkly but still face the opportunity cost of tying up 4 H20 GPUs for ~12 hours per training run.
Simulator training cost: The fine-tuning of the simulation LLM (Section 3.3) requires collecting interaction trajectories from a real search engine, labeling document quality with an LLM judge, and running SFT. This one-time cost is not included in Table 8. For the 14B simulator, this likely requires at least thousands of real API calls and additional GPU hours for fine-tuning—costs that should be amortized across training runs but are absent from the headline comparison.
Underutilization is not free: The paper notes that simulators sit idle during policy updates. If you're renting cloud GPUs by the hour, you pay for idle time. The paper's suggested mitigation—sharing a server across tasks—assumes multiple RL training jobs are running simultaneously and can be staggered so one job's rollouts overlap with another's updates. This is organizational overhead that small labs or single-experiment workflows cannot absorb. The true GPU cost for a single training run is higher than Table 8 suggests because you pay for idle time that produces no simulator throughput.
Scalability limits of on-premise simulation: The paper claims that "increasing the number of GPUs significantly accelerates the generation throughput of the simulation LLM" (Section 1). This is true but bounded: GPU-to-GPU communication overhead, the simulator model's maximum batch size, and the policy model's rollout parallelism all cap throughput. The paper does not characterize these limits or provide throughput measurements comparing simulator latency to real API latency.

What evidence exists in the paper: Table 8 provides the cost comparison, and Section D discusses resource utilization qualitatively. There are no throughput benchmarks, latency measurements, or scaling curves showing how simulator throughput scales with GPU count. The admission about underutilization signals awareness of the issue but provides no quantification of its magnitude.

Mitigation status: Partially addressed through the suggestion to share simulation servers across tasks (Section D). This is a credible mitigation for large-scale industrial deployments with multiple concurrent training jobs but does not address the single-experiment use case. The paper does not propose architectural changes (e.g., asynchronous simulation pipelining) that would reduce idle time within a single training run.

6.5 Narrow Performance Margins and Absence of Statistical Characterization Weaken the Central Claim of Outperformance

The paper's headline conclusion is that ZEROSEARCH "outperforms real search engine-based models" (Section 1, contributions). The evidence in Table 3, as analyzed in Section 5 (Experimental Analysis), shows narrow performance advantages: ZEROSEARCH-base achieves 40.93% average EM versus Search-R1-base at 39.51% (Qwen-2.5-7B), a 1.42-point gap; ZEROSEARCH-base achieves 34.47% versus Search-R1-base at 32.81% (Qwen-2.5-3B), a 1.66-point gap; ZEROSEARCH-base achieves 36.07% versus Search-R1-base at 34.21% (LLaMA-3.2-3B), a 1.86-point gap. These gaps are directionally consistent but small relative to the overall performance range and the variability one would expect from finite test sets. The paper reports no confidence intervals, no standard deviations, and no statistical significance tests.

The consequence is that the central claim of performance superiority is inadequately supported. A practitioner reading Table 3 cannot determine whether the 1.4-point average advantage is meaningful or whether it could be reversed by resampling the test set, using a different random seed, or running training for a different number of steps. This matters for decision-making: if a lab is deciding whether to abandon real-engine RL training in favor of ZEROSEARCH, "approximately equal performance at much lower cost" is a compelling argument. "Statistically indistinguishable performance" is a different—and still valuable—argument. "Definitively better performance" is a stronger claim that the evidence does not warrant.

The absence of statistical characterization also undermines per-dataset comparisons. On individual benchmarks, Search-R1 sometimes outperforms ZEROSEARCH (e.g., Qwen-2.5-7B-Inst on PopQA: Search-R1 51.60% vs. ZEROSEARCH 48.80%; Qwen-2.5-7B-Base on 2Wiki: Search-R1 37.20% vs. ZEROSEARCH 34.00%). Without variance estimates, we cannot tell whether these reversals are noise or signals of a genuine regime where real-engine training is preferable. The paper does not analyze these cases, treating them as unremarkable variation while emphasizing the average advantage.

The root issue is a mismatch between the strength of the paper's rhetoric ("surpasses," "consistently outperforms") and the strength of its evidence (directional trends without uncertainty quantification). A more precise framing—"ZEROSEARCH achieves performance competitive with or slightly exceeding real search engine-based training while eliminating API costs"—would align the claims with the data while still conveying the practical value. The paper's actual framing, which emphasizes superiority, overstates the evidence and risks misleading practitioners who might adopt ZEROSEARCH expecting unambiguous accuracy gains rather than cost savings with comparable accuracy.

What evidence exists in the paper: Table 3 provides point estimates of EM for each method/dataset combination. There are no error bars, no mention of test set sizes, no cross-validation folds, and no discussion of variance. The reward curves in Figures 2-4 show training metrics without standard deviation shading. The paper reports results from single training runs without seed variation.

Mitigation status: Not addressed. The paper does not acknowledge the absence of statistical characterization or discuss test-set size limitations. Given that some of the evaluation datasets are small (Bamboogle likely has only dozens of questions, based on the significant per-model variance in Tables 3-7), statistical testing is especially important for the out-of-domain results that the paper uses to claim generalization. Reporting bootstrap confidence intervals for the average EM or conducting paired permutation tests between ZEROSEARCH and Search-R1 would substantially strengthen the empirical claims without requiring additional experiments.

6.6 The Interaction Template and Structured Output Format May Mask Whether the Model Is Learning Search or Learning to Exploit Template Patterns

The paper uses a highly structured interaction template with XML-style tags (<thinking>, <search>, <information>, <answer>) that explicitly instructs the model about the expected multi-turn interaction protocol (Table 1, Section 3.2). The model learns to produce these tags and follows the protocol without any explicit format reward—only the F1 score on the answer content provides supervision (Section 3.5). The paper presents this as evidence that outcome-only RL can elicit complex structured behaviors.

The consequence is that we cannot distinguish between two competing explanations for the model's performance: (1) the model has learned genuine search skills—it understands when information is missing, formulates effective queries, evaluates result relevance, and integrates retrieved information into its reasoning; or (2) the model has learned to exploit statistical patterns in the template-simulator interaction that correlate with high rewards without developing robust, generalizable search capabilities. The second explanation is particularly concerning because the simulator's behavior is deterministic in quality: "useful" documents always contain the answer (because the simulator receives it in the prompt), and "noisy" documents never contain the answer. A model that learns to recognize superficial features distinguishing useful from noisy documents (e.g., the presence of named entities matching the question topic, or the presence/absence of specific phrasing patterns from the oracle-enhanced simulator) could achieve high training rewards without learning to evaluate document content meaningfully.

Several observations in the paper are consistent with either explanation. The interaction turn analysis (Figure 2c) shows the model learns to make multiple searches, but doesn't show whether those additional searches are genuinely information-seeking or simply a learned response to the curriculum's increasing noise. The case studies in Table 9 demonstrate correct behavior but are cherry-picked examples rather than a systematic behavioral analysis. The model could be learning that when documents don't contain the expected answer pattern, the correct response is to search again with a reformulated query—a useful heuristic but not the same as understanding document content.

What evidence exists in the paper: The case studies (Table 9) show two examples of successful multi-turn search. The interaction turn analysis (Figure 2c) shows aggregate behavioral trends. However, there is no systematic analysis of query quality (are the model's queries well-formed and specific?), no error analysis showing failure modes (does the model fail by issuing poor queries, misinterpreting results, or stopping too early?), and no experiment that manipulates document content independently of quality labels to test whether the model reads documents or relies on superficial cues. The paper also does not compare the search behaviors of ZEROSEARCH-trained and Search-R1-trained models—e.g., average query length, query specificity, number of searches per question, or types of reformulation—which would reveal whether simulation training produces systematically different search strategies.

Mitigation status: Not addressed. The paper treats behavioral complexity as an emergent property of outcome-only RL without investigating whether the behavior is robust or superficial. A targeted experiment—e.g., adversarial document manipulation where useful documents are rewritten to be superficially different from training while still containing the answer, or probing the model's ability to identify which specific document provided the answer—would distinguish genuine comprehension from pattern matching. Without such experiments, the claim that ZEROSEARCH "incentivizes the search capabilities of LLMs" (title, Section 1) conflates "the model produces search-like behavior" with "the model is capable of search." These are not the same thing, and the paper does not provide evidence that distinguishes them.

7. Implications and Future Directions

How This Work Changes the Landscape

ZEROSEARCH represents a conceptual reframing of the relationship between RL training and external tools, but not a paradigm shift. The core insight—that a learned simulator can replace a real-world system during RL training and produce equally good or better policies—is a methodology contribution that changes how practitioners approach search-augmented LLM training, not what they aim to achieve. The paradigm remains "train LLMs to use tools through RL"; ZEROSEARCH changes the implementation path from "pay for and tolerate the real tool" to "build a controllable proxy and train against it."

The magnitude of this shift varies by audience. For academic labs and smaller companies where API costs are a genuine barrier to entry, ZEROSEARCH is enabling: it reduces the financial cost of RL-based search training from ~ $587 to ~$ 18–$71 per run (Table 8, Section D), a ~8–33× reduction that makes iterative experimentation feasible where it was previously prohibitive. For well-resourced industrial labs with large API budgets, the more significant contribution is the controllability insight: the demonstration that a curriculum over document quality produces smoother training and (directionally) better final performance than training on uncontrolled real search results (Figures 2-3, Table 6). This challenges the implicit assumption in prior work that "more realistic = better" for training environments, suggesting instead that pedagogically structured environments outperform realistic but uncontrolled ones.

The paper resolves a practical contradiction that had been brewing in the RL-for-search literature. Search-R1 (Jin et al., 2025), DeepResearcher (Zheng et al., 2025), and WebThinker (Li et al., 2026) had demonstrated that RL with live search APIs works, but their training curves were noisy and their cost scaling was unsustainable. R1-Searcher (Song et al., 2025) and ReSearch (Chen et al., 2026) avoided API costs by using static Wikipedia corpora but sacrificed the realism needed for deployment generalization. ZEROSEARCH resolves this by showing that a fine-tuned LLM can be both realistic (through SFT on real search outputs) and controllable (through prompt-keyword quality modulation), achieving the best of both approaches. The reconciliation is: you don't have to choose between cost and realism if you're willing to invest one-time compute in training a simulator.

The work also shifts the conversation around what constitutes a "training environment" for LLM tool use. Prior work treated environments as external, fixed, and accessed through APIs. ZEROSEARCH demonstrates that environments can be internal, learned, and designed—that an LLM can simulate another LLM's tool interactions well enough to substitute for the real tool during training. This opens a broader design space: if a search engine can be simulated, what about a code interpreter, a database, a physics engine, or a dialogue partner? The simulator-as-trained-component pattern that ZEROSEARCH establishes is transferable to any RL setting where the environment output is text and ground-truth signals are available for supervision.

Research directions that become more attractive after this work:

Environment design for RL tool-use training. The paper shows that controllability (curriculum, noise scheduling) can matter more than fidelity. This encourages systematic research into environment parameters—difficulty, noise type, feedback granularity—rather than assuming that approximating reality as closely as possible is always optimal.
Scaling simulator models. Table 4's monotonic improvement from 3B to 14B simulators suggests that simulator quality is a lever worth pulling. Training simulators at the 70B+ scale, or using commercial API models as simulators (despite reintroducing some API cost), could yield further policy improvements. The diminishing returns (3B→7B: +3.06%; 7B→14B: +0.94%) suggest saturation, but the asymptote is unknown.
Curriculum learning for multi-agent training. The exponential noise schedule (Equation 1) is a simple mechanism that could be applied to any setting where one agent (the environment) provides progressively degraded information to another agent (the learner). Self-play, debate, and negotiation training could all benefit from structured difficulty scheduling.

Research directions that become less urgent:

Optimizing RL algorithms specifically for real-engine training. If simulation can match real-engine training, the pressure to make REINFORCE work well with noisy, uncontrollable real search results decreases. Algorithm development effort shifts toward methods that exploit controllable environments.
Building larger static retrieval corpora for RL training. The paper shows that LLM parametric knowledge plus oracle prompting is sufficient, making large-scale corpus curation (a la Wikipedia dumps for Search-R1) less necessary for factoid QA domains.

Important caveat on the magnitude of the shift: The paper's empirical advantage over real-engine training is narrow—1.4 to 1.9 percentage points of average EM across configurations (Table 3)—and lacks statistical characterization. The practical impact is therefore more about cost reduction and training stability than about unambiguous performance superiority. A lab that already has a Search-R1 pipeline working well with real APIs may see ZEROSEARCH as a cost optimization rather than a performance upgrade. The claim that simulation outperforms real engines should be understood as a demonstration of parity (with cost savings) rather than a breakthrough in capability—the 1.4-point gap on Qwen-2.5-7B could easily be noise, and even if real, is small relative to the gap between any search-capable method and no-search baselines (~15-25 points).

Follow-Up Research This Work Enables

Oracle-free simulation: Can a sufficiently large and well-trained simulator generate useful documents without ground-truth answer access?

The paper's simulation prompt (Table 2) includes the ground-truth answer, which means ZEROSEARCH is only applicable to tasks with labeled training data. The most impactful follow-up would test whether this oracle access is necessary. The experiment would train simulators of increasing size (7B, 14B, 32B, 70B) on the same SFT data (real search outputs) but evaluate them on generating useful documents without the oracle answer in the prompt—relying purely on parametric knowledge. If a 70B simulator without oracle access achieves comparable policy performance to a 14B simulator with oracle access, the method extends to unsupervised settings. If even large simulators fail without oracle answers, the method's applicability remains bounded to labeled QA, and alternative approaches (retrieving from a corpus to populate the simulator's prompt, or using the policy model's own generated hypotheses as pseudo-answers) become necessary research directions.

Cross-task generalization: Is search capability learned through ZEROSEARCH specific to factoid QA, or does it transfer to qualitatively different retrieval tasks?

The paper evaluates on seven factoid QA datasets, all structurally similar. A strong follow-up would train a ZEROSEARCH policy on the same NQ+HotpotQA mixture and evaluate on non-QA search tasks: fact verification (FEVER), multi-document summarization (requiring synthesis across retrieved documents rather than answer extraction), entity linking, or open-ended research question answering. This would test whether the policy model learns general search skills (query formulation, result evaluation, information integration) or QA-specific heuristics (find the answer string, output it concisely). The paper's F1 reward, which penalizes verbosity (Section 3.5), may be poorly suited to tasks requiring longer outputs—an experiment replacing F1 with a task-appropriate reward (e.g., ROUGE for summarization) while keeping the same simulator and curriculum would disentangle reward design from search capability.

Simulator scaling laws: Does policy performance follow a predictable function of simulator size, and where does it saturate?

Table 4 shows policy EM improving from 30.47% (3B simulator) to 33.53% (7B) to 34.47% (14B) when training a Qwen-2.5-3B policy. The gains diminish but don't clearly saturate. A systematic scaling study testing 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 70B simulators (all fine-tuned on the same data) would reveal whether the curve is logarithmic (diminishing returns, small marginal benefit beyond 14B), sigmoidal (sudden improvement at some threshold where parametric knowledge becomes sufficient), or linear (continued gains). If 32B+ simulators yield substantially better policies, the "zero-cost" framing weakens since running a 70B simulator is expensive—but if they don't, a 7-14B simulator is a sweet spot. The experiment should also test whether the scaling relationship differs for different policy model sizes—does a 7B policy benefit more from a larger simulator than a 3B policy does, or does the benefit saturate at the same simulator size regardless of policy capacity?

Query quality and search behavior analysis: What specific search skills does ZEROSEARCH teach, and are they different from those learned through real-engine training?

The paper demonstrates that ZEROSEARCH-trained models produce correct answers at rates comparable to Search-R1-trained models, but provides almost no analysis of how they search differently. A behavioral comparison would collect trajectories from both ZEROSEARCH-trained and Search-R1-trained models on the same test set and measure: average query length, vocabulary overlap between queries and questions, query specificity (do queries name specific entities vs. use broad terms?), number of searches per question, probability of reformulating a query after receiving results (vs. searching once and answering), and whether queries show evidence of reading document content (do queries incorporate terms from earlier results?). The paper's interaction turn analysis (Figure 2c) provides only aggregate turn counts—a richer behavioral comparison would reveal whether simulation-trained models develop different search strategies (e.g., more aggressive reformulation in response to noise, learned through the curriculum) or whether they converge to the same strategies as real-engine-trained models (suggesting the environment matters less than the reward structure). Negative result: if ZEROSEARCH models search differently but achieve similar accuracy, the behavioral differences are interesting but not practically consequential. Positive result: if ZEROSEARCH-trained models exhibit more systematic, noise-robust search behaviors (e.g., better at distinguishing when to search again vs. when to answer), the curriculum mechanism is teaching transferable skills rather than just optimizing the training objective.

Curriculum design space characterization: How sensitive is ZEROSEARCH to the noise schedule parameters, and is there a principled way to set them for new models?

The paper uses different start and end noise probabilities ( $p_s$ , $p_e$ ) for different models (Appendix F) but provides no guidance for selecting them. A systematic follow-up would train one model (e.g., Qwen-2.5-3B-Base) with a grid of ( $p_s$ , $p_e$ ) pairs spanning the range [0, 1], keeping the exponential base $b=4$ fixed, and plot the resulting policy EM as a function of both parameters. This would reveal: (1) Is there a broad plateau of good values, or a narrow peak? If broad, practitioners can choose heuristically. If narrow, automated tuning is necessary. (2) Does the optimal final noise probability correlate with measurable model properties (e.g., initial pass@1 on the training set, or learning rate during early training)? If so, a principled tuning heuristic exists. (3) How does the optimal schedule change when the total training steps $m$ is varied? The paper uses $m=200$ ; longer training might require different noise trajectories. Additionally, comparing the exponential schedule (base 4) against linear, step-function, and reverse (hard-to-easy) schedules would characterize how much the specific functional form matters versus the general "start easy, get harder" principle.

Generalization to other RL algorithms and reward structures: Can the simulator+curriculum approach improve PPO and GRPO to match or exceed REINFORCE?

Table 5 shows REINFORCE outperforming GRPO and PPO, but this comparison is at a single hyperparameter setting. Since ZEROSEARCH provides a more stable training environment (smoother reward curves), it may particularly benefit algorithms that are sensitive to reward variance—PPO's learned value function, for example, might perform better when the reward signal is less noisy. A follow-up would perform hyperparameter sweeps for GRPO and PPO within the ZEROSEARCH framework (varying learning rates, clipping parameters, number of epochs per update) to determine whether the REINFORCE advantage is fundamental to this task or an artifact of the specific hyperparameter configuration tested. If GRPO or PPO can match REINFORCE with proper tuning, practitioners gain algorithm flexibility. If REINFORCE consistently dominates regardless of tuning, it suggests that simpler policy gradient methods are inherently better suited to tool-use RL with outcome-only rewards—a finding with implications beyond ZEROSEARCH.

Practical Applications and Downstream Use Cases

Cost-efficient RL training for search-augmented LLMs in academic and startup settings.

The paper's cost analysis (Table 8, Section D) shows that a single ZEROSEARCH training run costs $17.70–$ 70.80 in GPU rental versus $586.70 for real search API calls—an 8–33× reduction. For a PhD student or small startup iterating on search-augmented LLM training, this is the difference between running 1-2 experiments (API budget limited) and running 10-20 experiments (GPU budget allows exploration). The practical workflow: train a simulation LLM once on a collected dataset of real search interactions (one-time cost of a few thousand API calls for data collection + SFT GPU hours), then use that simulator for all subsequent RL experiments, paying only GPU costs. If the simulator is shared across multiple projects (Section D's suggestion), the amortized cost per experiment drops further. The method is immediately deployable for any lab with access to 4 H20 GPUs (or equivalent) and a QA training dataset with ground-truth answers.

Training specialized search agents for domain-specific QA without ongoing API dependency.

An organization building a QA system for a specific domain (medical literature, legal documents, internal knowledge bases) can use ZEROSEARCH to train a search-capable LLM that is tailored to domain-specific query patterns and document styles. The key advantage: the simulator can be fine-tuned on domain-specific document formats (e.g., PubMed abstracts, legal opinions) rather than generic web snippets, producing a policy model that learns to search effectively within that domain's information landscape. Once trained, the policy model uses the real domain-specific search API at inference time, but the training process never called that API—eliminating concerns about API rate limits, cost scaling during RL's high-rollout phase, or exposing sensitive training queries to external services. The paper's finding that base models match or exceed instruction-tuned models after RL training (Appendix B, Figure 4a-b) means organizations can start from a strong open-source base model (e.g., Qwen-2.5-7B-Base) without needing expensive instruction-tuning infrastructure.

Data augmentation for retrieval-augmented generation through curriculum-based synthetic document generation.

The paper's simulation LLM, which can generate both useful and noisy documents on command (Table 2, Table 10), can serve as a standalone data generation tool for training or evaluating RAG systems beyond the RL setting. A practitioner training a Reranker model (which scores retrieved documents by relevance) needs training data with both relevant and irrelevant documents for each query. The ZEROSEARCH simulator provides this: for any query and answer pair, generate K useful documents (containing the answer) and M noisy documents (not containing the answer) with controlled quality. Compared to scraping real search results (which have unpredictable quality distributions), the simulator provides cleanly labeled, on-demand training data where the relevance signal is known exactly (the simulator was given the answer). The 14B simulator's case studies (Table 10) show that noisy documents are topically related but miss the specific fact—exactly the challenging near-miss examples that make good Reranker training data. This application does not require the full RL pipeline, making it accessible even to practitioners who only need synthetic retrieval training data.

When to Prefer This Method

The paper positions ZEROSEARCH explicitly against Search-R1 and other real-engine RL training methods, and provides cost and performance data to support a decision rule. The tradeoffs are:

Prefer ZEROSEARCH over real search engine RL training when:

API cost is a binding constraint on experimentation scale. If a training run with real APIs would cost hundreds of dollars (Table 8: $587 for a 64K-query run) and you need to run multiple experiments (hyperparameter sweeps, ablations, model comparisons), the 8-33× cost reduction enables exploration that would otherwise be infeasible.
Training stability matters more than exact environment fidelity. Figures 2a-b and Figure 3 (Appendix A) show ZEROSEARCH producing smoother reward curves than Search-R1, which the paper attributes to controllable document quality. If your training pipeline is sensitive to reward variance (e.g., you're using PPO with a learned value function that struggles with noisy rewards), the curriculum-based simulation provides a more stable learning signal.
You are training a base model without instruction tuning. Appendix B (Figure 4a-b) shows base models starting from lower rewards but achieving greater improvements and often matching or exceeding instruct models after RL training. ZEROSEARCH works without requiring a pre-existing instruction-following capability, making it suitable for training from raw pretrained checkpoints.
You want to train a domain-specific search agent where the simulator can be fine-tuned on domain-specific document formats (medical, legal, technical) rather than generic web search results. The SFT step (Section 3.3) adapts the simulator to any document style for which you can collect real interaction trajectories.

Prefer real search engine RL training when:

Oracle answers are unavailable for training questions. ZEROSEARCH requires ground-truth answers in the simulator prompt (Table 2) to control document quality. If you're training on tasks without labeled answers—open-ended research, dialogue, or generation where the "correct answer" is undefined—the method cannot be applied as described. Real search engines can be called without requiring answer labels, making them the only option for unsupervised search training.
The deployment search engine differs substantially from Google or from the engine used to train the simulator. The simulator is fine-tuned on Google Search outputs and may not generalize to other APIs (Bing, DuckDuckGo, domain-specific search) with different result formats, lengths, or quality characteristics. The paper evaluates only on Google Search, so the simulation-to-real gap for other engines is unknown. If your deployment environment uses a search API with qualitatively different behavior, real-engine training against that specific API may produce better transfer.
You are operating at a scale where simulation GPU costs approach or exceed API costs. The paper's cost comparison (Table 8) assumes 4 H20 GPUs for ~12 hours. If your training runs are short (few steps, small batch size) or you already own underutilized GPUs whose opportunity cost is near zero, the API cost advantage shrinks. Conversely, if you need a 70B+ simulator to achieve acceptable quality (extrapolating from Table 4's scaling trend, where larger simulators perform better) and that requires 8+ GPUs, the GPU cost may exceed the API cost for modest-scale training.
You need to characterize the policy's robustness to real-world result variability. Training against a controlled simulator may produce policies that are optimized for the simulator's specific noise distribution (binary useful/noisy, with the oracle answer either present or absent) rather than the continuous, multivalent relevance of real search results. If your application requires graceful degradation across the full spectrum of result quality (not just binary useful/noisy), real-engine training exposes the model to that spectrum during learning. The paper provides no evidence about how ZEROSEARCH-trained policies behave on partially relevant or tangentially useful documents—the simulator's "noisy" documents are explicitly designed not to contain the answer (Table 10), which is a harder binary distinction than the graded relevance of real search.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: RL Search Training is Bottlenecked by Real Search Engine Dependencies

Why Prior Approaches Fall Short

Conflicting Demands: Quality Control vs. Realism

How This Paper Positions Itself

The Hidden Assumption About LLM Knowledge Coverage

Summary of the Gap

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

Reinforcement Learning Without a Search Engine: The Formal Objective

Training Template: Structured Multi-Turn Interaction

Search Simulation Tuning: Building a Controllable Document Generator

Rollout with Curriculum Search Simulation: Progressive Difficulty Scheduling

Reward Design: F1-Based Scoring to Prevent Reward Hacking

Training Algorithm and Document Token Loss Masking

Why This Architecture Over Alternatives

4. Key Insights and Innovations

Innovation 1: Simulated Search Engines as a Controllable Training Environment, Not Just a Cost-Saving Hack

Innovation 2: The Curriculum Rollout Mechanism as Difficulty-Conditioned Environment Design

Innovation 3: Demonstrating That LLM Parametric Knowledge Is Sufficient for Search Simulation When Augmented with Oracle-Enhanced Prompting

Innovation 4: Identifying and Solving the Document Token Gradient Contamination Problem

Innovation 5: Outcome-Only Reward Is Sufficient for Learning Multi-Turn Search and Format Adherence

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Overall Performance: ZEROSEARCH vs. All Baselines (Table 3)

Training Dynamics: Reward Curve Comparison (Figures 2a-b and Figure 3)

Simulation LLM Scaling: How Simulator Size Affects Policy Performance (Table 4)

Interaction Turns Analysis: How Search Behavior Evolves During Training (Figure 2c)

Algorithm Compatibility: REINFORCE vs. GRPO vs. PPO (Table 5)

Ablation Studies and Robustness Checks

Critical Assessment

Claim 1: ZEROSEARCH outperforms methods trained with real search engines.

Claim 2: A 7B simulation LLM achieves comparable performance to Google Search, while a 14B simulation LLM surpasses it.

Claim 3: The curriculum rollout mechanism progressively elicits reasoning ability, and the curriculum outperforms uniform noise.

Claim 4: ZEROSEARCH generalizes across model families, sizes, and types (base/instruct).

Claim 5: ZEROSEARCH is compatible with a wide range of RL algorithms.

Missing Experiments and Methodological Gaps

6. Limitations and Trade-offs

6.1 Oracle Answer Access During Simulation: The Method Assumes Ground-Truth Labels Are Available for All Training Questions

6.2 The Difficulty Estimation and Curriculum Design Are Inadequately Characterized: No Sensitivity Analysis or Tuning Guidance

6.3 Evaluation Is Limited to Factoid QA with a Single Search API Configuration; Generalization to Other Tasks, Search Engines, or Retrieval Settings Is Untested

6.4 Simulation GPU Costs Are Not Fully Amortized and the Financial Analysis May Understate Deployment Overhead

6.5 Narrow Performance Margins and Absence of Statistical Characterization Weaken the Central Claim of Outperformance

6.6 The Interaction Template and Structured Output Format May Mask Whether the Model Is Learning Search or Learning to Exploit Template Patterns

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

When to Prefer This Method

Adjacent in domain

RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale

A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment