ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent

ArXiv: 2312.10003

🎯 Pitch

A multi-step search agent can bootstrap itself to near-teacher performance with no human-labeled data—by repeatedly generating its own trajectories and letting a language model rank them as synthetic training data. Starting from a huge prompted model, just two iterations of this self-improvement loop produce a fine-tuned small model 100× smaller that nearly matches the giant’s accuracy.

1. Executive Summary

This paper proposes a method for iteratively improving a multi-step reasoning LLM agent—a ReAct-style Search Agent with self-critique for long-form question answering—through a ReST-like self-improvement algorithm that generates synthetic trajectory data with AI feedback as the training signal, entirely without human-labeled data. Starting from a prompted PaLM 2-L model and applying two iterations of growing-batch reinforcement learning with LLM-based ranking (off-policy re-ranking of sampled actions using an instruction-tuned model rather than a trained reward model), the approach produces fine-tuned models that achieve comparable performance with two orders of magnitude fewer parameters, with the distilled PaLM 2-XS model reaching 65.9% auto-eval accuracy on the Bamboogle benchmark compared to the prompted PaLM 2-L teacher's 70.3%. The self-improvement process simultaneously serves as self-distillation—the higher-quality synthetic data from later iterations effectively trains smaller models—establishing that process-based agent trajectories can be refined through AI feedback loops without any human intervention in either data generation or filtering.

2. Context and Motivation

The Core Problem: LLM Agents Are Brittle, and We Can't Train Them End-to-End

The fundamental problem this paper addresses is deceptively simple: how do you improve a multi-step reasoning agent that interacts with external tools when you can't backpropagate through those interactions?

To understand why this matters, consider what happens when a language model is deployed as an agent—a system that doesn't just produce a single output, but executes a sequence of reasoning steps interleaved with actions like calling a search API, reading results, and deciding whether to search again. The agent's final answer quality depends on the entire trajectory: whether it formulated the right search queries, whether it correctly identified relevant snippets from the results, whether it knew when to stop searching, and whether it properly grounded its answer in the retrieved information. Each of these steps can fail in different ways.

In standard supervised learning, we would collect human demonstrations of correct trajectories and train the model to imitate them. This is what WebGPT (Nakano et al., 2021) did: it used imitation learning and reinforcement learning from a large number of human demonstrations to train a browser-assisted question-answering agent. But the paper identifies a sharp practical limitation of this approach that motivates their entire research direction (Section 1):

"For outcome-based systems, the solution is usually straightforward: we just collect more human-labeled data. However, acquiring such data is much more challenging and expensive for process-based systems: a significantly larger amount of data is needed (Uesato et al. (2022); Lightman et al. (2023)), and it is generally harder for humans to determine an optimal multi-step trajectory."

This is not a minor inconvenience—it's a fundamental scaling bottleneck. Process supervision requires annotators to evaluate every intermediate step of a reasoning chain, not just the final answer. If an agent takes 5–10 steps to answer a question, human annotators must assess whether each search query was well-formed, whether each snippet selection was relevant, whether the summarization accurately captured the retrieved information, and whether the decision to continue or terminate search was correct at each point. This is substantially more expensive per example than outcome labeling, and the cost compounds because process-based approaches typically need more data to cover the combinatorially larger space of possible trajectories.

Why This Problem Matters: The Rise of LLM Agents and the Training Gap

The paper is situated within a broader trend the authors describe as "explosive growth in techniques (Gao et al. (2023); Madaan et al. (2023)), frameworks (Dohan et al. (2022); Khattab et al. (2023b)), and libraries (Liu (2022), Chase (2022)) for defining process-based workflows with LLMs through human-understandable task decompositions" (Section 1). This explosion means that practitioners are increasingly deploying LLMs not as simple input-output functions, but as multi-step agents that orchestrate tool use, maintain state, and make sequential decisions.

The practical significance extends across several dimensions the paper touches on:

• Complex question answering: Many real-world questions cannot be answered with a single search query or a single reasoning step. They require decomposing the question, searching for multiple pieces of evidence, synthesizing across sources, and verifying that the final answer is both relevant and grounded. This is the specific task the paper's Search Agent targets, but the pattern generalizes.

• Non-differentiable tool interactions: When an agent calls a search API, a database, a calculator, or any external tool, the interaction breaks the gradient flow. You cannot backpropagate through the search engine's ranking algorithm to tell the model that it should have formulated a different query. This means standard end-to-end training is impossible, and alternative training paradigms—reinforcement learning, self-training, or imitation learning from demonstrations—are the only options.

• The training data bottleneck for process-based systems: As noted above, the shift from outcome-based to process-based evaluation creates a data acquisition problem. For outcome-based tasks like classification or short-form QA, we can collect human labels (is the answer correct? yes/no) relatively cheaply at scale. For process-based tasks, each trajectory might require multiple points of human judgment, making it economically infeasible to scale the approach WebGPT-style for every new domain or agent configuration.

The problem is therefore both practical (how do we build and improve agents without massive human annotation budgets?) and structural (how do we provide learning signals for non-differentiable, multi-step processes?).

Prior Approaches and Where They Fall Short

The paper positions itself against several lines of prior work, each of which has specific limitations that the proposed method aims to address:

WebGPT and Imitation Learning from Human Demonstrations

WebGPT (Nakano et al., 2021) demonstrated that a language agent could be trained to answer long-form questions using web search, with training based on imitation learning from human demonstrations and then fine-tuned with RL from human preference comparisons. The paper acknowledges this as tackling "the task of long-form question answering... in which the language agent uses web search as a tool to generate final answers with explicit references" (Section 7). However, the limitation is clear: WebGPT is "focused on imitation learning and RL from a large number of human demonstrations," while the present paper "aims to minimize human involvement" (Section 7). The few-shot exemplars in the agent's prompts are the only labeled demonstrations used.

This is not merely a cost argument—it's an argument about scalability and generality. If every new agent configuration or domain requires a fresh collection of human demonstrations, the approach doesn't scale to the diversity of tasks that LLM agents are being applied to. The paper's ambition is to remove human data from the loop entirely, using only AI feedback.

Prompting-Based Agents (ReAct, Reflexion, DSP)

The dominant paradigm for building LLM agents at the time of this work was manual prompt engineering. The paper cites ReAct (Yao et al., 2022) as the foundational approach: interleaving chain-of-thought reasoning with actions and observations in thought-action-observation rounds. The Search Agent follows this general format, extended with Reflexion-style self-critique (Shinn et al., 2023) where the agent checks its own answer for relevance and grounding.

The paper explicitly states that "setting up language agents with manually designed few-shot prompts is the most common practice" (Section 7). But the limitation of pure prompting is that performance plateaus at the quality of the prompt engineering. There's no learning mechanism—the model doesn't improve with experience, doesn't learn from its mistakes, and its behavior is entirely determined by the static few-shot examples and instructions provided at inference time. The DSP framework (Khattab et al., 2023a) partially addresses this by automatically tuning few-shot demonstrations, but it still requires some amount of labeled training examples for optimization and can only fine-tune specific components of the agent, not the entire multi-step policy end-to-end.

The key insight the paper draws from the prompting literature is that prompting alone gives you a working agent but no improvement mechanism. You can build a ReAct agent with careful prompt design, and it will perform at some baseline level, but without a training signal, you cannot fix its systematic failure modes or adapt it to new domains.

Fine-Tuned Agents (FireAct)

The paper identifies FireAct (Chen et al., 2023) as the closest prior work to its fine-tuning setup, but with a crucial difference: FireAct relies on human labels for training or data filtering. The paper's method, in contrast, is "building synthetic data with self-improvement from AI feedback" (Section 7), without using any human labels for determining whether a trajectory or action is good.

This distinction matters because it means the improvement loop can be run autonomously—generate trajectories, rank them with AI feedback, fine-tune, and repeat—without a human in the loop at any stage. This is what enables the iterative self-improvement to scale: each iteration produces higher-quality data, which in turn enables a better model, which can generate even better trajectories in the next iteration.

Self-Improvement for Outcome-Based Systems (STaR, ReST, ReST$^{EM}$, RAFT)

The paper explicitly connects to a line of work on self-improvement, including STaR (Zelikman et al., 2022), ReST (Gulcehre et al., 2023), ReST$^{EM}$ (Singh et al., 2023), and RAFT (Dong et al., 2023). However, it identifies a critical gap: all four of these papers "target outcome-based systems, while we focus on a process-based one" (Section 7).

This is not a trivial distinction. In STaR and ReST$^{EM}$, the training signal comes from the correctness of the final answer—the model generates reasoning chains, and those that lead to the correct answer are used for fine-tuning. This works for math problems (where correctness is well-defined) or other tasks with ground-truth labels, but it breaks down for open-ended long-form question answering where:

• There may not be a single "correct" answer
• The quality depends on factual accuracy, relevance, grounding, and comprehensiveness, not just a binary match
• Many reasonable answers exist, and many trajectories that fail to produce a perfect answer still contain useful reasoning steps

The paper explicitly notes this distinction: "We also note that, unlike STAR and ReSTEM, we don't use the correctness of the answer as a signal" (Section 7). Instead, the signal comes from the AI ranking model, which evaluates the quality of individual actions (reasoning steps) within a trajectory, regardless of whether the final answer is correct.

Similarly, ReST uses threshold-based filtering with a reward model trained on human preference data, and RAFT uses a reward model to rank sampled responses for fine-tuning. Both assume access to a reward model trained on human preferences. The paper's approach differs in using a zero-shot LLM-based ranking (an instruction-tuned PaLM 2-L prompted to compare actions) rather than a trained reward model, removing the need for human preference data.

How This Paper Positions Itself

The paper's positioning can be understood as synthesizing three ideas that hadn't been combined before:

1. ReAct-style agent architecture (process-based, multi-step, tool-using) as the system to be improved
2. ReST-style iterative self-training (grow the dataset by sampling from the current policy, improve the policy on the fixed dataset) as the improvement algorithm
3. AI feedback (LLM-based zero-shot ranking of actions) as the training signal, replacing both human labels and trained reward models

This synthesis addresses a specific gap: prior self-improvement work targeted outcome-based systems with either ground-truth correctness signals or human-preference-trained reward models. Prior agent work either relied on static prompts or required human demonstrations for fine-tuning. The paper's contribution is showing that process-based agents can be iteratively improved using only AI feedback, without human data at any stage of the training loop.

The framing is explicitly positioned around process supervision versus outcome supervision. The paper emphasizes that "combining a process-based approach (i.e., defining agent as a state machine) with high-temperature exploration, AI feedback (zero-shot 'reward' model used for actions re-ranking), and state-wise fine-tuning over completed trajectories" (Section 6) is what enables learning without outcome labels—the model learns from individual steps even in trajectories that don't produce a correct final answer.

Finally, the paper positions its evaluation carefully. The Bamboogle dataset (Press et al., 2023) serves as a development/evaluation benchmark (effectively a validation set), while BamTwoogle (a new dataset constructed by the authors) serves as a held-out test set. Both are small (125 and 100 questions respectively) but "have enough statistical power to capture the effects we are interested in studying" (Section 1). The auto-eval mechanism, validated against human judgments (Pearson correlation 0.98, Spearman 0.83), provides a scalable way to measure agent performance despite the challenges of evaluating long-form, open-ended answers. This evaluation infrastructure is itself a contribution, enabling the kind of iterative experimentation that would be prohibitively expensive with human-only evaluation.

3. Technical Approach

3.1 Reader Orientation

This paper builds a self-improving search agent—a language model system that can answer complex, open-ended questions by iteratively searching the web, reading results, deciding when it has enough information, and composing a long-form answer with citations. The problem it solves is that such multi-step agents are brittle (they make wrong decisions at any step—bad queries, poor snippet selection, premature termination) and we cannot train them end-to-end because calling external tools like search engines breaks gradient flow. The solution takes the shape of a closed-loop data engine: a large prompted model generates reasoning trajectories, an AI judge ranks which intermediate actions were best, those trajectories become training data for fine-tuning, and the improved model then generates higher-quality trajectories in the next iteration—all without human labels.

3.2 Big-Picture Architecture (Diagram in Words)

The system has six major components that form two nested loops:

The Agent (inference-time flow): A ReAct-style state machine with five distinct reasoning steps executed sequentially—(1) decide whether to search again or terminate, (2) if searching, summarize the retrieved search snippets and select relevant links, (3) when done searching, generate a draft answer from collected evidence, (4) self-check that the answer is relevant to the original question, (5) self-check that the answer is grounded in the retrieved sources—producing a final attributed answer.

The Data Engine (training-time loop): An outer "Grow" stage where the latest policy generates complete trajectories over a fixed set of 2000 questions, and an inner "Improve" stage where those trajectories are decomposed into per-step training examples, optionally re-ranked by an AI reward model, and used to fine-tune the agent model.

The AI Reward Model: A zero-shot prompted instruction-tuned PaLM 2-L that ranks multiple candidate actions per step based on relevance, groundedness, and efficiency, replacing the need for human preference data or trained reward models.

The Auto-Eval Judge: A separate prompted PaLM 2-L that compares a model's long-form answer to a reference answer and outputs a binary judgment (implies/does not imply), enabling scalable performance measurement without human evaluation.

The Fine-Tuning Pipeline: Full fine-tuning of PaLM 2 base models (XS, S, L) on a mixture constructed by splitting each completed trajectory into its constituent reasoning steps, with each step's input being the full agent state up to that point and the target being the selected action.

The Iteration Controller: A meta-process that decides whether to run another iteration by evaluating the fine-tuned model's auto-eval performance and, if improved, uses that model as the new policy for the next "Grow" stage.

Information flows as follows: 2000 seed questions enter the Grow stage → the current policy model generates complete trajectories (search loop → answer draft → self-checks → final answer) with temperature 0.5 and multiple samples per step → the AI reward model re-ranks candidate actions per step → trajectories are decomposed into fine-tuning examples → the target model is fine-tuned → auto-eval measures improvement → if improved, the fine-tuned model becomes the new policy for the next iteration. Distillation happens simultaneously: smaller models are fine-tuned on the same high-quality data produced by the large model's trajectories.

3.3 Roadmap for the Deep Dive

• First, the Search Agent's state machine and reasoning steps — since every trajectory, every fine-tuning example, and every evaluation depends on understanding exactly what decisions the agent makes and in what sequence.
• Second, the "code as prompt" design philosophy and the specific few-shot prompts — because prompt design is the only human supervision in the entire system, and the code-formatting choice determines which models can even participate.
• Third, trajectory generation mechanics — sampling temperature, multi-sample selection via perplexity, and the cost model for choosing how many samples to draw per step.
• Fourth, the AI reward model and re-ranking procedure — the mechanism that provides the training signal, and how it differs from ReST's threshold-based filtering.
• Fifth, fine-tuning and data mixture construction — how trajectories become training data, and the crucial detail that state-wise decomposition enables learning from "bad" trajectories.
• Sixth, the iterative self-improvement algorithm (Grow + Improve + Distill) — the outer loop that ties everything together, including the differences from canonical ReST.
• Seventh, the auto-eval infrastructure — the mechanism that makes the whole closed loop practical by providing cheap, validated performance measurement.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems paper whose core idea is that ReST-style iterative self-training, previously demonstrated only for outcome-based systems, can be adapted to process-based multi-step agents by replacing the outcome reward signal with AI feedback on individual reasoning steps, enabling self-improvement and self-distillation without any human-labeled data.

The Search Agent's State Machine (Inference-Time Architecture)

The agent follows a fixed five-phase execution flow, depicted as a state machine in Figure 2. The paper explicitly calls this "a flavor of ReAct agent with Reflexion" (Section 2): the ReAct component is the interleaving of reasoning and search actions, and the Reflexion component is the post-answer self-critique steps.

Phase 1: The Search Decision Loop (Search/Terminate). This is a repeating decision cycle. At each cycle, the agent's prompt includes the ORIGINAL_QUESTION, the full PAST_ACTIONS history (all previous search queries, link selections, and grounded summarizations), and a REMAINING_SEARCHES counter (initialized to at most 10). The agent must output an ActionWrapper containing two fields: thoughts (a string explaining the reasoning for the next action) and action (either a Search command specifying a query string, or a Terminate command signaling that enough information has been gathered). If Search is selected, the agent calls the internal Google Q&A API with the specified query, receives back top-3 snippets (each with a link_id, link_text, and snippet), and transitions to the summarization step. If Terminate is selected, the loop exits and answer generation begins.

The design choice of limiting searches to at most 10 prevents the agent from entering infinite search loops. The paper notes this is enforced through the REMAINING_SEARCHES field in the prompt, which "preserves the trajectory state" (Section 3.2) by providing the agent with explicit awareness of its remaining budget.

Phase 2: Search Result Summarization. When the agent has chosen to search, the next step is to process the returned results. The summarization prompt receives the ORIGINAL_QUESTION, PAST_ACTIONS, and CURRENT_SEARCH_RESULTS (the top-3 snippets from the query). The agent must output a LinkSelection action containing: thoughts (reasoning about which links are relevant), selected_link_ids (a list of integers identifying the chosen snippets), and grounded_summarization (a natural language summary that extracts the relevant information from the selected snippets and explicitly cites them using [link_id=X] notation).

This step is where the agent practices grounded attribution—every piece of information it extracts is tagged with the source link ID. This is critical for the subsequent self-checks, which verify that the final answer is supported by these cited sources. The summarization prompt is 6-shot (Listing 2 shows a 1-shot fragment), and the exemplars demonstrate the pattern of selecting subset of links with explicit reasoning about why each was included or excluded.

Phase 3: Answer Generation. Once the search loop terminates (the agent outputs a Terminate action), the answer generation prompt receives the ORIGINAL_QUESTION and the full PAST_ACTIONS history (which now contains all search queries, all link selections, all summaries, and the terminate action). The agent outputs an Answer action with thoughts (meta-reasoning about what to include) and answer (a paragraph-length response with inline citations, e.g., "according to [link_id=1]"). The 5-shot prompt (Listing 3 shows a 1-shot fragment) demonstrates that the agent should synthesize information across multiple search rounds, omit information from searches that turned out to be irrelevant, and produce a coherent narrative supported by citations.

Phase 4: Relevance Self-Check. The first of two self-critique steps, adopted from Reflexion (Shinn et al., 2023). The relevance check prompt receives the ORIGINAL_QUESTION, the full PAST_ACTIONS (including the generated answer), and the ANSWER text. The agent must output either a Check_Answer(passed=True) signaling that the answer addresses the original question, or a Revise_Answer(revised_answer=...) with an improved version if the check fails. The 6-shot prompt (Listing 5 shows a 1-shot fragment) teaches the agent to verify that the answer directly responds to what was asked—not a different question, not a tangential fact, but the specific information requested.

Phase 5: Grounding Self-Check. The second self-critique step verifies that all factual claims in the answer are supported by the sources cited in the PAST_ACTIONS history. The grounding check prompt receives the same inputs as the relevance check and follows the same output format (Check_Answer or Revise_Answer). The 5-shot prompt (Listing 6 shows a 1-shot fragment) demonstrates verifying that each citation in the answer actually says what the answer claims—checking, for example, that [link_id=1] genuinely states that "Jim Betts' competitor in 1980 was John Glenn" if the answer asserts it. The paper notes in the Discussion (Section 6) that these self-critique steps "have a small but positive effect on the overall performance of our multi-step reasoning setup" (on the order of 0.5-1.0% for most models, as detailed in Appendix Table 4), and that the benefit "depends on the model size (larger for larger models) but does not seem to be affected by the self-improvement process."

The final answer is whatever text emerges after both self-checks (if both pass, the original answer; if either triggered a revision, the revised version).

The "Code as Prompt" Design Philosophy

The paper makes a deliberate and non-obvious design choice: all prompts for the agent's reasoning steps are formatted as Python code rather than natural language instructions. Each prompt begins with a docstring describing the agent's function, imports dataclasses and type annotations, defines the action classes (Search, Terminate, Answer, LinkSelection, Check_Answer, Revise_Answer, etc.) as Python dataclasses, and then presents the few-shot examples as blocks of executed code with # [END] markers separating examples.

For example, Listing 1 (Decision Step prompt fragment) begins:

"""Implement an agent capable of answering complex queries by potentially search multiple times.
"""
import dataclasses

class Action:
    """Base class for different actions."""
    ...

@dataclasses.dataclass
class ActionWrapper:
    """Encapsulates the reasoning as well as the selected action.
    Attributes:
        thoughts: Record your thoughts on why we should do this action.
        action: The actually selected action.
    """
    thoughts: str
    action: Action

The model is then expected to continue this code—outputting valid Python that instantiates the appropriate action class with the correct fields.

The paper justifies this approach with three observations (Section 3.1):

• Parseability: "There is often a need to parse the LLM's output for integration with other systems and tools, which is much easier to do if the model's input and output are well-structured." Python code is inherently parseable—you can extract the action field from the generated ActionWrapper instantiation reliably, whereas free-form natural language would require fragile regex parsing or another LLM call.

• Dual nature of code: "code uniquely combines a structured aspect (keywords and syntax) with a natural language aspect (comments and descriptive naming)." This means the prompts can leverage both—the dataclass fields provide the formal schema, while docstrings, comments, and descriptive variable names communicate intent in natural language.

• LLMs can read and write code: The paper notes that "LLMs are capable of both reading and writing code," making this a viable communication medium. However, they also found that "only the PaLM 2-L (Anil et al., 2023) can perform it consistently well" out of the pre-trained models they tried (Section 3.1), which constrains which models can serve as the initial prompted teacher.

The practical consequence of this design choice is that the agent's interface is strongly typed in a machine-readable format. When the model generates Search(query='ikea klippan loveseat dimension'), the system can directly parse the query field and pass it to the search API without any natural language understanding step. This is particularly important for the tool-calling integration—the agent doesn't just describe what it wants to do; it produces executable commands.

Trajectory Generation Mechanics (How Rollouts Work at Test Time)

When the agent is run on a question, the generation process involves controlled stochasticity and selection:

Sampling temperature. The paper uses temperature $T = 0.5$ for all trajectory generation, determined through grid search using auto-eval as the selection criterion (Section 4.2 states: "What is the optimal sampling temperature for the agent? ($T = 0.5$)" as one of the questions auto-eval answers). Non-zero temperature is essential for two reasons: it enables exploration during the "Grow" stage (producing diverse trajectories that cover different failure modes and successes), and it creates variance that makes the multi-sample selection meaningful—if every sample were identical, there would be nothing to select among.

Multi-sample step selection. At each reasoning step (decision, summarization, answer generation, self-checks), the model generates multiple candidate outputs and one is selected to continue the trajectory. Specifically: "We usually produce multiple samples (with $T = 0.5$, see Section 4.2 for details on temperature selection) for each step and then select the one with the lowest perplexity (4 samples for L and S, 16 for XS) to continue the trajectory forward" (Section 3.2).

The sample counts differ by model size: 4 samples for the large and small models (L and S), 16 for the extra-small model (XS). This is a computational trade-off—larger models are more expensive to sample from, but also produce more coherent outputs on average, so fewer samples are needed to find a good candidate. The XS model is cheaper per sample but less reliable, so more samples are drawn to compensate.

Perplexity as the default selection criterion. For the "on-policy" trajectory rollouts (the actual trajectories used to answer questions at test time or to generate data for the next training iteration), the selection mechanism is based on the model's own perplexity score for each candidate. The paper chooses perplexity as the default because it requires no external judge or additional model call—it's a direct output of the generation process. Lower perplexity indicates the model considers that output more likely under its own distribution, which is a rough proxy for coherence and format correctness.

However, the paper acknowledges that "we might be able to do better than that by utilizing a more sophisticated way of selecting the best sample" (Section 3.5). This leads to the AI reward model, used off-policy (during data preparation for fine-tuning, not during live trajectory execution).

Trajectory state preservation. The agent maintains state through the PAST_ACTIONS field, which accumulates all actions taken so far. As the agent proceeds through the search loop, each new search query, link selection, and grounded summarization is appended. When multiple searches are needed to gather all information (e.g., search for part A, then search for part B, then synthesize), the full history provides context for both the decision step (what have we already searched for?) and the answer generation step (what information do we have available?). This is standard ReAct state management: the LLM sees its entire history at each decision point.

Search API details. The agent uses "internal Google Q&A API that returns top-k snippets, and we typically request top-3 snippets per query from it" (Section 3.2). The top-3 selection means the agent receives a manageable amount of text to process per search—not the full search results page, but three snippets that are likely to contain the relevant information. The agent's summarization step then decides which of these three snippets (if any) contain useful information.

Termination constraint. The agent is limited to "at most 10 per single agent trajectory" searches, enforced through the REMAINING_SEARCHES field in the decision step prompt. This is a hard cap that prevents the agent from searching indefinitely. The decision step prompt shows REMAINING_SEARCHES: int = 4 in the example, indicating that the agent is aware of its remaining budget and should plan accordingly (e.g., it might combine multiple pieces of information into a single query, or decide it has enough information even if not perfect).

The AI Reward Model and Re-Ranking Procedure

This is where the paper's approach diverges most significantly from standard ReST. Instead of training a reward model on human preference data (as in canonical ReST) or using ground-truth answer correctness as a signal (as in STaR/ReST$^{EM}$), the paper uses a zero-shot prompted instruction-tuned LLM to rank candidate actions.

The reward model prompt. The full prompt is shown in Appendix Listing 8. It is 0-shot and uses natural language (not the "code as prompt" approach), specifically because it's intended for an instruction-tuned model rather than a base model. The prompt structure is:

1. Rater instructions defining what constitutes a good action: "The goal of this rating is to filter out bad actions, so that they'll be excluded from the fine-tuning dataset. Overall, we want the agent to produce relevant and grounded answers with minimal steps. Anything that deviates from this goal is considered bad." An additional rule: "If any element (thoughts, comments etc.) is empty, then it's automatically bad."

2. Model input (the state the agent saw at that step—the ORIGINAL_QUESTION, PAST_ACTIONS, and any other context for that reasoning step).

3. Candidate outputs — "Model Output #1", "#2", "#3", and "#4" — each being one of the sampled actions for that step. The paper generates 4 samples for L and S models, 16 for XS (Section 3.2), but the ranking prompt is shown with 4 candidates, suggesting that for the L and S models, all candidates are presented simultaneously, while for XS, the ranking might be done in batches or the full set might be shown (the paper does not specify the exact number presented to the ranking model).

4. Output format instructions: "Output 3 lines when answering and make sure to follow the precise format. Explanation: why you think model output #X is the best; Answer: #X; Ranking: #X > #Y > ..."

The reward model is an instruction-tuned PaLM 2-L — the same size as the initial prompted teacher, but the instruction-tuned variant rather than the base model. The paper does not specify which instruction-tuned variant is used, but the key property is that it can follow the ranking instructions reliably without few-shot examples (it's 0-shot).

What the ranking captures. The reward model evaluates actions holistically based on the criteria specified in the prompt: relevance to the original question, grounding in the retrieved evidence (if applicable), and efficiency (minimal steps). It does NOT evaluate based on whether the final answer is correct—it evaluates the quality of the individual step given the context. This is the crucial property that enables learning from trajectories that ultimately fail: an action can be "good" (relevant search query, accurate summarization) even if the trajectory later goes wrong, and a "bad" action (empty thoughts, irrelevant query) can be filtered out.

Off-policy application. The paper states: "we mainly do the LLM-based selection off-policy, i.e., by updating the current action used for fine-tuning. The on-policy trajectory rollouts use perplexity" (Section 3.5). This means:

• During trajectory generation (the "Grow" stage), the agent uses perplexity-based selection to choose which action to take at each step and continue the trajectory. This is because calling the reward model at every step would make trajectory generation prohibitively expensive—each step would require an additional PaLM 2-L inference to rank candidates.

• After trajectories are collected, when building the fine-tuning mixture, the reward model is applied to the stored candidate sets. For each step in each trajectory, the 4 (or 16) candidate outputs that were generated are fed to the reward model, and the highest-ranked output replaces the perplexity-selected output as the training target.

This off-policy approach means the reward model influences what the model learns (the training data), not how the model behaves during data collection (the trajectories). This is a practical compromise: the ranking model improves data quality without increasing the cost of trajectory generation.

Relationship to ReST and RAFT. The paper explicitly contrasts its ranking approach with both methods (Section 3.5):

• ReST uses "threshold-based filtering with a reward model (RM) trained on human preference data" — the RM assigns a score to each trajectory, and trajectories above a threshold are kept. The paper's approach differs in two ways: the RM is zero-shot prompted (not trained on human preferences), and it does ranking (relative comparison among candidates) rather than absolute scoring.

• RAFT uses "the reward model ranks sampled responses to select high-scoring subsets for model fine-tuning, and the RM rankings matter much more than absolute scores." The paper's approach is closer to RAFT in spirit, but again differs in using a zero-shot LLM judge rather than a trained reward model.

The paper's RM is therefore best characterized as a zero-shot pairwise ranking model implemented via prompted LLM, applied to per-step actions rather than complete trajectories.

Fine-Tuning and Data Mixture Construction

Once trajectories are collected and actions are re-ranked, the system converts them into a supervised fine-tuning dataset. This conversion is where the process-based approach enables learning from partial successes.

Trajectory decomposition. Each complete trajectory is "split... into the reasoning steps" (Section 3.4). Concretely, if a trajectory consists of (Search 1 → Summarize 1 → Search 2 → Summarize 2 → Terminate → Answer → Relevance Check → Grounding Check), it produces 8 separate fine-tuning examples. Each example has:

• Input: The full prompt template for that reasoning step, filled with the appropriate state variables (ORIGINAL_QUESTION, PAST_ACTIONS up to that point, CURRENT_SEARCH_RESULTS if applicable, etc.).
• Target: The selected action output for that step (either the perplexity-selected action for the on-policy trajectory, or the RM-re-ranked best action for the improved training data).

Why state-wise decomposition matters. The paper emphasizes that "the model can learn something useful even from the states that eventually lead to the wrong final answer" (Section 6). Consider a trajectory where:

• Search 1 retrieves excellent results and the summarization is accurate.
• The agent then makes a poor decision to search again when it already has enough information.
• The final answer, while containing correct facts, misses a key piece because of the unnecessary second search diverting attention.

In outcome-based training (where only trajectories with correct final answers are used), this trajectory would be discarded entirely—the accurate summarization from step 2 would be lost. In the paper's state-wise decomposition, the summarization step can still serve as a positive training example even though the overall trajectory was suboptimal. This is the power of process-based feedback: good individual actions are rewarded regardless of the trajectory outcome.

Mixture construction. The fine-tuning mixture is simply the concatenation of all decomposed steps from all trajectories. The paper uses "full fine-tuning for all the experiments" (Section 3.4)—all model parameters are updated, not just a subset or adapter layers. The size of the mixture varies by iteration:

• Pilot data (unfiltered): 500 trajectories, producing 4,518 training examples (Section 5.3, Table 3).
• 1st gen: 2,000 trajectories, producing 17,970 training examples.
• 2nd gen (1x): 2,000 trajectories (one per question), producing 18,007 training examples.
• 2nd gen (2x): 4,000 trajectories (two per question), producing 36,238 training examples.
• 2nd gen (4x): 8,000 trajectories (four per question), producing 72,424 training examples.

The number of examples per trajectory varies because different trajectories have different numbers of search steps. Table 3 shows that for pilot unfiltered data, 500 trajectories produce 4,518 examples (approximately 9 steps per trajectory on average), while for 2nd gen, 2,000 trajectories produce about 18,000 examples (approximately 9 steps per trajectory as well).

Model sizes and training budgets. The paper fine-tunes three model sizes: PaLM 2-XS, PaLM 2-S, and PaLM 2-L. The authors note that "fine-tuning costs increase sharply for larger models" and therefore "do as many experiments as possible with XS model" (Section 3.4). Checkpoint selection is also model-size-dependent: "step 9K for XS, 5K for S, 3.5K for L" (Section 4.2), determined using auto-eval as the selection criterion. The paper does not specify whether these "steps" refer to gradient update steps or training examples processed, but the decreasing trend with model size is consistent with larger models converging faster (requiring fewer updates to fit the same data).

Early stopping with auto-eval. Rather than using a held-out validation set with ground-truth labels (which don't exist—the training data is synthetic with no labels), the paper uses Bamboogle auto-eval as a proxy validation metric to select checkpoints and determine "Should we proceed with another iteration of self-improvement?" (Section 4.2). This is structurally a form of validation on a different task than the training data, which tests generalization. The fact that Bamboogle is never used as a training set is explicitly stated: "we neither tune our prompts on Bamboogle nor use questions from it to generate fine-tuning trajectories" (Section 4.2).

Human filtering ablation. As a baseline, the paper manually reviewed 500 pilot trajectories and "filter out about 30% of the examples that are 'bad' in some way: an unhelpful query, empty thoughts, summary missing important information, etc." (Section 5.1). Surprisingly, this human-filtered data produces worse performance than the unfiltered version (44.7% vs. 47.2% for PaLM 2-XS, Table 3). The paper hypothesizes this is because:

• The reduced mixture size (3,015 vs. 4,518 examples) makes it harder for the model to learn proper prompt format, and
• Filtering only removes the immediate "bad" example, but that same "bad" step persists in the PAST_ACTIONS context of subsequent steps in the same trajectory, so the model still sees the problematic content.

This is an important negative result: it suggests that simple human filtering of process-based data may be counterproductive because the interdependence of steps means that "cleaning" one step doesn't fully remove its influence from the training data.

The Iterative Self-Improvement Algorithm (Grow + Improve + Distill)

This is the outer loop that ties the entire system together. The paper describes it as a sequence of steps (Section 3.6):

Initialization. "Start with a model capable of performing Search Agent task at a certain level, for example, with prompted PaLM 2-L model" (Section 3.6). The prompted model serves as the initial "policy" $\pi_0$. This model generates the first batch of trajectories.

Iteration 1 (Grow + Improve).

• Grow: Use $\pi_0$ (prompted PaLM 2-L) to "collect reasoning trajectories from this model based on our set of 2000 initial questions" (Section 3.6). The 2000 questions are drawn from four datasets: HotpotQA (500), Eli5 (500), Eli5-askH (500), and Eli5-askS (500). The paper explicitly notes that "we don't use any other information from these datasets, like labels" (Section 3.3)—the questions are used purely as seeds for trajectory generation, no ground-truth answers or human judgments are involved.

• Improve: "Convert the trajectories into the fine-tuning mixture. Apply re-ranking with RM during the conversion" (Section 3.6). Specifically, for each step in each trajectory, the 4 (or 16) candidate outputs generated during trajectory collection are fed to the AI reward model, and the top-ranked candidate replaces the (perplexity-selected) candidate as the training target. The paper notes this is "roughly equivalent to the 'improve' stage of ReST, though we only do one iteration of 'improve'" (Section 3.6)—meaning the ranking is done once on the fixed collected data, rather than iteratively refining the data within the improve stage.

• Fine-tune: Train "the new model (of the same size) on this mixture" (Section 3.6), producing $\pi_1$ for each model size. The paper notes that fine-tuning is done on all three sizes (XS, S, L) using the same mixture data.

• Verify: Run auto-eval on Bamboogle to check that $\pi_1$ outperforms $\pi_0$ (or at least the prompted baseline). The paper reports that 1st gen fine-tuned L improves from 70.3% (prompted) to 74.0%, while XS reaches 54.4% (Table 1).

Iteration 2 (Second Grow + Improve).

• Grow: Use $\pi_1$ (PaLM 2-L fine-tuned on 1st gen data) as the policy to generate a larger set of trajectories: "8000 trajectories (using the same 2000 initial questions, each repeated four times; analogous to a 'grow' stage in ReST)" (Section 5.2). The key insight here is that the policy model is now the improved version, so the generated trajectories should be higher quality even before re-ranking—better search queries, more accurate summarizations, better-grounded answers.

• Improve: Apply RM re-ranking to these 8,000 trajectories and build the 2nd gen fine-tuning mixture.

• Fine-tune: Train all three model sizes on the 2nd gen mixture, producing $\pi_2$ for each.

• Results: The 2nd gen L reaches 76.1% Bamboogle auto-eval (up from 74.0% at 1st gen), 2nd gen S reaches 69.7% (up from 61.9%), and 2nd gen XS reaches 65.9% (up from 54.4%). The 2nd gen XS model at 65.9% is approaching the original prompted L model's 70.3%—a model roughly two orders of magnitude smaller.

Differences from canonical ReST. The paper makes several modifications to the ReST algorithm (Gulcehre et al., 2023) to adapt it to the process-based agent setting:

1. Fixed question set: "we keep the set of initial questions fixed" (Section 3.6) rather than expanding the question set with new questions discovered during search or interaction. This simplifies the data management but means the model's improvement is bounded by the diversity of the initial 2,000 questions.

2. Only one improve step per grow: Canonical ReST involves multiple inner-loop "improve" iterations where the reward model is applied to re-rank or filter the data, and the policy is incrementally updated. The paper does "only one iteration of 'improve'" (Section 3.6)—one round of RM re-ranking followed by one fine-tuning step. This is a practical simplification that reduces computational cost.

3. Zero-shot AI feedback instead of trained RM: As discussed above, the ranking signal comes from a prompted LLM rather than a reward model trained on human preferences. This removes the need for human preference data entirely.

4. Process-based rather than outcome-based: The training signal is applied at the level of individual reasoning steps (actions) rather than complete trajectories. This is the fundamental adaptation that makes the approach work for multi-step agents rather than single-output models.

Self-distillation as a byproduct. The paper notes that "we can also train smaller models on the fine-tuning data from the different iterations of self-improvement, which will naturally give us a self-distillation algorithm" (Section 3.6). This is not a separate process—it's an inherent property of the data pipeline. Since the large model generates the training data, and the same data can be used to fine-tune models of any size, the iterative improvement loop simultaneously serves as a distillation loop: as the large model improves, the data quality improves, and smaller models trained on that data inherit the improvements.

The results in Table 1 demonstrate this dramatically:

• PaLM 2-XS on 2nd gen data: 65.9% (compared to prompted L: 70.3%, 2nd gen L: 76.1%)
• The model is approximately two orders of magnitude smaller than L, yet achieves comparable performance—within about 4 percentage points of the prompted teacher and within about 10 points of the fine-tuned teacher.

Multiple trajectories per question. A practical choice is how many different trajectories to generate per input question for the fine-tuning mixture. The paper reports (Section 5.3): "it helps to use two trajectories per question instead of 1 (2.2% gain) in the fine-tuning mixture, but more than that doesn't improve performance significantly." Table 3 shows this concretely for PaLM 2-XS: 2nd gen with 1 trajectory per question yields 63.4%, 2 trajectories yields 65.6% (+2.2%), and 4 trajectories yields 65.9% (+0.3% over 2x). The diminishing returns suggest that two trajectories per question provides sufficient diversity to cover different successful strategies and failure modes, and additional trajectories are largely redundant.

Quality vs. quantity. The paper emphasizes that "the quality of the data (e.g., 9% gain, when going from 1st gen to 2nd gen (1x) while keeping the size of the data roughly the same) matters more than its quantity" (Section 5.3). The comparison: 1st gen (17,970 examples) yields 54.4% for XS, while 2nd gen 1x (18,007 examples) yields 63.4%—a 9-point improvement with nearly identical dataset size. This is because the 2nd gen data was generated by a better policy (the 1st gen fine-tuned model), so the trajectories contain fewer bad actions and higher-quality reasoning even before RM re-ranking. "Notably, better data also reduces the variance of evaluation trajectories" (Section 5.3)—the standard deviation drops from ±3.6% for 1st gen to ±1.7% for 2nd gen (1x), suggesting that higher-quality training leads to more consistent agent behavior.

The Auto-Eval Infrastructure

The entire self-improvement loop depends on being able to measure agent performance cheaply and reliably. The paper develops an auto-eval system that replaces human judgment with a prompted LLM evaluator.

The need for auto-eval. The paper identifies two problems with human evaluation (Section 4.2):

• Cost: "doing one human eval is much easier than doing five," so human evaluations are typically sparse (one or a few judgments per question).
• Variance: "the agent's trajectories are stochastic (as a reminder, we use non-zero temperature when sampling reasoning steps), but we can't easily reduce the variance by increasing the number of repetitions per question with human evals." To get reliable performance estimates, you need many trajectories per question, but human evaluation at that scale is impractical.

Auto-eval solves both: it's cheap enough to run "a large number of repetitions to reduce variance," and the paper "typically aggregate[s] auto-eval over ten repetitions" (Section 4.2)—meaning for each Bamboogle question, 10 independent trajectories are generated and auto-evaluated, and the reported accuracy is the average across all questions and repetitions.

The auto-eval prompt. The full prompt is shown in Appendix Listing 7. It is 5-shot and formatted as Python code (consistent with the "code as prompt" approach). The prompt defines a Check_Answer function that takes three arguments: ORIGINAL_QUESTION, ANSWER (the model's long-form answer), and REF_ANSWER (the ground-truth answer). It must return True if the ANSWER "implies" the REF_ANSWER, and False otherwise.

The five few-shot exemplars demonstrate the judgment criteria:

• Example 1: Question about two mines being in the same country; model answer says "yes, they are in the same country"; reference answer is "yes"; judgment: True ("the ANSWER implies the answer to the original question is yes, this is consistent with the REF_ANSWER").

• Example 2: Question about which director has 67 films; model answer mentions one director with 5 films; reference answer is about the other director having 67 films; judgment: False ("the ANSWER does not imply the REF_ANSWER because ANSWER does not mention Paul Ludwig Stein").

• Example 3: Question about two genera in the same family; model answer says "we are not sure"; reference answer is "no"; judgment: False ("we cannot infer the REF_ANSWER given the ANSWER"—the model failed to determine the answer, even though the reference provides it).

• Example 4: Question about birth year; model answer correctly states the winner and their birth year; reference answer is that year; judgment: True (exact factual match).

• Example 5: Model says "the question is ill-formed or out-of-date"; judgment: False (the model refused to answer, so it doesn't imply the reference).

The key design choice is the use of implication rather than exact match or similarity. The auto-eval asks: "does the model's answer logically imply the reference answer?" This accommodates the long-form, verbose nature of the agent's answers—the model might provide context, cite sources, and phrase the answer differently from the reference, but if the factual content is consistent with the reference, it's counted as correct. This is a more forgiving evaluation than exact string matching, but more principled than unconstrained LLM-as-judge approaches because it's anchored to a specific logical criterion (implication).

Validation against human judgments. The paper validates the auto-eval by comparing it to human evaluations "on a diverse set of agents" (Section 4.2). The results:

• Pearson correlation: 0.98 with $p = 6.6 \times 10^{-8}$
• Spearman correlation: 0.83 with $p = 0.0015$

The near-perfect Pearson correlation indicates that the auto-eval and human ratings have a strong linear relationship—when humans think performance is higher, auto-eval does too, and to the same degree. The Spearman correlation (rank-based, less sensitive to outliers) at 0.83 confirms that the relative ordering of models by auto-eval matches human rankings reasonably well, though not perfectly. The paper doesn't report how many human evaluations were used for this validation or how many models/agents were included in the comparison set.

Auto-eval as a proxy validation set. Beyond final performance evaluation, the paper uses auto-eval to answer questions that would normally require a validation set (Section 4.2):

• Optimal sampling temperature selection ($T = 0.5$)
• Checkpoint selection per model size (9K steps for XS, 5K for S, 3.5K for L)
• Whether to proceed with another iteration of self-improvement
• Impact of multiple trajectories per question on the fine-tuned model
• Whether self-checks are helping or hurting (they help "slightly")

The paper emphasizes that "we never use Bamboogle as a training set, as we neither tune our prompts on Bamboogle nor use questions from it to generate fine-tuning trajectories" (Section 4.2). This preserves Bamboogle as a clean evaluation set, analogous to a validation set in standard machine learning, despite the paper not having a traditional train/val/test split because no labeled training data exists.

Computational cost of auto-eval. The auto-eval itself requires a separate PaLM 2-L call per evaluation. Since the paper aggregates over 10 repetitions per question for 125 Bamboogle questions, each model evaluation requires 1,250 auto-eval calls (plus the cost of actually running the agent 1,250 times to generate the trajectories being evaluated). The paper acknowledges this cost implicitly by noting that auto-eval "increases the computational costs significantly due to the need to run agent trajectories multiple times, as well as the use of PaLM 2-L model for auto-eval" (Section 6). This is a practical limitation—the auto-eval infrastructure is itself computationally expensive, even though it's cheaper than human evaluation.

BamTwoogle as the held-out test set. Because Bamboogle is used repeatedly for model selection (temperature, checkpoints, iteration decisions), there is a risk of overfitting to its specific distribution. The paper introduces BamTwoogle as "a test set... used exclusively to measure the final performance of the models" (Section 4.3). BamTwoogle is a handcrafted dataset of 100 questions, designed to be "a complementary, slightly more challenging sequel to Bamboogle" that addresses specific shortcomings: ensuring all questions require 2+ steps, verifying that answers don't appear on the first page of Google search results, and preferencing Wikipedia as the source of truth. Table 2 reports human evaluation results on BamTwoogle for the final models, serving as out-of-distribution validation that the self-improvement gains are real and not an artifact of Bamboogle-specific optimization.

Summary of Design Choices and Their Justifications

• Code-format prompts over natural language: enables reliable parsing of model outputs for tool integration; leverages code's dual structured/natural language nature; constrains which models can serve as the teacher (only PaLM 2-L was found to handle this consistently).

• State-wise trajectory decomposition over outcome-based filtering: allows learning from individual good actions even in trajectories that fail overall; critical for open-ended questions where a single "correct" answer doesn't exist; enables process supervision without human process labels.

• Zero-shot LLM ranking over trained reward model: removes the need for human preference data to train a reward model; uses an instruction-tuned model's general quality judgment; applied off-policy to keep trajectory generation costs manageable.

• Perplexity selection for on-policy rollouts over RM selection: avoids the computational cost of calling a separate large model at every step of trajectory generation; allows the RM's influence to be on training data quality rather than exploration behavior.

• Fixed question set for Grow stages over expanding question set: simplifies data management; ensures comparability across iterations; bounds the improvement to the diversity of the initial seed questions.

• Two trajectories per question (rather than one or many): provides sufficient diversity to cover different successful strategies; diminishing returns beyond two per question in the experiments.

• Auto-eval with implication-based judgment over exact match or human evaluation: enables cheap, high-repetition evaluation; validated against human judgments (Pearson 0.98); uses a principled logical criterion (implication) rather than arbitrary similarity scoring.

• Bamboogle as development set, BamTwoogle as test set: prevents overfitting from repeated model selection decisions; addresses specific shortcomings of Bamboogle discovered during human evaluation.

• Temperature 0.5 for all generation: selected via auto-eval grid search; non-zero temperature enables exploration during data generation and creates variance that makes multi-sample selection meaningful.

4. Key Insights and Innovations

Innovation 1: Process-Based Self-Improvement Without Outcome Labels

The paper's most fundamental conceptual move is demonstrating that multi-step agents can be iteratively improved without EVER checking whether the final answer is correct. This breaks from the dominant paradigm in self-improvement literature, where the training signal comes from filtering trajectories based on whether they reached the right answer (STaR, ReST$^{EM}$) or from a reward model trained on human preference comparisons (ReST, RAFT). The paper explicitly distances itself from both: "unlike STAR and ReSTEM, we don't use the correctness of the answer as a signal" and "unlike ReST and RAFT, we don't have the proper reward model trained on human preferences" (Section 7).

Why does this matter conceptually? Because it decouples the improvement mechanism from task-specific ground truth. For math problems, correctness is well-defined—you can check whether the final number matches. For open-ended long-form question answering, there is no single "correct" answer to check against. The quality of a long-form answer depends on factual accuracy, relevance, comprehensiveness, and grounding in sources—properties that don't reduce to a binary match against a reference string. The paper's approach sidesteps this entirely by providing learning signal at the level of individual reasoning steps (is this search query well-formed? is this summary accurate? is this answer relevant?), evaluated by an LLM judge rather than by outcome verification.

The evidence that this works is in Table 1: the 2nd gen PaLM 2-XS model reaches 65.9% on Bamboogle auto-eval after being trained exclusively on synthetic data where the training signal came from AI feedback on intermediate actions, not from knowing which final answers were correct. The paper emphasizes this explicitly in the Discussion (Section 6): "the model can learn something useful even from the states that eventually lead to the wrong final answer." This is process supervision implemented entirely through synthetic data generation, without human process labels (unlike Lightman et al., 2023, which required human annotators to label every intermediate step) and without outcome verification (unlike STaR, which required correct final answers to filter reasoning chains).

The conceptual shift here is moving from filtering good trajectories (the outcome-based paradigm) to extracting good actions from all trajectories (the process-based paradigm). A trajectory that ultimately produces a suboptimal answer still contains well-executed steps—accurate summarizations, relevant search queries, correct decisions to terminate—that can serve as positive training examples when decomposed into state-action pairs. This is only possible because the paper's architecture explicitly defines the agent as a state machine with named reasoning steps (Figure 2), making it natural to decompose trajectories into independently-trainable components. This is a fundamental contribution to how we think about training data for agentic systems: you don't need to know whether the whole trajectory succeeded; you only need to know, for each individual decision, whether it was reasonable given the context.

Innovation 2: The Difficulty-Estimation-Free Self-Improvement Loop

Traditional self-improvement pipelines (STaR, ReST$^{EM}$, even the ReST paper that this work adapts) face a fundamental tension: you need the model to succeed often enough on its initial attempts to generate useful training data, but if the model already succeeds consistently, you don't need self-improvement. This is sometimes called the "cold start" problem or the exploration bottleneck. STaR addresses it by using rationales—even when the final answer is wrong, the reasoning chain might contain useful steps. But STaR still requires some correct answers to identify which rationales to train on.

This paper's approach sidesteps this tension entirely through a different mechanism. Rather than relying on the model's success rate, it relies on an external quality signal from the AI reward model that evaluates actions independently of trajectory outcomes. The paper doesn't need the initial prompted model to produce correct answers—it only needs the AI reward model to distinguish better from worse actions among the candidates generated at each step. This means the data generation process doesn't face the exploration deadlock that plagues RL-based approaches: even when the model produces entirely wrong final answers, the AI judge can still identify which intermediate search queries were well-formulated, which summarizations were accurate, and which decisions to terminate were reasonable.

The evidence for this is subtle but important. The paper's initial prompted PaLM 2-L achieves 70.3% on Bamboogle auto-eval (Table 1)—a non-trivial but far-from-perfect success rate. On the 2,000 training questions drawn from HotpotQA, Eli5, and related datasets, the success rate is presumably lower (since Bamboogle is a curated evaluation set, while the training questions are diverse and include many hard cases). Yet the 1st gen model improves significantly over the prompted baseline (74.0% vs. 70.3%), and the 2nd gen improves further (76.1%). This improvement trajectory doesn't depend on the model's initial answers being correct—it depends on the AI judge being able to identify good component actions regardless of whether the final trajectory succeeded.

This is a conceptual advance over the "difficulty estimation" approach common in other self-improvement work. In the ReST paper for math reasoning, the compute-optimal policy depends on estimating question difficulty to decide how to allocate inference compute. That estimation itself requires generating many samples and checking correctness—a chicken-and-egg problem. The current paper never needs to estimate which questions are easy or hard, never needs to filter trajectories by outcome, and never needs to allocate compute differently across questions. The training signal is uniformly available: for every step in every trajectory, the AI judge provides a relative ranking of candidate actions. This makes the self-improvement loop simpler and more general—it applies to any task where you can define what makes individual reasoning steps good, even if you can't define what makes a final answer correct.

Innovation 3: Self-Distillation as an Emergent Property of Iterative Data Improvement

The paper presents self-distillation not as a separate technique but as something that "naturally give[s] us a self-distillation algorithm" (Section 3.6) because "the synthetic data produced as part of this iterative process could be used for distilling the agent into one or two orders of magnitude smaller models" (Section 1). This framing is subtle but significant: distillation is not something you do; it's something that happens when you have an improving teacher generating increasingly higher-quality data.

To see why this is distinctive, contrast with standard knowledge distillation (Hinton et al., 2015). In standard distillation, you train a large teacher model on labeled data, then train a smaller student to mimic the teacher's output distribution—either on the original labeled data or on unlabeled data with the teacher's predictions as soft targets. The teacher is fixed; the student learns from it once. In the paper's approach, the teacher improves with each iteration because it's being fine-tuned on higher-quality data from the previous iteration. The distillation is continuous and bidirectional: the teacher generates data that trains the student, and the improved teacher (after fine-tuning on that same data) generates even better data for the next round of distillation.

The result in Table 1 makes this concrete. PaLM 2-XS achieves 44.7% on the pilot (human-filtered) data, 54.4% on 1st gen data, and 65.9% on 2nd gen data. Each of these jumps is not from training the XS model differently—it's trained the same way each time, with full fine-tuning. The improvement comes entirely from the quality of the training data, which tracks the teacher model's own improvement (prompted L → 1st gen fine-tuned L → 2nd gen fine-tuned L). The XS model at 2nd gen (65.9%) approaches the prompted L teacher (70.3%) with roughly two orders of magnitude fewer parameters.

What makes this an innovation rather than just "distillation works" is the quality-improvement feedback loop. The paper shows that better data (2nd gen vs. 1st gen, holding size constant) matters more than more data (Table 3: 9% gain from quality vs. 2.2% from doubling quantity). This means the distillation pipeline benefits from a compounding effect: each iteration of teacher improvement produces better training data, which produces a better student, and—if the student's outputs were used as the teacher for the next round—the cycle could continue. The paper doesn't close this loop (the teacher is always PaLM 2-L fine-tuned on the latest data), but the observed pattern—that data quality improves with each teacher iteration and that this quality improvement transfers to smaller models—establishes the principle.

This is conceptually different from the "scaling laws" perspective on distillation. Scaling laws typically ask: given a fixed teacher, how does student performance scale with student size and data quantity? This paper's results suggest a different question: given an improving teacher, how does data quality scale with teacher improvement, and how does that quality improvement translate across model sizes? The finding that "better data also reduces the variance of evaluation trajectories" (Section 5.3)—the standard deviation drops from ±3.6% to ±1.7%—suggests that higher-quality data produces not just more capable models, but more consistent ones, which is a different axis of improvement that standard distillation doesn't capture.

Innovation 4: A Negative Result on Human Filtering of Process-Based Data

The paper includes what might be its most counterintuitive finding: manually filtering "bad" examples from the training data reduces performance rather than improving it. The pilot experiment (Section 5.1) compared PaLM 2-XS trained on human-filtered data (30% of examples removed for issues like "unhelpful query, empty thoughts, summary missing important information") against the same model trained on unfiltered data. The filtered version achieved 44.7% vs. 47.2% for unfiltered—a 2.5 percentage point drop (Table 3, pilot columns).

This is not a minor ablation; it challenges a fundamental assumption in data-centric ML that cleaning your training data improves results. The paper's proposed explanation reveals something deep about process-based systems: "filtering only affects the immediate 'bad' example, not the whole trajectory; the 'bad' step would often be preserved in the other fine-tuning examples as part of the PAST_ACTIONS field" (Section 5.3). In other words, removing a bad search query from its own training example doesn't remove it from the context of subsequent steps—the next step's prompt still includes the bad query in its PAST_ACTIONS history. So the model still sees the problematic behavior, but now with fewer examples of how to handle that exact situation.

This finding has implications beyond this specific system. It suggests that for process-based training data—where each example's input includes the history of previous steps—the unit of data cleaning cannot be the individual step. You would need to either clean entire trajectories (removing or rewriting all downstream steps that reference the bad action) or accept that the model must learn to handle imperfect intermediate states because those states will appear at inference time regardless. The paper's conclusion—that the quality of the data matters more than its quantity (9% gain from quality improvement between generations vs. 2.2% from doubling data)—suggests that the right approach is not to manually filter, but to generate better data by improving the policy that produces it. This is the conceptual justification for the iterative self-improvement loop: rather than trying to clean bad data, you improve the data generator so that bad data becomes rarer naturally.

The reduction in evaluation variance that accompanies higher-quality data (Table 3: standard deviation drops from ±3.1% on pilot to ±1.7% on 2nd gen 1x) supports this interpretation. The model isn't just more accurate—it's more reliable, producing consistent outputs across different random seeds. This consistency is exactly what you'd expect if the training data contains fewer confusing or contradictory examples. The human filtering, by contrast, presumably introduced asymmetries (some bad actions removed, others preserved in context) that made the training distribution less coherent, not more. </output>

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation uses the Bamboogle dataset (Press et al., 2023), consisting of 125 semi-adversarial 2-hop questions manually verified to be unanswerable by direct Google search but where both required pieces of evidence exist in Wikipedia. A complementary held-out test set, BamTwoogle, was constructed by the authors with 100 questions that are "slightly more challenging" and address Bamboogle shortcomings (ensuring all questions require 2+ steps, verifying answers don't appear on the first page of Google search results). For training data generation, 2,000 seed questions are drawn from the training splits of four datasets: HotpotQA (Yang et al., 2018) — 500 questions, Eli5 (Fan et al., 2019) — 500 questions, Eli5-askH (Blagojevic, 2022) — 500 questions, and Eli5-askS (Blagojevic, 2022) — 500 questions. No labels or other information from these training datasets are used, only the questions themselves as trajectory seeds.

• Base model(s). All experiments use the PaLM 2 family (Anil et al., 2023) — specifically the "base" (pre-trained, not instruction-tuned) variants at three scales: PaLM 2-XS, PaLM 2-S, and PaLM 2-L. The prompted teacher model that generates initial training data is PaLM 2-L, chosen because "only the PaLM 2-L can perform [the code-format prompting] consistently well" out of the models tried (Section 3.1). The instruction-tuned PaLM 2-L variant serves as the AI reward model for action re-ranking. The paper argues that PaLM 2 models are "representative of the capabilities of many contemporary LLMs" and sit in a useful performance regime — competent enough to produce working trajectories via prompting, but with substantial room for improvement through fine-tuning.

• Metrics. The primary performance metric is auto-eval accuracy (%) on Bamboogle, computed by generating 10 independent trajectories per question (at temperature T = 0.5), running the auto-eval judge (a prompted PaLM 2-L) on each trajectory's final answer against the ground-truth reference answer, and averaging the binary correct/incorrect judgments across all 125 questions × 10 repetitions = 1,250 evaluations. The auto-eval judge determines whether the model's long-form answer "implies" the reference answer (a logical implication criterion rather than exact string match). This is validated against human judgments, achieving Pearson correlation of 0.98 (p = 6.6 × 10⁻⁸) and Spearman correlation of 0.83 (p = 0.0015) on a "diverse set of agents" (Section 4.2). For the final models, human evaluation is also reported on both Bamboogle and BamTwoogle (Table 2) as a single-run verification (one trajectory per question, human-judged). The paper reports mean accuracy with standard deviation across the 10 auto-eval runs.

• Baselines. The paper uses several baselines:

  • Pre-trained (prompted) PaLM 2-L: The initial model with only few-shot prompts and no fine-tuning — this is the "teacher" whose performance the distilled models aim to match or approach (Section 5.2, Table 1).
  • Pilot, human-filtered: Models fine-tuned on 500 trajectories that were manually reviewed, with approximately 30% of examples filtered out for issues like "unhelpful query, empty thoughts, summary missing important information" (Section 5.1, Table 1).
  • Pilot, unfiltered (implicit): The same 500 trajectories without human filtering, producing 4,518 training examples vs. 3,015 for the filtered version (Section 5.3, Table 3).
  • 1st gen self-improvement: Models fine-tuned on 2,000 trajectories generated by the prompted PaLM 2-L, with RM re-ranking applied during data preparation (Section 5.2, Table 1).
  • Majority voting and verifier-based baselines from prior work are NOT used — the paper's comparison is entirely across model sizes and training data iterations within its own pipeline, not against external agent architectures. This is a notable omission; there is no comparison to a ReAct agent without self-improvement, a FireAct-style fine-tuned agent, or any other published agent system on these benchmarks.

• Generation budget / compute accounting. The paper does not use a formal compute budget metric (no FLOPs accounting or token counting as in scaling laws papers). Instead, data quantity is measured in three ways:

  • Number of trajectories collected during each "Grow" stage: 500 for pilot, 2,000 for 1st gen, and 8,000 for 2nd gen (2,000 questions × 4 repetitions each).
  • Number of fine-tuning examples produced from trajectory decomposition: ranging from 3,015 (pilot, filtered) to 72,424 (2nd gen, 4× per question) as shown in Table 3.
  • Model size as a proxy for inference and training cost: PaLM 2-XS, S, and L, with the paper noting that "fine-tuning costs increase sharply for larger models" (Section 3.4) and therefore doing "as many experiments as possible with XS model."
  • For trajectory generation, multi-sample selection uses 4 samples per step for L and S models, 16 for XS (Section 3.2), and the RM re-ranking evaluates all candidates off-policy during data preparation rather than during trajectory rollout. No wall-clock time or FLOPs budgets are reported.

• Cross-validation / statistical protocol. The paper does not use standard cross-validation (no train/validation/test split of labeled data, since no labeled data exists in the pipeline). Instead:

  • Bamboogle serves as a development/validation set: used for hyperparameter selection (temperature, checkpoint selection, iteration decisions) but explicitly "never used as a training set" — the paper states "we neither tune our prompts on Bamboogle nor use questions from it to generate fine-tuning trajectories" (Section 4.2).
  • BamTwoogle serves as a held-out test set: "used exclusively to measure the final performance of the models" (Section 4.3) and only evaluated with human judgments (Table 2), not used for any model selection decisions.
  • Auto-eval over 10 repetitions provides the statistical power to detect differences: each model's reported accuracy is a mean over 10 independent trajectory runs per question, with standard deviations reported in Table 1 and Table 3. This accounts for the stochasticity from temperature-based sampling.
  • Human evaluation on a single run (Table 2) provides a one-shot verification that auto-eval trends hold under human judgment, but lacks the repetition-based variance reduction of auto-eval and is therefore noisier.

Main Quantitative Results

Self-Improvement Improves Performance Across All Model Sizes

The headline result (Table 1) shows that iterative self-improvement on synthetic data consistently improves Bamboogle auto-eval accuracy across all three model sizes, with the largest relative gains for the smallest model:

For PaLM 2-L: The prompted baseline achieves 70.3%. After fine-tuning on 1st gen data (generated by the prompted model itself), accuracy rises to 74.0% — a 3.7 percentage point absolute improvement. After the 2nd iteration (using the 1st gen fine-tuned model to generate higher-quality trajectories), accuracy reaches 76.1% — an additional 2.1 point gain, for a total improvement of 5.8 points over the prompted baseline. Notably, the standard deviation narrows from ±3.5% (prompted) to ±1.3% (2nd gen), indicating more consistent behavior.

For PaLM 2-S: The improvement is more dramatic. From 56.6% (pilot filtered) to 61.9% (1st gen) to 69.7% (2nd gen) — a 13.1 percentage point gain from the pilot baseline to the 2nd generation. The standard deviation also tightens substantially: ±3.8% → ±1.9% → ±1.3%.

For PaLM 2-XS: The smallest model shows the largest relative gains. Starting from 44.7% on pilot filtered data, 1st gen reaches 54.4% (+9.7 points), and 2nd gen reaches 65.9% (+11.5 points over 1st gen, +21.2 points over pilot). The 2nd gen XS model at 65.9% is within 4.4 percentage points of the prompted PaLM 2-L teacher (70.3%) — a model approximately two orders of magnitude larger. This is the core distillation result: the synthetic data from the self-improvement loop enables a small model to approach the large teacher's prompted performance.

The standard deviation pattern is consistent across all model sizes: later-generation data produces not just higher accuracy but also lower variance. For XS: ±3.1% (pilot) → ±3.6% (1st gen) → ±2.6% (2nd gen). The 2nd gen XS model is more reliable than the 1st gen L model (±2.6% vs. ±3.3%).

Human Evaluation Confirms Auto-Eval Trends (Table 2)

Human evaluation on a single run of Bamboogle and BamTwoogle (Table 2) validates the auto-eval findings:

On Bamboogle: The 2nd gen L reaches 74.4% (compared to 76.1% auto-eval), closely tracking the auto-eval estimate. The 2nd gen XS at 67.2% is within 1.6 points of the prompted L at 68.8% under human evaluation — even closer than the 4.4-point gap in auto-eval (65.9% vs. 70.3%). This suggests auto-eval may be slightly conservative in estimating the small model's relative performance.

On BamTwoogle (the held-out test set): The pattern holds. 2nd gen L achieves 74.0%, matching its Bamboogle human score closely. The 2nd gen XS drops to 63.0% — a larger gap from the prompted L (68.0%) than on Bamboogle, suggesting that the more challenging questions in BamTwoogle expose some remaining capability gap that Bamboogle doesn't fully capture. Both XS and S models score identically on BamTwoogle (63.0%), indicating that on the harder test set, the S model's advantage over XS seen on Bamboogle (69.7% vs. 65.9% auto-eval) does not materialize — possibly a ceiling effect from the single-run human evaluation's higher variance, or a genuine indication that the harder questions require capabilities that even S doesn't fully capture from the distilled data.

Self-Critique Provides a Small but Consistent Benefit (Figure 4, Appendix Table 4)

Figure 4 compares model performance with and without the two self-critique steps (relevance check and grounding check). Across all model sizes and training data generations, self-critique provides a small positive boost. Appendix Table 4 provides the detailed numbers:

For the answer generation step (before self-critique):

Comparing these "before self-critique" numbers with the final answer numbers in Table 1 (which include self-critique), the benefit ranges from approximately +0.3 to +1.1 percentage points depending on model size and generation:

• For PaLM 2-L: The benefit is largest, ranging from +0.6 (pilot) to +1.1 (2nd gen).
• For PaLM 2-S: The benefit is +2.2 (pilot) — an outlier that may reflect noise — to +0.5 (2nd gen).
• For PaLM 2-XS: The benefit is small but positive in most cases, ranging from -0.4 (pilot, a slight negative) to +0.4 (1st gen) to +0.3 (2nd gen).

The 1st gen PaLM 2-XS is the only model to show a slight improvement from self-critique reversal (+0.4% — the before-critique score is actually HIGHER than the after-critique score for this model). The paper notes that the self-critique benefit "depends on the model size (larger for larger models) but does not seem to be affected by the self-improvement process" (Section 6). This is consistent with larger models being better at the meta-cognitive task of checking their own work — the self-critique steps require evaluating whether the answer is relevant and grounded, which may require capabilities that don't fully transfer through distillation to smaller models.

Data Quality Dominates Data Quantity (Table 3)

Table 3 presents a controlled comparison of how data characteristics affect PaLM 2-XS performance:

The comparison that isolates data quality from data quantity is between 1st gen and 2nd gen (1×). Both have approximately 18,000 training examples. The 2nd gen data was generated by the 1st gen fine-tuned model (higher quality trajectories) while the 1st gen data was generated by the prompted model. The performance gap is 63.4% vs. 54.4% — a 9.0 percentage point improvement from data quality alone, holding dataset size essentially constant. This is the paper's key finding on data efficiency: better data generation policy matters more than having more data from a worse policy.

The comparison that isolates data quantity within the same data quality is between 2nd gen (1×), (2×), and (4×). Doubling from 18,007 to 36,238 examples improves accuracy from 63.4% to 65.6% (+2.2 points). Doubling again to 72,424 examples yields only 65.9% (+0.3 points). The diminishing returns are stark — quadrupling the data from 1× to 4× produces only a 2.5 point gain, while improving data quality between generations (1st to 2nd, same size) produces a 9.0 point gain. This justifies the paper's iterative approach: rather than scaling up data collection from a fixed policy, invest compute in improving the policy that generates the data.

The standard deviation pattern reinforces this conclusion. The 2nd gen data at any multiplicity has substantially lower variance (±1.7% to ±2.6%) than the 1st gen data (±3.6%) or pilot data (±3.1%). Higher-quality training data produces not just more accurate but more consistent models — the performance is more predictable across different random seeds and trajectory rollouts.

Human Filtering of Training Data Reduces Performance (Table 3, Pilot Columns)

Comparing pilot filtered (44.7%) vs. pilot unfiltered (47.2%) for PaLM 2-XS reveals that manually removing approximately 30% of "bad" examples (reducing the dataset from 4,518 to 3,015 training examples) decreases performance by 2.5 percentage points. This is a genuine negative result that challenges the intuitive assumption that cleaning training data improves model quality.

The paper proposes two explanations (Section 5.3): (1) the reduced dataset size makes it harder for the model to learn proper prompt formatting, and (2) more interestingly, filtering only removes the direct training example for a bad action, but that same bad action persists in the PAST_ACTIONS context of subsequent steps in the same trajectory — so the model still sees the problematic behavior in the input of other training examples, but with fewer examples of how to handle or recover from it.

This finding has implications beyond this paper: for process-based training data where each example's input includes the history of previous actions, per-example filtering is insufficient — you would need to either filter (or rewrite) entire trajectories or accept that the model must learn from imperfect intermediate states.

Ablation Studies and Robustness Checks

Multiple trajectories per question: Increasing from 1 to 2 trajectories per question in the 2nd gen data provides a 2.2 percentage point gain (63.4% → 65.6%), but further increasing to 4 trajectories yields negligible additional improvement (65.9%, +0.3). This is shown in Table 3, 2nd gen columns. The implication is that two diverse trajectories per question capture most of the benefit of exploration, and additional trajectories are largely redundant. This is a cost-efficiency finding: generating 2× trajectories costs twice the compute of 1×, but 4× costs twice again with minimal return.

Model size and checkpoint selection: The paper reports that optimal checkpoint selection varies by model size: "step 9K for XS, 5K for S, 3.5K for L" (Section 4.2), determined by Bamboogle auto-eval. The decreasing number of steps with increasing model size is consistent with larger models converging faster (requiring fewer gradient updates to fit the same data), though the paper does not discuss learning rate schedules, batch sizes, or other optimization details that would contextualize these numbers.

Self-critique impact by model size (Figure 4, Appendix Table 4): The self-critique benefit is largest for PaLM 2-L (up to +1.1 percentage points for 2nd gen), moderate for S (up to +0.5), and negligible-to-absent for XS (+0.3 for 2nd gen, actually slightly negative for pilot at -0.4). This pattern — larger models benefiting more from self-critique — is consistent with self-critique being a meta-cognitive capability that requires sufficient model capacity. The paper notes this "does not seem to be affected by the self-improvement process" (Section 6): the self-critique benefit is stable across training data generations for each model size, suggesting it's a property of the model architecture rather than something learned during fine-tuning.

Temperature selection: The paper reports that T = 0.5 was selected as optimal based on auto-eval (Section 4.2), but provides no sweep data or comparison to alternative temperatures. This is a practical hyperparameter choice rather than a studied ablation — we cannot assess how sensitive results are to temperature from the reported data.

Absence of the reward model: The paper does not include an ablation comparing RM-re-ranked training data against data prepared with only perplexity-based selection. This is a significant missing ablation — we cannot determine how much of the self-improvement gain comes from the RM re-ranking specifically versus from simply having more trajectories (the 1st gen data is both re-ranked and larger than the pilot data). The comparison between 1st gen (RM-re-ranked, 17,970 examples) and pilot unfiltered (no RM, 4,518 examples) confounds RM quality with dataset size and diversity.

Absence of a "no self-improvement" baseline for distillation: The paper trains small models on improving data but does not report what happens if you train a small model directly on trajectories from the prompted teacher without any RM re-ranking. This would isolate the benefit of the RM from the benefit of simply having fine-tuning data at all. The closest comparison is pilot unfiltered XS (47.2%) vs. 1st gen XS (54.4%), but these differ in both dataset size (4,518 vs. 17,970) and RM use.

Absence of a "distillation from fixed teacher" baseline: The paper shows that XS trained on 2nd gen data (65.9%) approaches prompted L (70.3%), but does not compare against standard knowledge distillation — training XS to directly mimic the prompted L's output distribution on the same 2,000 questions. This would distinguish whether the synthetic trajectory data is more effective than standard distillation logits for transferring capabilities to smaller models.

Critical Assessment

Claim 1: "The performance of the agent could be effectively improved through ReST-style iterative fine-tuning on its reasoning traces."

The experiments genuinely support this claim with specific evidence. Table 1 shows consistent improvement across all model sizes: prompted L (70.3%) → 1st gen L (74.0%) → 2nd gen L (76.1%). This is a 5.8 percentage point absolute gain from two iterations of self-improvement. The improvement is not an artifact of auto-eval — human evaluation (Table 2) confirms the pattern, with 2nd gen L at 74.4% vs. prompted L at 68.8% on Bamboogle, and 74.0% vs. 68.0% on BamTwoogle.

However, what is demonstrated is narrower than "ReST-style iterative fine-tuning works." Specifically:

• The improvement is measured over only two iterations (prompted → 1st gen → 2nd gen). We do not know whether additional iterations would continue to improve, plateau, or degrade. The paper acknowledges this as a limitation: "How many additional iterations of self-improvement can we undertake past the 2nd one that still results in non-trivial gains?" (Section 6).

• The improvement is on a single task (long-form question answering with web search) with a single agent architecture (the specific five-phase state machine) and a single model family (PaLM 2). The claim's generality to other agent architectures, tasks, or model families is not tested.

• The prompted baseline already performs well (70.3% on Bamboogle). We don't know whether self-improvement works when starting from a weaker teacher — the paper does not test whether self-improvement can bootstrap from a model that initially fails to produce coherent trajectories. If the initial prompted model produced gibberish or empty outputs, the trajectory data would be unusable and the AI judge would have nothing meaningful to rank.

• The improvement mechanism is confounded: the 1st gen improvement could come from (a) fine-tuning on any trajectory data (getting better at the format and task), (b) the RM re-ranking providing cleaner targets, (c) the increased dataset size (2,000 trajectories vs. 500 pilot), or (d) some combination. The paper does not ablate these factors independently.

Claim 2: The improvement works "purely from stepwise AI feedback without using human-labeled training data."

This claim is supported in the sense that no human labels are used for training. The only human involvement in the entire pipeline is: (1) writing the few-shot prompts (a one-time cost), (2) constructing the Bamboogle and BamTwoogle datasets for evaluation, and (3) manually filtering the pilot data (which was shown to be counterproductive). The training data — trajectories and their re-ranked actions — is entirely synthetic.

However, the claim elides an important dependency: the AI feedback comes from a model (instruction-tuned PaLM 2-L) that was itself trained with human feedback (instruction tuning typically involves human demonstrations and/or human preferences). The zero-shot RM is not "trained on human preferences" for this specific task, but the underlying model's ability to rank actions as "good" or "bad" is derived from its instruction tuning, which almost certainly involved human supervision. This doesn't invalidate the claim — the method indeed doesn't use task-specific human labels — but it means the system is not entirely independent of human supervision. The AI judge's quality judgments inherit from whatever human values and preferences were baked into its instruction tuning.

A stronger version of this claim would require using a base model (not instruction-tuned) as the RM, or showing that the self-improvement works even with a simple heuristic RM that doesn't rely on human-trained capabilities. The paper does not explore this.

Claim 3: "Synthetic data produced as part of this iterative process could be used for distilling the agent into one or two orders of magnitude smaller models with performance comparable to the pre-trained teacher agent."

This is the strongest and best-supported claim in the paper. The evidence is in Table 1: 2nd gen PaLM 2-XS at 65.9% vs. prompted PaLM 2-L at 70.3% — a 4.4 percentage point gap with approximately two orders of magnitude fewer parameters. Under human evaluation (Table 2), the gap narrows further: 67.2% for XS vs. 68.8% for prompted L on Bamboogle (1.6 point gap).

However, "comparable" is doing work here. The 2nd gen XS model is still below the prompted L, and substantially below the 2nd gen L (65.9% vs. 76.1% auto-eval — a 10.2 point gap). On the harder BamTwoogle test set, the gap between 2nd gen XS and prompted L is larger: 63.0% vs. 68.0% (5 point gap under human evaluation). The distillation is effective but not lossless — there is a genuine capability gap that widens on more challenging questions.

The paper also does not compare against the obvious alternative: directly distilling the prompted L model's outputs (standard knowledge distillation) without the iterative self-improvement loop. Would XS trained directly on prompted L's trajectories (with RM re-ranking) achieve similar performance to XS trained on 2nd gen data? If so, the iterative self-improvement is not necessary for distillation — a single round of trajectory collection from the prompted teacher would suffice. The paper cannot answer this question from the reported experiments because the comparison between 1st gen XS (54.4%) and 2nd gen XS (65.9%) confounds data quality with other factors like dataset size and RM re-ranking.

Claim 4 (implicit): The self-critique steps (Reflexion) improve agent performance.

The paper states that self-critique provides "a small but measurable positive boost (on the order of 0.5-1.0% for most models)" (Section 5.3). The data in Appendix Table 4 supports this for PaLM 2-L (0.6 to 1.1 point benefit) and PaLM 2-S (0.5 to 2.2 points, though the 2.2 for pilot is an outlier). For PaLM 2-XS, the evidence is weaker — the benefit ranges from -0.4 (pilot, a slight negative) to +0.4 (1st gen) to +0.3 (2nd gen). The claim of a positive boost holds for larger models but is ambiguous for the smallest model. Additionally, the paper does not report whether these differences are statistically significant given the reported standard deviations (e.g., for XS 2nd gen: 65.6 ± 3.0% before vs. 65.9 ± 2.6% after — a 0.3 point difference with overlapping confidence intervals).

Overall strengths of the experimental design:

• The auto-eval validation (Pearson 0.98 correlation with humans) provides credible evidence that the automated metric tracks human judgment, enabling the high-repetition evaluation that gives the results statistical power despite small test sets.
• The BamTwoogle test set serves as a genuine held-out evaluation, addressing overfitting concerns from using Bamboogle for model selection. The pattern of results holds on BamTwoogle (Table 2), which strengthens the findings.
• The multiple model sizes (XS, S, L) enable studying how self-improvement and distillation interact with scale — the finding that relative gains are largest for the smallest model is practically important.
• The human filtering negative result (Table 3) demonstrates intellectual honesty in reporting a counterintuitive finding and provides genuine insight about process-based data.

Significant weaknesses and missing experiments:

• Small evaluation sets: Bamboogle has 125 questions, BamTwoogle has 100. Split across model sizes and training data variants, the differences being measured are on the order of a few percentage points, which corresponds to a handful of questions. A difference of 2-3% on a 125-question test set is approximately 2-4 questions — well within the noise floor given trajectory stochasticity. The 10-repetition auto-eval partially mitigates this by expanding to 1,250 evaluations per model, but the underlying question diversity is still limited.

• Single agent architecture, single task, single model family: All experiments use the same five-phase state machine, the same long-form QA task, and PaLM 2 models. The paper makes no claim of generality, but it also provides no evidence that the approach transfers to other agent designs, other tasks (code generation, tool use beyond search, multi-agent coordination), or other model families.

• Missing baselines: There is no comparison to (a) a ReAct agent without self-improvement (prompted only), (b) FireAct-style fine-tuning with human labels, (c) standard knowledge distillation from the prompted teacher without iterative improvement, (d) training directly on human-labeled data for the same task (which would establish an upper bound on what synthetic data can achieve), or (e) simply prompting the instruction-tuned model directly as the agent rather than using it as the RM.

• No RM ablation: We cannot determine how much the AI reward model contributes vs. simply having more trajectories. The 1st gen data uses RM re-ranking and is larger than pilot — these factors are confounded. An ablation training on 1st gen data without RM re-ranking (using only perplexity selection) would isolate the RM's contribution.

• No cost analysis: The paper doesn't report the computational cost of generating trajectories, running RM re-ranking, or fine-tuning at different scales. The claim that 2nd gen XS "achieves comparable performance with two orders of magnitude fewer parameters" needs to be contextualized with inference and training cost. If generating the 8,000 2nd gen trajectories costs 100× more than what was saved by using a smaller model, the distillation benefit is purely about deployment cost, not total cost.

• Limited iteration depth: Only two iterations are tested. We don't know whether the approach saturates (does a 3rd iteration help? does it hurt?), whether different model sizes saturate at different rates, or whether there is a point where self-improvement begins to degrade due to amplification of the RM's biases or the policy collapsing to a narrow distribution.

• No negative controls for the RM: The paper doesn't test whether a deliberately bad RM (e.g., a random ranking, or an RM that always prefers shorter outputs) would still produce improvements via the self-training loop, which would indicate that the gains come from the training procedure rather than the quality of the AI feedback.

6. Limitations and Trade-offs

Limitation 1: The Difficulty Estimation and Trajectory Generation Cost Is Unaccounted For in the Headline Efficiency Claims

The assumption or constraint. The paper's central efficiency claim is that self-improvement with AI feedback enables a small model (PaLM 2-XS) to approach the prompted large teacher's performance "with two orders of magnitude fewer parameters" (Section 1). However, this comparison counts only the final model's parameter count during inference, not the computational cost of producing the training data that enabled the distillation. The data generation pipeline is extraordinarily expensive: the 2nd generation training data required generating 8,000 complete trajectories using PaLM 2-L — each trajectory involving multiple search API calls, up to 10 search steps, multi-sample generation (4 samples per step) at every reasoning phase, and finally RM re-ranking of all candidate actions using a separate instruction-tuned PaLM 2-L call per step. The paper acknowledges the computational burden implicitly in the Discussion: running auto-eval "increases the computational costs significantly due to the need to run agent trajectories multiple times, as well as the use of PaLM 2-L model for auto-eval" (Section 6), but never quantifies the total FLOPs or wall-clock time of the data generation phase relative to the distillation savings.

The consequence. A practitioner deciding whether to deploy this method needs to know: does the cost of generating high-quality synthetic trajectories outweigh the savings from running a smaller model at inference time? The paper provides no way to answer this question. If generating 8,000 trajectories with PaLM 2-L costs 100× more than the inference savings from switching to PaLM 2-XS, the approach is a net loss in total compute despite the parameter-count reduction. This is analogous to the pretraining-vs-inference tradeoff analyzed in compute-optimal scaling papers (e.g., Hoffmann et al., 2022), but the paper performs no such analysis. The "two orders of magnitude" reduction in parameters is a deployment-time efficiency gain, not a total-cost efficiency gain. For applications where the model will serve billions of queries, the upfront training cost amortizes favorably; for applications with modest inference volume, the data generation cost could dominate. The paper does not characterize this crossover point.

What evidence exists in the paper. The paper reports data quantities in Table 3 (500 to 8,000 trajectories, 3,015 to 72,424 training examples) and notes that trajectory generation uses 4 samples per step for L and S models, 16 for XS (Section 3.2), with each sample being a PaLM 2-L inference. The search tool returns top-3 snippets per query (Section 3.2), and trajectories can involve up to 10 search steps (Section 3.2), each triggering an external API call. The RM re-ranking for the 2nd generation data processes all candidate actions across 8,000 trajectories — likely hundreds of thousands of individual RM calls. None of these costs are aggregated or compared to the inference cost of the distilled model. The auto-eval itself requires 1,250 PaLM 2-L calls per model evaluation (125 questions × 10 repetitions), though the paper correctly excludes this from the training cost since it's an evaluation expense.

Mitigation status. Not addressed. The paper does not report any cost accounting, does not propose a FLOPs-matched comparison framework, and does not discuss the data-generation-to-inference-savings tradeoff. The Section 6 discussion of computational costs focuses only on auto-eval, not on the data generation pipeline. A practitioner cannot determine from the paper whether the total cost (data generation + training + inference) favors the distilled small model or simply using the prompted large model directly. This is a significant barrier to adopting the method in resource-constrained settings.

Limitation 2: The Method Has Only Been Demonstrated on a Single Task with a Single Agent Architecture and a Single Model Family

The assumption or constraint. All experiments use the same Search Agent architecture (the specific five-phase state machine with search loop, answer generation, and two self-critique steps), the same task (long-form question answering with web search), and models from a single family (PaLM 2 at XS, S, and L scales). The paper acknowledges this scope limitation only indirectly: the future work section asks "if the same self-improvement algorithm applies to multiple tool settings and, especially, if the ability to handle unseen tools could be improved in such a way" (Section 6), implying the current work does not test these generalizations. The paper makes no claim of task- or architecture-generality, but it also provides zero evidence about transferability.

The consequence. The core mechanism — AI feedback on individual reasoning steps within a process-based agent — depends on several design choices whose necessity is unknown:

• The code-format prompts (Section 3.1) were chosen because they enable reliable parsing of structured outputs, but this design constrains which models can serve as the teacher ("only PaLM 2-L can perform it consistently well" out of the models tried). Would the self-improvement loop work with natural language prompts, enabling a wider range of base models? The paper doesn't test this.
• The specific five-phase state machine (search decision → summarization → answer → relevance check → grounding check) provides natural decomposition points for state-wise fine-tuning. Would the approach work with a less structured agent — one where reasoning steps are not cleanly separable into named phases? The paper's claim that "the model can learn something useful even from the states that eventually lead to the wrong final answer" (Section 6) depends on the ability to identify and extract good individual actions from failed trajectories. If the agent's actions are not cleanly typed and separable, this decomposition becomes harder or impossible.
• The PaLM 2 model family may have specific properties (code-generation capability, instruction-following, calibration of perplexity scores) that the self-improvement loop relies on. The AI reward model is an instruction-tuned PaLM 2-L — if a different model family's instruction-tuned variant produces lower-quality rankings, the self-improvement signal degrades. The paper provides no evidence about RM quality sensitivity.
• The long-form QA task has specific properties that may enable the approach: answers must be attributed to sources (giving the grounding check a clear criterion), questions are open-ended enough that process-based feedback (rather than outcome-based) is genuinely necessary, and the search tool provides clean input-output interfaces. A task like code generation (where the "tool" might be a code interpreter with complex state) or multi-agent coordination (where actions depend on other agents' behaviors) might not decompose as cleanly.

What evidence exists in the paper. None beyond the single-task, single-architecture, single-family results. The paper's evaluation is entirely on Bamboogle and BamTwoogle (125 and 100 question-answering problems). The training data comes from four QA datasets (HotpotQA, Eli5, Eli5-askH, Eli5-askS), but the evaluation never tests the agent on tasks outside this distribution — no code generation, no mathematical reasoning, no dialogue, no multi-tool orchestration. The paper does not ablate the architecture (e.g., removing self-critique steps and measuring whether self-improvement still works, or testing with a simpler decision loop), so we cannot distinguish whether the self-improvement benefit comes from the ReST-like algorithm or from the specific agent design.

Mitigation status. The paper explicitly identifies this as a direction for future work: "Future work could explore if the same self-improvement algorithm applies to multiple tool settings" (Section 6). However, this is stated as an open question, not as a limitation the authors have begun to address. A single-task, single-architecture demonstration is appropriate for a first paper establishing the feasibility of an approach, but it means the method's generality is entirely unproven.

Limitation 3: The Improvement Is Demonstrated Over Only Two Iterations, with No Characterization of the Saturation Point or Potential Degradation

The assumption or constraint. The paper's self-improvement loop runs for exactly two iterations: prompted PaLM 2-L generates 1st generation data, the 1st gen fine-tuned model generates 2nd generation data, and results are reported at each stage. The paper explicitly asks but does not answer: "How many additional iterations of self-improvement can we undertake past the 2nd one that still results in non-trivial gains? What does the saturation look like for smaller models? Will they all eventually converge to the same performance, or will the smaller models always be capped by the performance of the initial prompted large model?" (Section 6).

The consequence. Without knowing the saturation behavior, a practitioner cannot decide how many iterations to run. Three critical unknowns remain:

• Does improvement continue? If a 3rd iteration produces another 2-point gain, the approach is more valuable than the 2-iteration results suggest. If it plateaus or reverses, 2 iterations is near-optimal and the method has limited headroom.
• Does the policy collapse? In iterated self-training, there is a well-documented risk of "policy collapse" or "mode collapse" where the model's output distribution narrows with each generation, losing diversity and eventually producing degenerate outputs. This happens when the reward model's preferences amplify certain patterns and the policy overfits to those patterns, producing data that scores well under the RM but is actually worse. The paper's RM is itself an LLM with its own biases — if it consistently prefers shorter answers, answers that cite many sources, or answers that express high confidence, these preferences could be amplified across iterations. The ReST$^{EM}$ experiment in the companion paper (Appendix K) showed exactly this failure mode with revision models, where additional sequential revisions "substantially hurt" performance. The current paper has not tested whether a 3rd or 4th iteration would trigger similar degradation.
• Do different model sizes saturate at different points? The paper shows XS benefits more from data quality improvement than L (XS gains 21.2 points from pilot to 2nd gen, L gains 4.6 points). This pattern might continue — XS might catch up further with additional iterations — or XS might hit a ceiling imposed by its capacity while L continues to improve. The paper's question about whether smaller models are "always capped by the performance of the initial prompted large model" (Section 6) is exactly the right question but remains unanswered.

What evidence exists in the paper. The only evidence about the iteration trajectory comes from the two data points (prompted → 1st gen → 2nd gen) in Table 1. For PaLM 2-L: 70.3% → 74.0% → 76.1%. The gain from 1st to 2nd iteration (2.1 points) is smaller than the gain from prompted to 1st (3.7 points) — this could indicate diminishing returns that would lead to saturation within 1-2 more iterations, but with only two points, we cannot distinguish diminishing returns from noise. For XS, the pattern is different: 44.7% (pilot) → 54.4% (1st gen) → 65.9% (2nd gen). The second gain (11.5 points) is actually larger than the first (9.7 points) — this suggests XS is not yet saturating and might benefit substantially from additional iterations. But these are only two transitions, and the pilot-to-1st comparison for XS confounds dataset size (500 vs. 2,000 trajectories) with iteration quality, so the apparent acceleration may be an artifact.

Mitigation status. Not addressed experimentally. The paper identifies the question as future work (Section 6) but provides no empirical characterization of the saturation curve. Running even one more iteration (to produce 3rd gen models) would have substantially strengthened the paper by providing a third data point and revealing whether the trend is converging, continuing linearly, or reversing. The computational cost of a 3rd iteration is the same as the 2nd, so this is a missed opportunity rather than an infeasible experiment.

Limitation 4: The Reward Model's Contribution Is Not Isolated from Other Confounding Factors

The assumption or constraint. The paper's central methodological claim is that "stepwise AI feedback" (Section 1) — the zero-shot LLM-based ranking of candidate actions — enables self-improvement without human labels. However, the experiments that demonstrate improvement confound the RM's contribution with at least two other factors: (1) the increased size of the training dataset when moving from pilot to 1st generation, and (2) the benefit of fine-tuning on any trajectory data (even without RM re-ranking), which teaches the model the task format and basic agent behavior.

The consequence. We cannot determine from the reported experiments whether the AI feedback is actually necessary for the observed improvements, or whether similar gains could be achieved by simply:

• Fine-tuning on a larger quantity of trajectories with only perplexity-based selection (no RM)
• Fine-tuning on the same trajectories with a simple heuristic ranking (e.g., prefer shorter search queries, prefer answers with more citations)
• Using the prompted teacher's outputs directly for standard knowledge distillation without any iterative loop

This matters for two reasons. First, if the RM is unnecessary, practitioners can avoid the substantial cost of running a separate PaLM 2-L inference for every candidate action during data preparation. Second, if the RM is genuinely contributing, we need to understand how sensitive the approach is to RM quality — would a weaker RM (e.g., a smaller instruction-tuned model) still work? Would a stronger RM (e.g., an ensemble) produce substantially better results? Without isolating the RM's effect, the paper establishes correlation (self-improvement works when RM re-ranking is used) but not causation (RM re-ranking causes the improvement).

What evidence exists in the paper. The closest comparison that isolates dataset size is between pilot unfiltered (500 trajectories, 4,518 examples, no RM) and 1st generation (2,000 trajectories, 17,970 examples, with RM) in Table 3, showing 47.2% vs. 54.4% for PaLM 2-XS. But these differ in trajectory count, example count, trajectory quality (the 1st gen trajectories are from the same model but cover 4× more diverse questions), and RM use — all simultaneously. The comparison that isolates data quality is between 1st gen (17,970 examples, with RM) and 2nd gen 1× (18,007 examples, with RM) — but both use the RM, so this comparison tells us about the policy improvement, not about the RM's contribution.

The paper never reports: (a) the performance of a model trained on the same trajectories but using only perplexity selection (no RM re-ranking), (b) an ablation where the RM is replaced with a random ranking or a simple heuristic, or (c) the agreement rate between perplexity selection and RM selection (if they usually choose the same candidate, the RM adds little). These are the standard ablations needed to establish that the RM specifically — rather than the overall data pipeline — is driving the improvement.

Mitigation status. Not addressed. The paper treats the RM as an integral part of the ReST adaptation without ablating its contribution. The Discussion (Section 6) credits the RM as part of the approach ("combining a process-based approach... with high-temperature exploration, AI feedback (zero-shot 'reward' model used for actions re-ranking), and state-wise fine-tuning"), but never isolates which of these components is necessary versus merely present. This is the most significant missing ablation in the paper, and it substantially weakens the claim that AI feedback specifically — as opposed to self-training in general — is the key enabler.

Limitation 5: The Evaluation Sets Are Small, Domain-Specific, and Potentially Correlated with Training Data

The assumption or constraint. The paper's primary evaluation uses Bamboogle (125 questions) as a development set and BamTwoogle (100 questions) as a held-out test set. The training data is drawn from four QA datasets (HotpotQA, Eli5, Eli5-askH, Eli5-askS), with 500 questions from each, totaling 2,000 seed questions. The paper explicitly states that Bamboogle is "never used as a training set" and that questions from it are not used "to generate fine-tuning trajectories" (Section 4.2), and that BamTwoogle is used "exclusively to measure the final performance" (Section 4.3). However, the paper does not analyze the distributional relationship between the training seed questions and the evaluation questions.

The consequence. Three distinct concerns arise:

First, small absolute numbers: A 2-3 percentage point difference on a 125-question test set corresponds to approximately 2-4 questions changing from incorrect to correct (or vice versa). Given that each model is evaluated with 10 stochastic trajectory runs per question (producing 1,250 total evaluations), the statistical power comes from averaging over runs, not from question diversity. This means that if a few questions in Bamboogle happen to be particularly easy or hard for the specific agent architecture, they can substantially influence the reported accuracy. The paper reports standard deviations across the 10 runs (typically ±1.3-3.6%), which capture sampling variance but not finite-question-set bias. Confidence intervals around the mean accuracy would need to account for the small number of underlying questions, but the paper reports only run-level standard deviations.

Second, domain overlap concern: Both Bamboogle and BamTwoogle are described as datasets of multi-hop questions that require web search to answer, drawn from Wikipedia-covered topics. The training seed questions come from HotpotQA (multi-hop reasoning over Wikipedia), Eli5 (long-form QA requiring explanation), and two other QA datasets. All four training sources and both evaluation sources are in the broad category of "knowledge-seeking questions answerable through web search." The paper provides no analysis of whether the evaluation questions are systematically easier, harder, or different in kind from the training questions. If the training seed questions happen to cover similar topics, entities, or reasoning patterns as the evaluation questions, the self-improvement gains may partially reflect memorization or narrow generalization rather than genuine improvement in the agent's search and reasoning capabilities.

Third, Bamboogle as a repeatedly-queried development set: Bamboogle is used for "temperature selection, checkpoint selection, iteration decisions, and measuring the impact of various hyperparameters" (Section 4.2). While the paper correctly notes that Bamboogle questions are not used for training, the repeated use of the same 125 questions for model selection decisions creates a risk of overfitting through these selection choices. If, by chance, a particular temperature or checkpoint performs well on Bamboogle but would not generalize, the paper's pipeline would select it. The BamTwoogle results partially mitigate this concern — the pattern of improvements holds on BamTwoogle (Table 2) — but BamTwoogle is also small (100 questions) and was constructed by the same authors with knowledge of Bamboogle's characteristics.

What evidence exists in the paper. Table 2 reports human evaluation on both datasets for the final models. The BamTwoogle results are consistently slightly lower than Bamboogle (e.g., 2nd gen L: 74.0% vs. 74.4%; prompted L: 68.0% vs. 68.8%), consistent with the paper's claim that BamTwoogle is "slightly more challenging" (Section 4.3). However, the gap between prompted L and 2nd gen XS is larger on BamTwoogle (68.0% vs. 63.0% = 5 point gap) than on Bamboogle (68.8% vs. 67.2% = 1.6 point gap), suggesting that the distillation advantage shrinks on harder questions — a trend the paper does not discuss.

Mitigation status. The paper introduces BamTwoogle specifically "to guard against... overfitting" to Bamboogle (Section 4.3), which is a genuine mitigation. However, BamTwoogle is also small (100 questions), also constructed by the same authors, and described as "a complementary, slightly more challenging sequel to Bamboogle" — meaning it shares the same domain and question style. The paper does not evaluate on any existing benchmark from the broader QA literature (e.g., Natural Questions, TriviaQA, or a task outside long-form QA entirely). The mitigation therefore addresses the risk of overfitting to Bamboogle's specific 125 questions but does not address the risk that the entire approach is tuned to the specific characteristics of multi-hop Wikipedia-based question answering as operationalized by these two small, author-constructed datasets.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a fundamentally new algorithm—ReST was proposed by Gulcehre et al. (2023), ReAct by Yao et al. (2022), and self-critique by Shinn et al. (2023). Its contribution is a synthesis that shifts the conversation around agent training from "can we avoid human labels?" to "can we build training pipelines where the only human input is the initial prompt engineering?" This is an incremental but practically significant reframing: the paper draws a line in the sand declaring that for process-based agents doing long-form QA, human labels are unnecessary at every stage—trajectory collection, data filtering, reward modeling, and evaluation. Before this work, the dominant assumption was that process-based systems required either human process supervision (Lightman et al., 2023) or outcome-based filtering (STaR, ReST^EM). The paper empirically challenges that assumption with a concrete working pipeline.

The conceptual shift is specifically about where the learning signal comes from in multi-step agent training. The paper demonstrates that an LLM judge evaluating individual reasoning steps—not final outcomes, not human preference comparisons, not ground-truth answer verification—can provide sufficient signal for iterative improvement. This is a different axis from prior work. STaR and ReST^EM require correct final answers to filter reasoning chains; they work for math (where correctness is binary) but cannot handle open-ended questions where many answers are acceptable. ReST and RAFT require reward models trained on human preferences; they work when human preference data is available but don't when it isn't. This paper shows that for a certain class of tasks (long-form QA with attribution), a zero-shot prompted LLM evaluating intermediate actions is sufficient. The finding that "the model can learn something useful even from the states that eventually lead to the wrong final answer" (Section 6) is the key conceptual insight that enables this—it decouples step quality from trajectory quality.

The work also provides a partial resolution to a tension in the self-improvement literature. Prior work had shown self-correction either helps (Madaan et al., 2023) or hurts (Huang et al., 2023) depending on the setup. This paper adds a nuance: self-critique helps, but the benefit is model-size-dependent. Large models (PaLM 2-L) gain 0.6-1.1 percentage points from self-checks; the smallest model (PaLM 2-XS) shows negligible benefit (0.3 points, with one case of -0.4). This suggests that self-critique is a capability that emerges with scale and does not transfer well through distillation—a finding that contextualizes the contradictory prior results by suggesting they tested different model scales and task difficulties.

The paper also shifts the narrative around distillation in agent contexts. Standard distillation asks: can a small model mimic a fixed teacher? This paper asks: can a small model benefit from a teacher that is itself improving through the same data pipeline? The finding that data quality improves more than data quantity for small models (9% gain from quality improvement between 1st and 2nd gen vs. 2.2% from doubling data, Table 3) suggests that the main distillation bottleneck is not teacher capacity but data quality, and that iterative teacher improvement is an effective way to raise that quality ceiling. This makes iterative self-improvement a more attractive research direction for agent training than it was before this paper: it's not just about improving the large model; it's about generating better training data that benefits all model sizes downstream. Conversely, it makes static prompting-only approaches less attractive as a long-term strategy—if a prompted model can bootstrap its own improvement loop, then manual prompt engineering alone leaves substantial capability on the table.

The paper's negative result on human filtering (Table 3, pilot columns: human-filtered data reduces PaLM 2-XS performance from 47.2% to 44.7%) has implications beyond this specific system. It challenges the widely-held assumption in data-centric ML that cleaning training data improves outcomes. The explanation—that filtering individual steps doesn't remove them from the PAST_ACTIONS context of downstream steps—identifies a structural challenge for process-based training data: the unit of data quality is the trajectory, not the step. This insight should influence how future work approaches data curation for agent training: rather than filtering bad actions, improve the policy that generates trajectories so that bad actions become rarer in the first place. This is a shift from curation to generation quality as the primary lever for training data improvement.

Follow-Up Research This Work Enables

RM contribution isolation: a minimal-data ablation with perplexity-only selection. The most urgent open question from this paper is whether the AI reward model is actually necessary for the observed improvements. A clean experiment would replicate the 1st generation data pipeline (2,000 trajectories from prompted PaLM 2-L) but build two fine-tuning mixtures: one with RM re-ranking (the paper's method) and one using only perplexity-based selection (the on-policy default). Fine-tune PaLM 2-XS, S, and L on both mixtures and compare Bamboogle auto-eval performance. If the perplexity-only models match or approach the RM-re-ranked models, the RM contributes little and practitioners can skip the expensive re-ranking step. If there's a substantial gap, the RM is load-bearing, and follow-up work should characterize how RM quality affects downstream performance—comparing instruction-tuned vs. base-model RMs, different RM model sizes, and ensemble RMs. This ablation is straightforward to run (no new data generation needed—just re-process the existing candidate sets) and would substantially clarify the method's active ingredients.

Saturation analysis of the iterative loop beyond two iterations. The paper's central unanswered question is: "How many additional iterations of self-improvement can we undertake past the 2nd one that still results in non-trivial gains? What does the saturation look like for smaller models?" (Section 6). A direct continuation experiment would run 3-5 additional iterations using the same protocol: at each iteration, use the best PaLM 2-L model from the previous iteration to generate 2,000-8,000 new trajectories, apply RM re-ranking, and fine-tune all three model sizes. Plot the Bamboogle auto-eval accuracy for each model size against iteration number. The key measurements are: (a) does performance plateau, continue to improve linearly, or eventually degrade? (b) do different model sizes plateau at different rates—specifically, does XS continue to catch up to L, or does it hit a capacity ceiling? (c) does trajectory diversity collapse in later iterations (measurable as the entropy of generated actions decreasing)? The ReST^EM experience (Singh et al., 2023) where additional revisions degraded performance, and the broader risk of mode collapse in iterated self-training, make this a genuine stress test. A positive result (continued improvement) would validate the approach's scalability; a negative result (plateau or degradation by iteration 3-4) would define the method's practical boundary and motivate techniques to prevent collapse, such as mixing in data from earlier iterations or constraining the policy update.

Cross-architecture transfer: testing the ReST-like loop with a different agent design. The paper's entire pipeline is built on a specific five-phase state machine with code-format prompts. An open question is whether the self-improvement loop depends on this particular architecture or generalizes across agent designs. A strong follow-up would: (a) build a ReAct agent for the same long-form QA task but using natural language prompts (no code formatting) and without the explicit five-phase decomposition—just the standard ReAct thought-action-observation loop; (b) generate trajectories from prompted PaLM 2-L with this agent on the same 2,000 training questions; (c) apply the same RM re-ranking to individual actions within those trajectories; (d) fine-tune models and measure self-improvement gains. If the gains are comparable (or even exist at all), the paper's approach is architecture-agnostic and the key ingredients are the iterative loop plus AI feedback on actions. If the gains disappear, the structured state machine with its cleanly typed actions is load-bearing—perhaps because the RM's ranking quality depends on clearly defined action types, or because state-wise decomposition is harder when actions aren't explicitly typed. This experiment would define the scope of the method's applicability.

Grounding signal replacement with a simple heuristic RM. The paper's AI reward model is a black-box instruction-tuned PaLM 2-L. This raises cost and reproducibility concerns: running a large model to rank every candidate action is expensive, and the RM's behavior is determined by opaque instruction-tuning choices. A practical follow-up would test whether a simpler, cheaper RM can substitute. Specifically: design a heuristic RM that scores actions based on observable features—does the search query contain key terms from the question? Does the summarization cite specific link IDs? Is the answer length within a reasonable range? Are self-checks passing vs. failing? This heuristic RM would be deterministic, fast, and model-agnostic. Generate the same 2,000 trajectories, apply heuristic ranking instead of LLM ranking, fine-tune PaLM 2-XS, and compare against the LLM-ranked baseline. If the heuristic RM approaches the LLM RM's performance, the key signal is simply "prefer well-formed, non-empty, properly-cited actions"—not sophisticated quality judgment. If performance drops substantially, the LLM judge is capturing non-trivial quality dimensions that heuristics miss, justifying the cost. This experiment would also be a negative result stress test: if even a weak heuristic RM produces improvements (just smaller ones), it suggests the self-training loop itself—not the specific RM—is doing most of the work.

Distillation baseline comparison: standard knowledge distillation from a fixed prompted teacher. The paper claims that self-improvement enables distillation, but never compares against the simpler alternative: directly distilling the prompted PaLM 2-L teacher's outputs. The experiment: take the same 2,000 training questions, generate trajectories from the prompted (not fine-tuned) PaLM 2-L, apply RM re-ranking, and fine-tune PaLM 2-XS. Compare this "single-shot distillation" against the 1st gen and 2nd gen XS models from Table 1. If the single-shot distilled model matches the 1st gen model (~54%), then the first iteration of self-improvement adds nothing beyond what distillation from the prompted teacher already provides—the RM re-ranking and trajectory decomposition are doing the work, not the iterative loop. If it matches the 2nd gen model (~66%), then the entire iterative self-improvement is unnecessary for distillation—you just need better data processing (RM re-ranking) applied to the prompted teacher's outputs. This experiment would clarify whether the gains attributed to "self-improvement" are actually from the ReST-like iteration or from the off-policy RM re-ranking applied to a fixed teacher's data.

Cross-model-family replication to probe the code-format dependency. The paper notes that "only PaLM 2-L can perform [the code-format prompting] consistently well" (Section 3.1). This is simultaneously a finding and a limitation—it means the entire pipeline depends on a model family that can follow code-format prompts reliably. A replication experiment using a different model family that also supports code-format prompting (e.g., GPT-4, Claude, or CodeLlama at sufficient scale) would test whether the self-improvement gains are specific to PaLM 2 architecture and training or generalize to any sufficiently capable code-literate LLM. Generate 2,000 trajectories from the prompted GPT-4 (or equivalent), apply RM re-ranking using the same instruction-tuned PaLM 2-L (to isolate the policy model from the RM), fine-tune a smaller model from the same family, and measure Bamboogle auto-eval. If the gains replicate, the approach is model-family-agnostic. If they don't—perhaps because PaLM 2 has specific properties that make its trajectory data more suitable for self-training—the method's generality is limited. This experiment also tests whether the RM can effectively rank trajectories from a different model distribution, which the paper's within-family experiments don't address.

Practical Applications and Downstream Use Cases

Cost-efficient agent deployment with small on-device models for long-form QA. The most direct application of this work is deploying small language models as capable search agents for question-answering tasks where large-model inference is cost-prohibitive. The paper's headline result shows PaLM 2-XS reaching 65.9% Bamboogle auto-eval accuracy—within 4.4 percentage points of the prompted PaLM 2-L teacher (70.3%)—after training on 2nd generation synthetic data. Under human evaluation, the gap narrows to 1.6 percentage points (67.2% vs. 68.8%, Table 2). A deployment scenario: an organization wants to offer a search-augmented QA agent to users but cannot afford PaLM 2-L inference costs at scale. Instead, they use the prompted PaLM 2-L to generate ~8,000 high-quality trajectories (one-time cost), apply RM re-ranking, and fine-tune a PaLM 2-XS model that is approximately two orders of magnitude cheaper per query. The fine-tuned XS model handles the vast majority of user queries with near-teacher accuracy, while the large teacher model is reserved only for edge cases or periodic re-generation of training data. The specific cost tradeoff depends on inference volume—the paper doesn't provide the crossover point—but for applications serving millions of queries, the upfront training data generation cost amortizes favorably. The practical recipe is: (1) craft few-shot prompts for your agent architecture, (2) generate 2,000-8,000 trajectories with a large prompted model on a diverse set of seed questions, (3) apply LLM-based re-ranking to select the best action at each trajectory step, (4) fine-tune your deployment-scale model on the decomposed step-wise data.

Synthetic training data generation for agent bootstrapping in new domains. The paper's pipeline provides a template for creating training data for multi-step agents in domains where human demonstrations are unavailable or too expensive. The key insight for practitioners is that the seed questions for trajectory generation—the 2,000 questions drawn from HotpotQA, Eli5, and related sources—are used as inputs only. No labels, answers, or human judgments are needed. This means the approach can bootstrap agent training for any domain where you can: (a) define the agent's action space (what tools it can call, what reasoning steps it performs), (b) write few-shot prompts demonstrating correct behavior, and (c) collect a modest number (hundreds to low thousands) of seed queries representative of the deployment distribution. The large prompted model generates trajectories for these seed queries, the RM ranks actions, and small models are fine-tuned. This is immediately applicable to domains like: customer support agents that search internal knowledge bases, research assistants that query scientific databases, or coding agents that interact with documentation and version control. The paper's finding that 500 trajectories (pilot data) already produces a competent PaLM 2-XS agent (47.2% accuracy, Table 3) suggests that even a modest initial investment in trajectory collection can bootstrap a working system—you don't need the full 8,000-trajectory pipeline to get started. The diminishing returns beyond 2 trajectories per question (Table 3: 65.6% for 2× vs. 65.9% for 4×) means practitioners can be efficient: generate 2 diverse trajectories per seed question, apply RM re-ranking, and stop—quadrupling the data provides essentially zero additional benefit.

Iterative improvement of production agents without human annotation pipelines. For organizations already deploying LLM agents in production, the paper provides a mechanism for continuous improvement that doesn't require setting up human annotation workflows. The self-improvement loop can operate on logged production queries: collect a set of representative user questions, run them through the current production agent to generate trajectories, apply RM re-ranking off-policy to improve data quality, fine-tune, and deploy the improved model. This creates a feedback loop where the agent improves from its own (re-ranked) behavior on real user queries, without any human labeling. The finding that "better data also reduces the variance of evaluation trajectories" (Section 5.3, ±3.6% → ±1.7% standard deviation from 1st gen to 2nd gen) is practically significant here: the improved agent is not just more accurate but more reliable, producing consistent behavior across different runs. For production systems where unpredictability is as problematic as inaccuracy, this variance reduction is a concrete operational benefit. The practical limitation is the cost of RM re-ranking—running a separate large LLM on every logged trajectory step. The paper's off-policy approach (RM used only during data preparation, not during live trajectory execution) makes this feasible as a periodic batch job (e.g., weekly or monthly data refresh) rather than a per-query cost.

When to Prefer This Method

The paper does not explicitly articulate a decision rule comparing its ReST-like AI-feedback approach against named alternatives (e.g., "use this instead of WebGPT-style human demonstrations when X" or "prefer this over STaR when Y"). It positions itself as demonstrating feasibility—that self-improvement for process-based agents works without human labels—rather than establishing a clear set of conditions under which practitioners should choose it over other methods. The paper's comparisons are primarily internal (across model sizes and training data generations within its own pipeline) rather than external (against FireAct, standard ReST, WebGPT-style imitation learning, or other published agent training methods). The absence of baselines against these alternatives means the paper does not provide the evidence needed to construct a principled decision rule. A practitioner reading this paper learns that the method works for the specific Search Agent on the specific task, but cannot determine from the reported experiments whether it would outperform simply fine-tuning on human-labeled data, distilling from the prompted teacher without iterative improvement, or using an instruction-tuned model directly as the agent. Any "when to prefer" guidance would need to be constructed from external knowledge about the cost of human annotation, the availability of capable prompted models, and the importance of process-based vs. outcome-based evaluation—none of which is quantified or compared in this paper.