RL Training2023~111 minarXiv:2308.02151

Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization

score 9Weiran Yao, Shelby Heinecke, Juan Carlos Niebles +12↑ 21★ 39Aug 4, 2023cs.CLhttps://github.com/weirayao/retroformer

🎯 Pitch

LLM agents can reward-train themselves without backprop through the frozen model. A compact retrospective model learns to diagnose failures and rewrite prompts using policy gradients, boosting success by up to 36% over verbal self-reflection.

1. Executive Summary

This paper introduces Retroformer, a principled framework for reinforcing large language agents by learning a plug-in retrospective model that automatically tunes the agent's prompts from environment feedback through policy gradient optimization. The method is evaluated on three real-world benchmarks—HotPotQA (search-based question answering), AlfWorld (embodied robotics tasks), and WebShop (web browsing)—using GPT-3 and GPT-4 as frozen actor LLMs paired with a fine-tunable LongChat-7b retrospective model. Retroformer learns from arbitrary reward signals by treating the actor LLM as part of the environment and applying proximal policy optimization (PPO) to the retrospective model, which generates self-reflections that diagnose failure root causes and propose corrective action plans (for instance, detecting that an agent submitted both "Teen Titans" and "Teen Titans Go" when only the spinoff series was correct, then prompting focused retrieval from prior search results). The framework improves success rates substantially—18% on HotPotQA with 4 retries, 36% on AlfWorld with 3 retries, and 4% on WebShop—while circumventing the need to access or propagate gradients through the frozen actor model, establishing that gradient-based fine-tuning of a lightweight retrospective module can outperform verbal-only self-reflection baselines when the retrospective component is trained to perform credit assignment on environment-specific failure trajectories rather than relying on a frozen LLM's generic self-critique capabilities.

2. Context and Motivation

The Core Problem: Language Agents Cannot Learn From Environment Rewards

The fundamental gap this paper addresses is deceptively simple: most current large language agents cannot systematically improve their behavior from environment-specific reward signals. The recent explosion of autonomous language agents—systems that use LLMs to generate text-based actions, make API calls, and execute multi-step tasks in environments—has demonstrated remarkable zero-shot capabilities. ReAct (Yao et al., 2023) pioneered the integration of reasoning traces with action generation, enabling agents to interleave thought processes with environment interactions. Toolformer (Schick et al., 2023) showed LLMs can teach themselves to use external tools. HuggingGPT (Shen et al., 2023), Generative Agents (Park et al., 2023), WebGPT (Nakano et al., 2021), AutoGPT (Gravitas, 2023), and BabyAGI (Nakajima, 2023) have all successfully demonstrated viable agent architectures across diverse domains.

Yet these systems share a critical limitation: they are frozen at deployment time. Once the LLM's parameters are trained, the agent's behavior is fixed—it cannot adapt, improve, or learn from the consequences of its own actions within a specific environment. An agent that repeatedly makes the same mistake (for instance, searching for overly broad terms or forgetting its original goal during a lengthy interaction chain) will continue making that mistake on every subsequent attempt, because there is no mechanism for converting environment feedback—such as task success or failure—into behavioral improvement.

This is a significant gap for several practical reasons the authors highlight throughout the paper (Section 1, Appendix A):

Persistent failure modes: As showcased in Figure 1, agents can enter infinite loops—for example, repeatedly examining a stoveburner instead of heating a mug with it, or submitting incorrect answers that include both correct and incorrect components (e.g., "Teen Titans and Teen Titans Go!" when only "Teen Titans Go!" is the spinoff). Without a learning mechanism, these failure modes are permanent.
Deployment cost: When agents are deployed in real-world settings (customer support, robotic control, web automation), failure carries a cost—wasted API calls, incorrect purchases, or physical errors. An agent that cannot learn from these failures is inefficient and potentially dangerous.
Environment specificity: Pre-trained LLMs possess broad world knowledge, but they lack understanding of the specific reward structure and failure patterns of particular environments. An agent navigating WebShop's product database or AlfWorld's physical rooms needs to learn environment-specific strategies that no pre-training corpus can provide.

The paper frames this as a credit assignment problem (Sutton & Barto, 2018): when an agent executes a multi-step trajectory that ultimately fails, which specific actions or decisions caused the failure? And how should future behavior be modified to avoid this outcome? This is a classic reinforcement learning challenge, but applying it to language agents raises unique difficulties because actions are generated autoregressively in natural language rather than selected from a discrete action space.

The Reflexion Approach and Its Limitations

The paper identifies Reflexion (Shinn et al., 2023) as the most directly relevant prior work that attempts to address this gap. Reflexion introduces verbal reinforcement: after a failed episode, a frozen LLM is prompted to generate a textual self-reflection summarizing what went wrong and what should be done differently. This reflection is appended to the agent's prompt as long-term memory (Section 4.1), providing context intended to prevent repetitive errors in subsequent attempts.

Related approaches include Self-Refine (Madaan et al., 2023b), which uses a single LLM as generator, refiner, and feedback provider for iterative output improvement, and Generative Agents (Park et al., 2023), which uses self-reflection for memory consolidation in social simulation scenarios. RAP (Hao et al., 2023) repurposes the LLM as both a world model and reasoning agent, incorporating Monte Carlo Tree Search for strategic exploration with environment rewards.

However, the paper argues that these verbal-only reflection approaches suffer from a fundamental weakness: the frozen LLM is not optimized for credit assignment in specific environments. As Figure 1 demonstrates concretely, the self-reflections generated by a frozen LLM can be uninformative or actively harmful. The example shows a HotPotQA task where the agent failed because it submitted both "Teen Titans" and "Teen Titans Go!" as the answer to a question about a Teen Titans spinoff series—the correct answer is only "Teen Titans Go!" The frozen model's self-reflection merely rephrases the prior failed action sequence as a proposed plan:

"I should have searched for Lollipop Chainsaw first and looked up the Canadian-American actress who voiced Juliet Starling afterwards. I also should have looked up Tara Strong's filmography and searched for any voice roles she did specifically for Teen Titans or Teen Titans Go!"

This reflection does not identify the actual error (submitting "Teen Titans" alongside the correct spinoff) and instead recommends actions that the agent already performed in the failed episode. When this reflection is appended to the prompt for the next attempt, the agent repeats essentially the same actions, leading to the same failure—potentially creating an infinite loop.

The core issue is that generating useful reflective feedback requires two capabilities that a frozen, general-purpose LLM may lack:

Environment-specific credit assignment: Understanding which specific action in a multi-step trajectory caused the failure, given the environment's particular reward structure and action semantics. This is not a generic language understanding task—it requires modeling the causal structure of how actions lead to outcomes in a specific environment.
Actionable insight generation: Producing a summary that not only identifies the error but proposes a concrete, environment-appropriate correction that will actually prevent the failure in the next attempt. A vague suggestion like "search more carefully" is less useful than "search for the spinoff series specifically and check the search results for Tara Strong's Teen Titans roles."

The paper argues (Section 1) that these capabilities can be acquired through gradient-based learning from environment rewards, but that existing approaches do not exploit the available reinforcement learning techniques to do so.

The Disconnect Between Language Agents and Reinforcement Learning

A deeper motivation the paper identifies is a methodological gap between two fields that should be naturally aligned: language agents and reinforcement learning. The authors observe that:

Transformer reinforcement learning (RLHF, PPO, ILQL, DPO) has become standard practice for aligning LLM outputs with human preferences in domains like machine translation, summarization, and helpfulness (Ouyang et al., 2022; Rafailov et al., 2023; Snell et al., 2022). These methods have demonstrated that gradient-based optimization can steer language models toward desired behaviors defined by reward functions.
Yet existing language agents do not reason and plan in ways that are compatible with differentiable, gradient-based learning from rewards. As Table 1 in the paper summarizes, approaches like CoT, ReAct, Self-Refine, RAP, and Reflexion all lack gradient learning from arbitrary reward signals. They may use rewards for search (RAP) or verbal feedback (Reflexion), but none apply policy gradient optimization to directly tune model parameters based on environment returns.

The paper positions this disconnect as an opportunity. If we can bridge the gap—applying gradient-based RL to language agent components while respecting the constraints of frozen, cloud-hosted actor LLMs—we can create agents that genuinely learn from experience rather than merely prompting for verbal feedback.

Why Direct RL on the Actor LLM Is Impractical

The paper acknowledges a practical constraint that shapes the entire approach: most well-performing LLMs are too large to fine-tune directly, and cloud-hosted models (e.g., OpenAI GPT, Google Bard) have inaccessible parameters (Appendix A, Section 4). This creates a tension:

On one hand, we want the agent to learn from environment rewards through gradient-based optimization, which traditionally requires updating model parameters.
On the other hand, the actor LLM—the component that generates actions and interacts with the environment—is frozen by practical necessity.

Prior RLHF approaches assume access to the model being fine-tuned. But for language agents deployed with cloud APIs, this access does not exist. The paper frames this as a "prohibitive training" challenge (Appendix A): "it is technically challenging to optimize the LLMs directly as is done in the classical reinforcement learning setting. In particular, OpenAI has not provided any solution for RL based finetuning."

This constraint motivates the central architectural insight of the paper: rather than trying to optimize the frozen actor LLM, we can optimize a separate, smaller model whose outputs influence the actor's behavior indirectly. The retrospective model acts as an automatic prompt tuner—it learns to generate better reflections over time, which in turn improves the actor's decisions when appended to the prompt. The actor LLM itself is treated as part of the environment (Section 3, Equation 2), since its parameters are fixed and its behavior is a deterministic function of its input prompt.

The Meta-Challenge: Can Verbal Feedback Really Be Optimized?

Underlying the entire paper is a deeper question: can we treat the generation of self-reflective text as an optimization problem that gradient-based RL can solve? This is not obvious. Self-reflection is an open-ended natural language generation task—the "action space" is the set of all possible text sequences, and the "reward" (improvement in the next episode's return) is delayed, noisy, and depends on the complex interaction between the reflection text, the actor LLM's interpretation of it, and the environment dynamics.

The paper's response rating scheme (Section 4.2) provides the key insight that makes this optimization tractable: the difference in episode returns between consecutive trials ( $\Delta G_{k,i} = G_{k,i+1} - G_{k,i}$ ) serves as a natural reward signal for the reflection generated after trial $i$ . If the reflection helps the agent improve, the next episode's return increases (positive reward); if it leads to worse performance, the return decreases (negative reward). Because the actor LLM is frozen and operates at low temperature, the authors argue that most of the variance in $\Delta G$ comes from differences in the reflection quality—making $\Delta G$ a reasonable proxy for reflection quality.

This formulation transforms the open-ended problem of "generate good advice" into a well-defined RL problem: learn a policy $\pi_\phi(y|x)$ (the retrospective model) that maps reflection prompts $x$ (containing the failed trajectory and reward) to reflection responses $y$ , such that the expected improvement in the next episode's return is maximized. The PPO objective in Equation 6 operationalizes this, with the added KL constraint ensuring the fine-tuned model doesn't diverge too far from its pre-trained behavior.

How This Paper Positions Itself

The paper positions Retroformer at the intersection of two trends—autonomous language agents and transformer reinforcement learning—that have developed largely independently. The contributions section (Section 1) frames the work as addressing a specific, actionable gap: existing agents don't use gradient-based learning from environment rewards, and this limits their ability to improve over time.

The positioning relative to Reflexion is direct and explicit: Reflexion shows that some learning from verbal feedback is possible with frozen models, but Retroformer demonstrates that reinforcing the reflection generator with policy gradients yields substantially better learning—faster improvement and higher asymptotic performance. The paper does not claim Reflexion is wrong, but rather that it is suboptimal because the reflection component is not trained for the specific credit assignment demands of the environment.

The paper also positions its framework as agnostic and modular (Section 6): because only the retrospective model is fine-tuned (not the actor LLM), Retroformer can be plugged into any agent architecture that uses cloud-hosted LLMs. The approach is not limited to refining the retrospective model alone—it could be applied to other components like memory modules, summarizers, or even the actor prompt itself—but the retrospective model is the natural first target because it is the component responsible for converting environment feedback into behavioral guidance.

Summary of the Gap

To synthesize: the paper identifies a clear, multi-layered gap in the language agent literature:

Behavioral gap: Agents cannot learn from environment rewards, leading to persistent failure modes and no improvement over time.
Methodological gap: Existing verbal feedback approaches (Reflexion, Self-Refine) rely on frozen LLMs that are not optimized for the credit assignment demands of specific environments, producing uninformative or counterproductive reflections.
Architectural gap: Direct RL fine-tuning of actor LLMs is impractical due to model size and cloud API constraints, requiring a different optimization target.
Conceptual gap: The language agent and RL communities have not been bridged—there is no principled framework for applying policy gradient optimization to language agent components while respecting the constraints of frozen, cloud-hosted models.

Retroformer is positioned as the first framework to address all four simultaneously: it learns from arbitrary reward signals, optimizes the reflection generator specifically for credit assignment, treats the frozen actor as part of the environment, and applies standard PPO to a lightweight plug-in retrospective module. The experimental results—18% improvement in HotPotQA, 36% in AlfWorld, 4% in WebShop—are presented as evidence that gradient-based optimization of the verbal feedback mechanism yields substantial gains over verbal-only approaches.

3. Technical Approach

3.1 Reader Orientation

Retroformer is a system that adds a trainable "reflection module" to an existing frozen language agent—this module looks at what the agent did wrong in a failed task and generates written advice that gets appended to the agent's prompt for the next attempt, with the crucial property that the reflection module itself is fine-tuned using reinforcement learning to generate better and better advice over time. The problem it solves is that existing language agents cannot learn from their mistakes in a given environment—they repeat the same errors because their behavior is frozen at deployment time—and the shape of the solution is to treat the frozen actor LLM as a fixed part of the environment, then apply policy gradient optimization (specifically PPO) to a smaller, locally-hosted language model whose outputs (self-reflections) influence the actor's future decisions, thereby improving the overall agent without ever needing access to the actor model's parameters or gradients.

3.2 Big-Picture Architecture (Diagram in Words)

The system comprises five major components interacting in a loop:

Actor LLM (frozen): A cloud-hosted large language model (e.g., GPT-3, GPT-4) that receives a prompt containing the current task, past observations, past actions, and any accumulated self-reflections, and generates the next action or final answer. Its parameters are never modified.
Environment: The task environment (HotPotQA, AlfWorld, or WebShop) that receives the actor's text actions, executes them, and returns observations and rewards. It includes APIs like Wikipedia search, robotic object manipulation commands, or web browsing actions.
Retrospective Model (trainable): A smaller, locally-hosted language model (LongChat-7b-16k) that takes as input the full trajectory of a failed episode (the sequence of states, actions, and rewards) plus the final episode return, and outputs a self-reflection—a textual diagnosis of what caused the failure and a proposed corrective plan for the next attempt.
Memory Module: Two types of memory buffer the agent's experience. Short-term memory holds the current episode's trajectory in the prompt. Long-term memory holds accumulated self-reflections from prior failed attempts, appended to the actor prompt to prevent repetitive errors. A replay buffer stores reflection instruction-response pairs with their associated episode returns across multiple tasks and environments, enabling offline RL training.
PPO Trainer: The reinforcement learning optimization loop that fine-tunes the retrospective model. It uses the difference in episode returns between consecutive trials as a reward signal for the quality of a reflection response, treating the frozen actor and environment as part of the environment dynamics.

Information flows as follows: a task is presented → the actor prompt is assembled with prior reflections (if any) → the actor generates actions and interacts with the environment → at episode end, the trajectory and return are fed to the retrospective model → the retrospective model generates a self-reflection → this reflection is appended to the actor prompt as long-term memory → the actor attempts the task again with the enriched prompt → the difference in returns between the two attempts serves as a rating for the reflection → these ratings are stored in the replay buffer → PPO periodically fine-tunes the retrospective model to generate reflections that produce higher improvement.

3.3 Roadmap for the Deep Dive

First, the formal mathematical formulation (Section 3 of the paper) that treats the actor LLM as part of the environment and defines the RL objective—this establishes what is being optimized and why standard RL algorithms apply despite the unusual setting.
Second, the actor model and retrospective model architecture (Section 4.1)—how they communicate, what prompts they receive, and how reflections are integrated into the agent's decision-making loop—since the precise information flow determines what the PPO trainer can optimize.
Third, the memory module design (short-term, long-term, and replay buffer memory)—because the distinction between in-episode trajectory context and cross-episode reflective memory is central to how the agent accumulates knowledge over multiple retries.
Fourth, the reward shaping and response rating mechanism (Sections 4.1 and 4.2)—how episode returns are computed in each environment, and how the difference in returns between consecutive trials is converted into a training signal for the retrospective model, since this rating scheme is the key innovation that makes gradient-based optimization tractable.
Fifth, the policy gradient optimization pipeline (Section 4.2 and Algorithm 1 in Appendix C)—the three-step offline training procedure (data collection, reward model learning, PPO fine-tuning), the exact PPO objective, and the hyperparameters—because this is the actual learning mechanism that distinguishes Retroformer from frozen verbal feedback approaches.
Sixth, the training and data collection protocol (Appendix C)—including the specific models, hardware, datasets, and the offline data collection strategy—since the practical feasibility of the approach depends on how reflection samples are gathered and rated.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methods paper whose core idea is that a lightweight retrospective language model can be trained via policy gradient to generate high-quality self-reflections that improve a frozen actor LLM's performance, by treating the frozen actor as part of the environment and using the difference in episode returns between consecutive trials as a reward signal for reflection quality.

Formal Problem Formulation: Treating the Frozen Actor as Part of the Environment

The paper's mathematical formulation in Section 3 establishes the foundation that makes standard RL applicable to the Retroformer setting. The key insight is that by treating the frozen actor LLM as a component of the environment dynamics, the problem of improving the agent through prompt refinement reduces to a standard RL optimization over the retrospective model's parameters.

The agent and environment definitions. The paper defines a language-model-based action agent as a function:

$M_{\xi_l} : \mathcal{X} \to \mathcal{A}$

where $\mathcal{X}$ is the space of prompts (including user queries $x_u$ and contextual information $c \in \mathcal{C}$ representing the current state from the environment $\Omega$ ), $\mathcal{A}$ is the space of text actions, and $\xi_l$ denotes the random variables involved in autoregressive sampling (making $M$ a stochastic function). The agent is explicitly stateless—all state and memory are represented as text in the prompt $x$ , not in the model's internal activations.

The environment is defined as:

$\mathcal{E} = (\mathcal{T}_{\xi_o}, \mathcal{R})$

where $\mathcal{T}_{\xi_o} : \mathcal{S} \times \mathcal{A} \to \mathcal{S}$ is the state transition function (with $\xi_o$ capturing environmental randomness), $\mathcal{S}$ is the space of text-represented states, and $\mathcal{R} : \mathcal{S} \to \mathbb{R}$ is the reward function that assigns a scalar reward to each state. Rewards are typically sparse—nonzero only at the terminal state to indicate task success or failure.

The retrospective model as the learnable component. The retrospective model takes the full history of states, actions, rewards, and the user prompt as input, and produces a new prompt to be consumed by the actor LLM:

$\Gamma_{\xi_r, \Theta} : [S_i, A_i, R_i, X_i^u]_{i=1}^t \to \mathcal{X}$

where $\xi_r$ is the randomness in the retrospective model's generation, and $\Theta$ is the set of learnable parameters. In plain language: the retrospective model consumes the trajectory of everything that happened in the current task (all states, actions, and rewards observed so far, plus the original user question) and outputs a refined prompt—this prompt contains the original task description plus a self-reflection that diagnoses failures and proposes corrections.

The RL objective. The optimization goal is:

$\arg\max_{\Theta} \mathbb{E}_{\xi_l, \xi_o, \xi_r} \left[ \sum_{t=1}^T \mathcal{R}(s_t) \right]$

subject to the dynamics constraint that the next state is determined by the environment transition function applied to the current state and the actor's action, where the actor's action is generated from the prompt that the retrospective model constructed:

$s_{t+1} = \mathcal{T}_{\xi_o}\left(s_t, \mathcal{L}_{\xi_l} \circ \Gamma_{\xi_r, \Theta}\left([s_i, a_i, r_i, x_i^u]_{i=1}^t\right)\right)$

for all time steps $t \in \{1, \ldots, T-1\}$ .

What it computes: This formulation says: we want to find parameters $\Theta$ of the retrospective model that maximize the expected total return (sum of rewards) over an episode, where the expectation is taken over three sources of randomness—the actor LLM's sampling ( $\xi_l$ ), the environment's stochasticity ( $\xi_o$ ), and the retrospective model's own generation randomness ( $\xi_r$ ). The dynamics equation shows how the retrospective model's output flows into the actor: the retrospective model reads the history up to time $t$ and produces a prompt; this prompt is passed through the frozen actor LLM ( $\mathcal{L}_{\xi_l}$ ) to produce an action; this action and the current state are fed to the environment transition function ( $\mathcal{T}_{\xi_o}$ ) to produce the next state.

Why this form: The critical sentence in the paper is: "Note that the only learnable parameters are in the retrospective model $M_r$ . Since the LLM action agent is frozen, it can be considered as part of the environment." By composing $\mathcal{T} \circ \mathcal{L}$ into a new transition function $\mathcal{T}' = \mathcal{T}(S, \cdot) \circ \mathcal{L} : \mathcal{S} \times \mathcal{X} \to \mathcal{S}$ with the same reward function $\mathcal{R}$ , Equation (2) becomes a regular RL optimization problem—the retrospective model is the policy, its action space is the space of prompts $\mathcal{X}$ , and the environment (including the frozen LLM) is a black-box dynamics model. This decomposition is what makes all standard RL algorithms directly applicable without needing to modify or backpropagate through the actor LLM. An alternative approach—trying to optimize the actor LLM directly—would require model access that cloud APIs don't provide and would be computationally prohibitive for large models.

Actor Model: The Frozen Decision-Making Component

The actor model (Section 4.1) is the LLM that actually interacts with the environment by generating actions. It is hosted in the cloud (e.g., OpenAI GPT-3 text-davinci-003, or GPT-4), its model parameters are inaccessible, and it is kept frozen throughout all training and evaluation.

Action generation protocol. The actor model generates actions following a ReAct-style (Yao et al., 2023) interleaved reasoning-and-acting pattern. At each time step $t$ in episode $i$ of environment $k$ , the actor receives a prompt containing the trajectory history up to the current moment and the current observation, and generates the next thought and action:

$a_{k,i,t} = M_a\left([s_{k,i,\tau}, a_{k,i,\tau}, r_{k,i,\tau}]_{\tau=1}^{t-1}, s_{k,i,t}\right)$

where $[s_{k,i,\tau}, a_{k,i,\tau}, r_{k,i,\tau}]_{\tau=1}^{t-1}$ is the sequence of all prior states, actions, and rewards in the current episode (the short-term memory), and $s_{k,i,t}$ is the current state observation from the environment. The output $a_{k,i,t}$ is a text string that typically includes a reasoning thought (e.g., "I need to search for Washington State Cougars and find the coach of the 2016 team") followed by a structured action (e.g., Search[Washington State Cougars] or Finish[Texas Tech University]).

Prompt construction. The prompt consumed by the actor model is assembled from multiple components, as shown in the full prompt examples in Appendix E.1:

A system preamble that explains the agent's role and the available action types (for HotPotQA: SEARCH[ENTITY], LOOKUP[KEYWORD], FINISH[ANSWER]; for AlfWorld: GOTO[LOCATION], TAKE[OBJ], OPEN[OBJ], etc.; for WebShop: SEARCH[QUERY], CHOOSE[BUTTON]).
Few-shot demonstrations showing examples of correct reasoning-and-action trajectories, following the ReAct format.
The current task description provided by the user or benchmark.
Long-term memory: self-reflections from prior failed attempts on the same task, each labeled with the trial number (e.g., "Trial 0: I was stuck in a loop... New plan: I should have..."). This is the component that the retrospective model produces and that Retroformer optimizes.
Short-term memory: the trajectory of the current episode (states, actions, rewards observed so far), appended as the interaction progresses.

Temperature and determinism. The paper sets the actor model's temperature to zero ( $T=0$ ) and top-p to 1 across all experiments, meaning actions are generated deterministically (greedy decoding). The authors acknowledge that higher temperature would encourage exploration but state that it "can obscure the impact of the proposed approaches, making it difficult to compare against existing baselines with T=0" (Appendix C.1). This is an important design choice because it means that, in Retroformer, the only source of behavioral change between consecutive attempts on the same task is the appended self-reflection—any improvement or degradation in performance can be attributed to the reflection's quality, not to random variation in action sampling.

Why freeze the actor. Freezing the actor model serves three purposes: (1) it makes the framework compatible with cloud-hosted, API-only LLMs where parameter access is impossible; (2) it isolates the learning problem to the smaller retrospective model, making training computationally feasible on modest hardware; (3) it ensures that improvements generalize—the retrospective model learns to write better reflections for any actor that can interpret natural language advice, not just for a specific fine-tuned actor variant.

The Retrospective Model: The Trainable Reflection Generator

The retrospective model $M_r$ (Section 4.1) is the component that learns. It is instantiated as a local, smaller language model—specifically, LongChat-7b-16k, which is a fine-tuned version of Llama-7b with a 16,000-token context window, trained on instruction-following conversations from ShareGPT. This model can be fine-tuned on GPUs that are accessible for research (A100 40GB), unlike the massive cloud-hosted actor models.

Input: the reflection prompt. After an episode $i$ in environment $k$ concludes (either with success, failure, or hitting the maximum number of actions), the retrospective model receives a structured reflection prompt that contains:

The full trajectory of the completed episode: all states $s_{k,i,\tau}$ , actions $a_{k,i,\tau}$ , and rewards $r_{k,i,\tau}$ for $\tau = 1, \ldots, T$ .
The episode return $G_{k,i}$ (a scalar summarizing the overall success or failure, such as the F1 score in HotPotQA or the binary success/failure in AlfWorld).

This prompt is formatted with a preamble that instructs the model to "Diagnose a possible reason for failure and devise a new, concise, high level plan that aims to mitigate the same failure" (Appendix E.2 shows the full retrospective instruction prompt). It also includes few-shot examples of good reflections paired with failed trajectories.

Mathematically, the reflection response is generated as:

$y_{k,i} = M_r\left(\underbrace{[s_{k,i,\tau}, a_{k,i,\tau}, r_{k,i,\tau}]_{\tau=1}^T, G_{k,i}}_{\text{Reflection prompt } x_{k,i}}\right)$

where $x_{k,i}$ denotes the complete reflection prompt (trajectory + return + instruction preamble + few-shot examples), and $y_{k,i}$ is the generated self-reflection text.

Output: the self-reflection text. The output $y_{k,i}$ is a natural language paragraph that typically contains two components that the paper observes emerging naturally (Appendix D, Figure 7):

A diagnosis of what went wrong in the failed episode—identifying which specific action or decision caused the failure. For example, in the HotPotQA example from Figure 5, the reinforced model's reflection identifies: "The agent forgot its original goal during a chain of lengthy interactions" and "I should have searched Tara Strong and Teen Titans or Teen Titans Go! specifically in my previous action to find the correct answer."
A corrective plan that proposes specific, actionable steps for the next attempt. For instance: "focus on spinoff series and find the answer in the previous actions and observations with the search results of Tara Strong."

The paper reports that the reinforced retrospective model spontaneously organizes its output into two labeled sections—"Reflection:" and "New plan:"—even though it was never explicitly trained or prompted to do so. This emergent structure is contrasted with the frozen model's output, which interleaves diagnosis and plan in one unstructured paragraph and often includes hallucinated content like spurious "Next trial:" prefixes.

How reflections influence the actor. The generated reflection $y_{k,i}$ is appended to the actor model's prompt as long-term memory (Section 4.1). In the next attempt on the same task (episode $i+1$ ), the actor sees the accumulated reflections from all prior failed attempts— $y_{k,0}, y_{k,1}, \ldots, y_{k,i}$ —in its prompt, labeled by trial number. This means the actor's behavior in episode $i+1$ is a function of the original task instructions, the few-shot demonstrations, and the history of self-reflections. Since the actor is deterministic (T=0), any change in its actions between episode $i$ and episode $i+1$ is caused entirely by the addition of $y_{k,i}$ to the prompt.

Why use a separate model instead of prompting the actor to self-reflect. The paper argues in Section 1 that generating useful self-reflections requires two capabilities that a frozen, general-purpose LLM may not possess for a specific environment: (1) accurate credit assignment—identifying which action in a multi-step trajectory caused the failure, given the environment's specific dynamics—and (2) actionable insight generation—proposing a corrective plan that will actually work in that environment. By fine-tuning a dedicated retrospective model on environment-specific trajectories, these capabilities can be learned from data rather than relying on the frozen LLM's generic reasoning.

Memory Module: Short-Term, Long-Term, and Replay Memory

The memory architecture (Section 4.1) is what enables the agent to carry information across time—within an episode, across episodes of the same task, and across different tasks during training.

Short-term memory. The trajectory history of the current episode is maintained in the actor's prompt. As the agent takes actions and receives observations, each $(s_t, a_t, r_t)$ tuple is appended to the prompt, creating a growing context that the actor conditions on when generating the next action. This is the standard approach in ReAct-style agents and serves as working memory for the current task attempt.

The paper acknowledges the limited prompt length challenge (Appendix A): as the trajectory grows, the prompt eventually exceeds the LLM's context window. This is handled implicitly by the actor model's context limit—older interactions are truncated when the prompt exceeds the maximum length.

Long-term memory. Self-reflections from prior failed attempts on the same task are stored and appended to the actor prompt as long-term memory. Each reflection is labeled with the trial number and contains the summary of what went wrong and what to do differently. In subsequent attempts, the actor sees all accumulated reflections, enabling it to avoid repeating mistakes identified in earlier trials. The paper shows examples where this accumulation is visible: in the AlfWorld prompt (Appendix E.1), the actor sees "Trial 0: I will try to find a different task to complete..." as long-term memory from the first failed attempt.

Critically, long-term memory is task-specific—it persists across retry attempts on the same task but does not transfer to new tasks. This is because the reflections are specific diagnoses of what went wrong on a particular question or instruction.

Replay buffer. To enable cross-task learning for the retrospective model, the paper introduces a replay buffer (denoted $\mathcal{D}_{\text{RL}}$ ) that stores reflection instruction-response pairs from multiple tasks and environments:

$\mathcal{D}_{\text{RL}} = \{(x_{k,i}, y_{k,i}, G_{k,i})\}$

where $x_{k,i}$ is the reflection prompt (trajectory + return) for trial $i$ in environment $k$ , $y_{k,i}$ is the generated reflection response, and $G_{k,i}$ is the episode return. This dataset is accumulated across all training tasks and all trials, providing a diverse set of examples of good and bad reflections that the PPO trainer samples from during fine-tuning.

The replay buffer serves a different purpose from long-term memory: long-term memory helps the actor avoid repeating mistakes on a specific task, while the replay buffer helps the retrospective model learn to generate better reflections in general. The paper describes this as enabling the agent to "not only exploit lessons learned over failed trials in the current task, but also explore by learning from success in other related tasks" (Section 4.1).

Data collection for the replay buffer. The specific data collection protocol is described in Appendix C.1:

For HotPotQA: 3,383 reflection samples were collected by running the base rollout policy for 3 trials each on 3,000 tasks in the training set. Among these, 1,084 instruction-response pairs have positive ratings (meaning the reflection led to an improvement in the next attempt).
For AlfWorld: 523 reflection samples were collected.
For WebShop: 267 reflection samples were collected.

The relatively small number of samples for AlfWorld and WebShop (compared to HotPotQA) reflects that these environments produce fewer failed trajectories suitable for learning—if the agent succeeds on the first try, no reflection is generated.

Reward Shaping and Response Rating

The response rating mechanism (Section 4.2) is the core innovation that converts the abstract goal of "generate better reflections" into a concrete supervised signal for RL training.

Environment-specific reward functions (Section 4.1 and Appendix C.3). The paper uses different reward functions for each environment, chosen to provide more informative feedback than simple binary success/failure:

HotPotQA: The reward is the F1 score between the generated answer and the ground-truth answer. Specifically, after removing stopwords from both answers, precision is computed as the number of common tokens divided by the number of generated answer tokens, recall as common tokens divided by ground-truth answer tokens, and F1 as the harmonic mean of precision and recall. This soft matching reward captures partial correctness—an answer that gets some tokens right but includes extraneous tokens (like "Teen Titans and Teen Titans Go!" instead of "Teen Titans Go!") receives a lower but nonzero score.
AlfWorld: The reward is a binary success indicator (1 for task completion, 0 for failure) at the terminal state. This is the sparsest reward signal of the three environments.
WebShop: The reward is a weighted matching score at the final state (when the agent clicks "Buy"), defined as:

$r = \frac{r_{\text{type}} \cdot |U_{\text{att}} \cap Y_{\text{att}}| + |U_{\text{opt}} \cap Y_{\text{opt}}| + \mathbb{1}[y_{\text{price}} \leq u_{\text{price}}]}{|U_{\text{att}}| + |U_{\text{opt}}| + 1}$

where $U_{\text{att}}$ and $U_{\text{opt}}$ are the user's desired attributes and options, $Y_{\text{att}}$ and $Y_{\text{opt}}$ are the chosen product's attributes and options, $y_{\text{price}}$ and $u_{\text{price}}$ are the chosen and budget prices, and $r_{\text{type}}$ is a text-matching heuristic that assigns a low reward when the chosen product is obviously the wrong type despite sharing attributes (e.g., "butter" vs. "plant-based meat" sharing "cruelty-free"). The numerator sums three terms: a type-matching weighted attribute overlap, an option overlap, and a binary indicator for budget compliance; the denominator normalizes by the total number of desired attributes, options, plus one.

Episode returns. The episode return $G_{k,i}$ for trial $i$ in environment $k$ is the cumulative reward over the episode, which in these sparse-reward settings is essentially the terminal reward (F1 score, binary success, or shopping score) since intermediate rewards are zero.

The response rating formula. The key insight is that the improvement in episode return from one trial to the next can serve as a rating for the quality of the reflection generated after the first trial. Formally (Section 4.2):

$r(x_{k,i}, y_{k,i}) \triangleq G_{k,i+1} - G_{k,i}$

where $x_{k,i}$ is the reflection prompt (the failed trajectory from trial $i$ ), $y_{k,i}$ is the reflection response generated by the retrospective model after trial $i$ , $G_{k,i}$ is the return from trial $i$ , and $G_{k,i+1}$ is the return from the next trial after the actor's prompt was augmented with $y_{k,i}$ .

What it computes: This formula takes the reflection response $y_{k,i}$ generated after observing the failed trial $i$ , appends it to the actor's prompt for trial $i+1$ , runs the actor on the same task with this augmented prompt, and measures how much the performance changed. If the reflection helped (the actor improved), $G_{k,i+1} > G_{k,i}$ and the rating is positive—the reflection receives a high score. If the reflection was unhelpful or harmful (the actor got worse or stayed the same), the rating is zero or negative—the reflection receives a low score.

Why this form: This rating scheme has three crucial properties that make gradient-based optimization possible:

It is a direct measure of reflection utility. The only thing that changes between trial $i$ and trial $i+1$ is the addition of $y_{k,i}$ to the actor's prompt. Since the actor is frozen and deterministic (T=0), any change in behavior and therefore return can be attributed to the reflection's content. This is stated explicitly: "Because the actor is a frozen LM and the temperature is low as default, the injected randomness that leads to differences in returns $\Delta G_{k,i} = G_{k,i+1} - G_{k,i}$ are mostly from the reflection responses $y_{k,i}$ ."
It handles both improvement and degradation. Unlike a scheme that only rewards positive outcomes, this differential rating penalizes reflections that make things worse, providing a balanced training signal. The paper uses both positive and negative ratings in the offline RL dataset.
It is environment-agnostic. The rating formula is identical regardless of whether the environment returns F1 scores, binary success, or shopping scores. The only requirement is that returns are comparable between trials (which they are, since the same task is attempted). This allows the replay buffer to mix data from multiple environments.

Alternative approaches and why they would fail. If the rating were based only on the absolute return $G_{k,i+1}$ (ignoring $G_{k,i}$ ), it would not distinguish between a reflection that improved a poor-performing agent and one that was appended before a lucky success on an easy task. If the rating were based only on whether the reflection was correct (human-labeled), it would require expensive annotation and might not correlate with actual behavioral improvement—a reflection can be factually correct about what went wrong but still not help the actor produce the right answer. The differential rating avoids both problems by directly measuring the reflection's causal effect on performance.

Data collection protocol for ratings. The paper describes a specific procedure in Algorithm 1 (Appendix C.1): for each unsuccessful task, the retrospective model generates two reflection responses (by sampling at temperature $t_s = 0.9$ , introducing diversity). Both responses are used in separate next-episode rollouts, producing two returns $G^{(1)}_{k,i+1}$ and $G^{(2)}_{k,i+1}$ . The response with the higher rating is labeled as "accepted" and the lower as "rejected." This pairwise comparison data is used to train a reward model (step 2 of Algorithm 1) that learns to predict which reflection is better, providing a more robust training signal for PPO than raw differential ratings alone.

Proximal Policy Optimization for the Retrospective Model

The training of the retrospective model uses a standard RLHF pipeline adapted for the agent setting (Section 4.2, Algorithm 1). The pipeline has three stages: offline data collection, reward model training, and policy gradient fine-tuning with PPO.

Stage 1: Offline data collection. The base policy—consisting of the frozen actor LLM and the initialized (pre-trained, not yet fine-tuned) retrospective model—is rolled out on training tasks for $N = 3$ trials per task. For each trial, the trajectory, return, reflection prompt, and reflection response are recorded. For unsuccessful tasks, two alternative reflection responses are sampled (at temperature 0.9) and each is rolled out for one additional episode to compute the rating via the difference formula. The accepted/rejected labeling based on the pairwise comparison is stored in the replay buffer $\mathcal{D}_{\text{RL}}$ .

This stage produces the dataset sizes described in Appendix C.1: 3,383 reflections for HotPotQA (of which 1,084 have positive ratings), 523 for AlfWorld, and 267 for WebShop.

Stage 2: Reward model training. Using the RewardTrainer from the TRL library, a reward model $r_\theta(x, y)$ is trained to predict the quality of a reflection response $y$ given its prompt $x$ . The reward model is trained on the accepted/rejected pairs from the data collection stage: it learns to assign higher scores to accepted responses (those that led to larger $\Delta G$ ) than to rejected ones. This is a binary classification objective, but the resulting model produces a scalar score that serves as a learned reward function for PPO.

The reward model is necessary because using raw $\Delta G$ directly in PPO would be noisy—the reward model smooths over the training data and provides a more stable learning signal by learning to predict which reflection characteristics correlate with performance improvement across many tasks.

Stage 3: Policy gradient fine-tuning with PPO. The retrospective model is fine-tuned using the PPO trainer from TRL, with the objective:

$\mathcal{L}_{\text{PPO}} = \mathbb{E}_{x \sim \mathcal{D}_{\text{RL}}} \mathbb{E}_{y \sim \text{LLM}_{\phi}^{\text{RL}}(x)} \left[ r_\theta(x, y) - \beta \log \frac{\text{LLM}_{\phi}^{\text{RL}}(y|x)}{\text{LLM}^{\text{Ref}}(y|x)} \right]$

where $x$ is a reflection prompt sampled from the replay buffer, $y$ is a reflection response sampled from the current retrospective model policy $\text{LLM}_{\phi}^{\text{RL}}(y|x)$ , $r_\theta(x, y)$ is the learned reward model's score for that response, $\text{LLM}^{\text{Ref}}(y|x)$ is the probability assigned to the same response by a frozen reference model (the pre-trained retrospective model before fine-tuning), and $\beta$ is a KL penalty coefficient.

What it computes: This objective maximizes two terms simultaneously:

The first term $r_\theta(x, y)$ is the expected reward—it encourages the model to generate reflections that the reward model predicts will lead to higher episode returns.
The second term $-\beta \log \frac{\text{LLM}_{\phi}^{\text{RL}}(y|x)}{\text{LLM}^{\text{Ref}}(y|x)}$ is a KL divergence penalty that keeps the fine-tuned model close to the reference model. If the fine-tuned model assigns much higher probability to a response than the reference model would, this term is large and negative, discouraging the policy from deviating too far from its pre-trained behavior.

The expectation is taken over both the replay buffer distribution of prompts and the model's own generated responses, making this an on-policy-like objective (the model is trained on its own outputs).

Why this form: The KL penalty is crucial because it prevents the fine-tuned model from collapsing into a degenerate policy that generates high-reward but nonsensical or repetitive reflections. Without the KL penalty, the model might learn to exploit the reward model by generating reflections that score highly under $r_\theta$ but are not actually useful (reward hacking). The KL penalty anchors the model to its pre-trained language understanding, ensuring that the generated reflections remain coherent natural language. This is the same motivation as in standard RLHF for dialogue models (Ouyang et al., 2022).

Hyperparameters. The specific training hyperparameters from Appendix C.1 are:

Supervised fine-tuning (on positive-rating samples, run for 2 epochs before PPO): learning rate $1 \times 10^{-5}$ , batch size 32, maximum 5,000 steps.
Reward model training: learning rate $2.5 \times 10^{-5}$ , batch size 32, maximum 20,000 steps.
PPO fine-tuning: learning rate $1.4 \times 10^{-5}$ , maximum 20,000 steps, output maximum length 128 tokens, batch size 64, gradient accumulation steps 8, PPO epochs 4.
Model quantization and adapters: The retrospective model is fine-tuned using 4-bit quantized LoRA (Low-Rank Adaptation) with rank $r=1$ or $r=4$ (swept as a hyperparameter). With $r=1$ , the number of trainable parameters is 0.53 million, which is 0.015% of Llama-7b's total parameters. With $r=4$ , it is 2.25 million parameters.
Overall training epochs: 4 epochs on the offline RL dataset.

Why LoRA and quantization: Fine-tuning the full 7-billion-parameter model would require substantial GPU memory and compute. By using 4-bit quantization and LoRA adapters, the entire training pipeline can run on a single A100 40GB GPU (as stated in Appendix C.1). This makes Retroformer practical for researchers and practitioners without access to massive compute clusters—the trainable component is two to three orders of magnitude smaller than the actor LLM it is optimizing prompts for.

Online execution with best-of-n sampling. During evaluation (online execution), the paper uses a best-of-n sampling strategy: the fine-tuned retrospective model generates multiple candidate reflection responses (at non-zero temperature), the learned reward model scores each candidate, and the highest-scoring reflection is selected to append to the actor's prompt for the next trial. This provides an additional quality improvement at inference time without further training.

Design Choices and Their Justifications

The paper makes several deliberate design choices that together form the Retroformer framework:

Choice 1: Fine-tune the retrospective model, not the actor. This is the central architectural decision. It is justified by the practical constraint that cloud-hosted LLMs have inaccessible parameters (Section 4), but the paper also argues it has conceptual benefits: the retrospective model can be trained on environment-specific data while the actor retains its broad pre-trained knowledge, and the modular separation means the same retrospective model could in principle be plugged into different actor LLMs without retraining.

Choice 2: Use episode return differences as ratings rather than absolute returns or human labels. This differential rating scheme (Equation 5) provides a causal measure of reflection utility that is environment-agnostic and requires no human annotation. It is made possible by the deterministic actor (T=0), which isolates the reflection as the sole cause of behavioral change between trials.

Choice 3: Use a three-stage offline RL pipeline (data collection, reward model, PPO) rather than online RL. Offline training avoids the cost and complexity of running RL rollouts online, which would require continuous interaction with the environment and the cloud-hosted actor API during training. The reward model smooths noisy differential ratings into a stable learned reward function. The paper acknowledges that this approach inherits the standard offline RL limitations (distribution shift between the data collection policy and the learned policy).

Choice 4: Use PPO with a KL penalty rather than simpler policy gradient methods. The KL penalty prevents the fine-tuned model from diverging too far from its pre-trained language capabilities, which is important because reflection generation is an open-ended language task where degenerate policies (e.g., always outputting the same generic advice) could achieve spuriously high rewards.

Choice 5: LongChat-7b-16k as the retrospective model base. The 16,000-token context window is necessary because reflection prompts contain full episode trajectories, which can be thousands of tokens long for multi-step tasks. A standard 2,048-token context model would truncate important trajectory information.

Choice 6: Pairwise comparison data for reward model training (Algorithm 1). By generating two alternative reflections and comparing their effects, the paper creates a preference dataset that is more robust than absolute ratings. This mirrors the preference-based approach used in RLHF and DPO, adapted to the agent setting.

4. Key Insights and Innovations

Innovation 1: Reframing the Frozen Actor as Part of the Environment — A Conceptual Bridge Between Language Agents and RL

The paper makes a fundamental conceptual move that redefines how we think about optimizing language agents: it treats the frozen, cloud-hosted actor LLM not as the agent being trained, but as a component of the environment dynamics. This is not merely an implementation trick to work around inaccessible model parameters—it is a reframing that dissolves a previously intractable problem.

Prior assumption and why it was limiting. The dominant assumption in the language agent literature—implicit in ReAct (Yao et al., 2023), Reflexion (Shinn et al., 2023), and related work—was that if you want an agent to improve from environment feedback, you must somehow optimize the component that generates actions: the actor LLM itself. This creates a dead end because (a) the most capable LLMs are cloud-hosted with inaccessible parameters, (b) even if accessible, they are prohibitively large to fine-tune with RL on modest hardware, and (c) standard RLHF assumes parameter access that doesn't exist in an API-only setting. The Reflexion approach sidesteps this by keeping everything frozen and using verbal feedback—but as the paper demonstrates in Figure 1, verbal feedback from a frozen LLM can be uninformative or actively harmful because the frozen model is not optimized for credit assignment in specific environments.

What the reframing achieves. By composing the frozen actor $\mathcal{L}$ with the environment transition $\mathcal{T}$ into a new transition function $\mathcal{T}' = \mathcal{T}(S, \cdot) \circ \mathcal{L} : \mathcal{S} \times \mathcal{X} \to \mathcal{S}$ , the paper reveals that optimizing a language agent with inaccessible actor parameters is mathematically equivalent to a standard RL problem where the policy is the retrospective model, the action space is the space of prompts, and the environment includes the frozen LLM as a deterministic (or low-temperature) transformation. This is stated explicitly in Section 3: "Since the LLM action agent is frozen, it can be considered as part of the environment."

This is not an incremental refinement—it is a fundamental shift in perspective that opens up the entire RL toolbox (policy gradients, PPO, reward shaping, replay buffers) for language agent optimization, without ever needing to touch the actor's parameters. The significance extends beyond the specific Retroformer architecture: any component in a language agent system that influences the actor's behavior through its prompt can now be viewed as a policy operating in an environment that includes the frozen LLM, and can be optimized with standard RL methods. The paper explicitly notes this generality in Section 6: "our approach is not limited to enhancing the retrospective model alone; it can be applied to fine-tune other components within the agent system architecture, such as the memory and summarization module, or the actor prompt."

Evidence. The mathematical formulation in Equations (1) and (2) provides the formal grounding, and the experimental results—where a 0.53M-parameter LoRA adapter (0.015% of the retrospective model's parameters) produces substantial improvements over frozen Reflexion baselines (Table 2: +11% HotPotQA, +15% AlfWorld with GPT-4 at N=1)—validate that optimizing through this composited environment is not just theoretically sound but practically effective.

Innovation 2: Differential Episode Returns as a Causal Reward Signal for Open-Ended Language Generation

The paper's second conceptual contribution is the response rating scheme in Equation (5): $r(x_{k,i}, y_{k,i}) \triangleq G_{k,i+1} - G_{k,i}$ . While this formula appears simple, it embodies a diagnostic breakthrough that converts the amorphous problem of "generate good advice" into a well-defined causal optimization problem.

The difficulty this solves. Training a model to generate high-quality self-reflections faces a fundamental challenge: what is the ground-truth label for "good advice"? Human annotation is expensive and subjective—two annotators might disagree on whether a reflection correctly diagnoses a failure. Even worse, a reflection can be factually accurate about what went wrong (good credit assignment) but still fail to produce behavioral improvement (poor actionable insight). Conversely, a reflection that is somewhat imprecise might still steer the agent toward a correct answer. The absolute quality of a reflection text is not what we care about—we care about its causal effect on subsequent agent behavior. But measuring causal effects of open-ended language generation has no standard methodology in the language agent literature.

How the differential rating solves it. By exploiting two properties of the Retroformer setup—(1) the actor LLM is frozen and deterministic (T=0), and (2) the only change between consecutive trials on the same task is the appended reflection—the paper isolates the reflection's causal effect as the difference in returns. If $\Delta G$ is positive, the reflection helped; if negative, it hurt. This transforms reflection evaluation from a subjective language quality judgment into an objective behavioral measurement.

This is a fundamental conceptual contribution rather than an incremental improvement because it establishes a general principle: when a frozen deterministic policy's input is augmented with generated text between episodes, the performance delta serves as a valid causal reward signal for the text generator, without requiring any ground-truth labels for what the text "should" say. This principle could apply beyond self-reflections to any setting where a frozen model's behavior can be influenced by generated context and outcomes can be measured.

Why prior approaches couldn't do this. Reflexion (Shinn et al., 2023) and Self-Refine (Madaan et al., 2023b) use the generated reflection directly without evaluating its causal effect—they assume the frozen LLM's reflection is useful by default. RAP (Hao et al., 2023) uses environment rewards for search but does not use performance improvement as a training signal for a reflection generator. The differential rating scheme is novel in the language agent literature because it creates a feedback loop where the consequences of generated text, not the text itself, determine its training signal.

Evidence. The rating scheme's validity rests on the deterministic actor assumption—acknowledged by the paper: "Because the actor is a frozen LM and the temperature is low as default, the injected randomness that leads to differences in returns $\Delta G_{k,i} = G_{k,i+1} - G_{k,i}$ are mostly from the reflection responses $y_{k,i}$ ." The practical success of PPO training using these ratings (Table 2 improvements) provides indirect validation. The pairwise comparison protocol in Algorithm 1 (generating two reflections, comparing their $\Delta G$ , labeling accepted/rejected) further strengthens the signal by converting noisy absolute differences into robust preference pairs—an approach that echoes the preference-based methodology from RLHF (Ouyang et al., 2022) but applied to causal effect estimation rather than human judgment.

Innovation 3: The Bottleneck Is Credit Assignment, Not Action Generation — Refocusing the Agent Learning Problem

The paper's diagnostic framing of the language agent learning problem represents a third conceptual contribution: it identifies that the critical failure mode in self-reflective agents is not in planning or action execution, but in credit assignment—the ability to correctly identify which specific action in a multi-step trajectory caused the failure. This refocuses the research agenda from improving the actor (the dominant prior approach) to improving the diagnostician.

The diagnostic evidence. Figure 1 provides the paper's clearest articulation of this insight. The agent's failure on the Teen Titans spinoff question is not due to a lack of search capability or reasoning ability—the actor successfully found the relevant information (Tara Strong's filmography, her connection to Lollipop Chainsaw and Teen Titans). The failure is purely in credit assignment: the agent "forgot its original goal during a chain of lengthy interactions" and submitted an answer ("Teen Titans and Teen Titans Go!") that includes the original series alongside the spinoff, even though only the spinoff was asked. The frozen LLM's self-reflection completely misses this diagnosis—it recommends actions the agent already performed, demonstrating that general-purpose LLMs are poor at environment-specific credit assignment even when the environment is described in natural language.

The reinforced retrospective model (Figure 5), by contrast, correctly identifies the root cause ("I failed to find the Teen Titans spinoff series... I should have searched Tara Strong and Teen Titans or Teen Titans Go! specifically") and proposes the corrective focus ("focus on spinoff series and find the answer in the previous actions and observations with the search results of Tara Strong"). This is not a difference in language quality—it's a difference in diagnostic accuracy that gradient-based training on environment-specific trajectories enables.

Why this reframes the problem. Prior work on language agent improvement has largely focused on two directions: (1) better prompting strategies for the actor (chain-of-thought, ReAct, few-shot demonstrations) that improve the generation of actions, and (2) better search strategies (RAP's Monte Carlo Tree Search) that explore the space of possible actions. Both assume the bottleneck is in action generation or selection. Retroformer's analysis suggests a different bottleneck: the actor is often capable of producing the right answer (it has the necessary knowledge and reasoning ability), but it needs to be steered away from specific failure modes that it cannot self-diagnose. The retrospective model's job is therefore not to plan better than the actor, but to assign credit more accurately than the actor can introspect.

This is a fundamental reframing because it implies that investment should go into training environment-specific diagnosticians (via gradient-based RL) rather than into better general-purpose prompting or more sophisticated search algorithms—at least for the class of tasks where the actor possesses the underlying capability but fails due to specific, identifiable error patterns.

Evidence. The paper provides multiple examples in Figures 5 and 7 where the reinforced retrospective model's reflections demonstrate concrete credit assignment that the frozen model misses—not just in HotPotQA but in the AlfWorld environment as well (Figure 7: the reinforced model correctly identifies that "I was stuck in a loop in which I continually examined stoveburner 1 instead of heating mug 1 with stoveburner 1"). The quantitative results in Table 2 show that improving credit assignment (via fine-tuning the retrospective model) yields substantial performance gains even though the actor LLM is identical across all conditions—confirming that better diagnosis, not better action generation, is the operative mechanism.

Innovation 4: Emergent Structured Reflection — Evidence That RL Fine-Tuning Produces Qualitatively Different (Not Just Better-Scored) Outputs

The paper reports a striking empirical finding in Appendix D: the reinforced retrospective model spontaneously organizes its output into structured sections ("Reflection:" and "New plan:") even though it was never explicitly trained or prompted to produce this format. This emergent behavior is not merely a curiosity—it provides evidence that RL fine-tuning produces qualitatively different linguistic behavior, not just quantitatively higher-scoring text under some metric.

Why this matters conceptually. A skeptical interpretation of Retroformer's results would be: "the PPO fine-tuning is just making the reflections slightly better-written versions of what the frozen model would produce anyway—cosmetic improvements that happen to correlate with higher returns." The emergent structure contradicts this interpretation. The frozen model's reflections, shown in Figures 1, 5, and 7, are unstructured paragraphs that interleave diagnosis, proposed actions, and sometimes hallucinated content (e.g., "Next trial: Question:..."). The reinforced model's reflections consistently separate the retrospective diagnosis from the prospective action plan—a format that is demonstrably more useful for the actor to parse and act upon. The paper states this explicitly: "The paraphrased response retrospects in the first paragraph and provides actionable insights next, while the response from the frozen LM interleaved both parts in one paragraph, making it hard to comprehend."

This is a significant empirical insight because it suggests that gradient-based optimization of language models for environment-specific rewards can induce functional linguistic adaptations that go beyond what supervised fine-tuning on human-written text would produce. The model is not just learning to say the right things—it's learning to structure its communication in ways that are more effective for its specific audience (the actor LLM) and task (improving subsequent performance). This connects to broader questions about whether RL fine-tuning can induce capabilities (like strategic communication formatting) that emerge from the optimization pressure itself rather than from mimicking training data.

Connection to the RLHF literature. This emergent structuring echoes findings from the RLHF literature where models fine-tuned with PPO develop systematic behaviors (e.g., format adherence, refusal patterns) not explicitly present in the supervised fine-tuning data. However, in Retroformer's case, the emergent behavior is directly linked to a measurable performance improvement—the structured format likely contributes to the reflection's causal efficacy—rather than being a side effect of aligning with human preferences. This makes it a cleaner demonstration of functional adaptation through reward optimization.

Evidence. The paper shows explicit side-by-side comparisons: Figure 5 contrasts the structured reinforced response (with clear diagnosis and corrective plan) against the unstructured frozen response, and Figure 7 shows the same phenomenon in the AlfWorld environment. The fact that this emerges consistently across different environments (HotPotQA and AlfWorld) and without any formatting instructions suggests it is a robust consequence of the PPO optimization, not an artifact of a particular prompt or environment.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses three open-source benchmarks: HotPotQA (Yang et al., 2018) for search-based multi-hop question answering, AlfWorld (Shridhar et al., 2021) for embodied robotics tasks executed via text commands, and WebShop (Yao et al., 2022) for web browsing and product purchase. For HotPotQA, agents are evaluated on 100 validation tasks from the distractor dev split, following the setup in Shinn et al. (2023). For AlfWorld, 134 validation tasks across six task types (finding hidden objects, moving objects, manipulating objects with other objects, etc.) are used. For WebShop, 100 validation tasks are used. Training data for offline RL collection uses 3,000 HotPotQA training tasks, plus the full training splits for AlfWorld and WebShop. The choice of three diverse environments spanning question answering, physical interaction, and web navigation is deliberate: it tests whether the retrospective model's learned credit assignment ability generalizes across fundamentally different action spaces, reward structures, and failure modes.
Base model(s). The frozen actor model is GPT-3 (text-davinci-003) and GPT-4, accessed via API. These are chosen because they represent the state-of-the-art in cloud-hosted LLMs whose parameters are inaccessible—precisely the constraint that motivates Retroformer's architecture of optimizing a separate retrospective model rather than fine-tuning the actor. The retrospective model is instantiated from LongChat-7b-16k, a fine-tuned version of Llama-7b with a 16,000-token context window, trained on instruction-following conversations from ShareGPT. This model is small enough to fine-tune on a single A100 40GB GPU (a deliberate practical constraint) while having the extended context length needed to process full episode trajectories as reflection prompts. The temperature of the actor LLM is set to zero (T=0, top-p=1) across all experiments to isolate the effect of reflections from random action sampling—any behavioral change between consecutive trials is attributable solely to the appended self-reflection.
Metrics. The primary metric is success rate over validation tasks in each environment, reported as a percentage. For HotPotQA, success is determined by F1 score matching between the generated answer and ground truth (though the paper uses exact match for the binary success/failure reporting in the main results). For AlfWorld, success is binary: the agent either completes the specified task (e.g., "heat some mug and put it in coffeemachine") or fails. For WebShop, success is based on the composite shopping reward defined in Equation (7) in Appendix C.3, which considers product type matching, attribute overlap, option overlap, and budget compliance—the paper reports the percentage of episodes where the purchased product matches the target specification. The paper also reports results broken out by trial number (episode ID), showing how success rate improves as the agent accumulates self-reflections across retry attempts on the same task.
Baselines. Three baselines are compared (Table 2). (1) ReAct (Yao et al., 2023): the base frozen language agent architecture with no learning from environment rewards—the agent generates actions via reasoning-and-acting interleaving but receives no feedback from prior attempts and does not improve over retries. This serves as a lower bound showing agent performance without any environment feedback. (2) Reflexion (Shinn et al., 2023): the state-of-the-art verbal reinforcement agent that uses a frozen LLM to generate self-reflections after failures, appending them to the actor prompt for subsequent attempts. This is the most direct comparison—it uses the same information (trajectory history) as Retroformer but applies no gradient-based optimization to the reflection generator. (3) Soft Actor-Critic (SAC) (Haarnoja et al., 2018): an online RL baseline that applies traditional continuous-action RL to text-based games. The paper uses mean-pooled embeddings of generated text outputs (e.g., Search[It Takes a Family] → 768-dimensional embedding) as actions in a continuous action space, with LoRA adapters (r=4) on the actor model instantiated from LongChat-16k. SAC is trained online with discount factor γ=0.99, polyak averaging coefficient 0.995, learning rate 0.01, entropy regularization α=0.2, and batch size 8. This baseline tests whether standard continuous RL can achieve comparable improvements without Retroformer's verbal reflection mechanism.
Generation budget / compute accounting. The paper does not measure compute in FLOPs or GPU-hours for the main comparisons. Instead, the relevant budget dimension is number of retry attempts (reported as N=1 or N=4 in Table 2), representing how many times the agent is allowed to attempt the same task with accumulated reflections. All methods within a column use the same number of retries, making the comparison fair in terms of environment interaction budget. The SAC baseline uses N=1 or N=4 episodes but learns parameters online during those episodes. For Retroformer, the training compute is separate from evaluation compute—the retrospective model is fine-tuned offline on pre-collected data (~3,383 HotPotQA samples, ~523 AlfWorld samples, ~267 WebShop samples) using a single A100 40GB GPU, and then evaluated with the same number of retries as baselines. The key comparison is therefore: given equal numbers of environment interactions (retries), does a retrospective model trained offline with gradient-based RL produce better self-reflections than a frozen LLM prompted to self-reflect? The paper does not report the FLOPs or wall-clock time for offline training, which is a limitation for assessing total efficiency.
Cross-validation / statistical protocol. No explicit cross-validation or statistical significance testing is reported. The paper evaluates on fixed validation splits (100 HotPotQA tasks, 134 AlfWorld tasks, 100 WebShop tasks) and reports aggregate success rates without confidence intervals. This means the reported differences (e.g., 54% vs. 50% on HotPotQA with N=4) should be interpreted with appropriate caution—on a test set of 100 questions, a 4-percentage-point difference could arise from 4 additional correct answers, which may or may not be statistically significant. The paper does not discuss whether results were averaged over multiple random seeds or whether the offline RL training procedure (which involves stochastic gradient updates) was run multiple times to assess variance. This is a methodological weakness that the Reflexion baseline paper partially addresses and that limits the strength of conclusions that can be drawn from small numerical margins.

Main Quantitative Results

Aggregate Performance Across All Three Environments

Table 2 provides the primary head-to-head comparison across all environments, actor models, and retry budgets. The headline results are:

HotPotQA (GPT-4, N=4): Retroformer achieves 54% success rate with both LoRA ranks (r=1 and r=4), compared to Reflexion's 52% and ReAct's 40%. With GPT-3 (N=4), Retroformer achieves 53–54% vs. Reflexion's 50% and ReAct's 34%. The improvement over Reflexion is 2–4 percentage points at N=4, but the more dramatic gains appear in early trials: at N=1 (single attempt with one prior reflection), Retroformer achieves 48–51% (GPT-4) vs. Reflexion's 46% and ReAct's 40%, representing a 2–5 point gain.
AlfWorld (GPT-4, N=3): Retroformer solves the environment completely—100% success rate with both r=1 and r=4, compared to Reflexion's 85.07% and ReAct's 77.61%. With GPT-3 (N=3), the gap is even larger: Retroformer achieves 100% vs. Reflexion's 84.33% and ReAct's 62.69%. At N=1 (single attempt), Retroformer with GPT-3 achieves 93.28–97.76% vs. Reflexion's 76.87%—a 16–21 percentage point improvement that suggests the reinforced reflections are qualitatively better at preventing the specific failure modes (infinite loops, wrong object interactions) that plague the frozen Reflexion agent in AlfWorld.
WebShop (GPT-4, N=4): Retroformer achieves 45–46% success rate vs. Reflexion's 44% and ReAct's 42%. The improvement is modest—2–4 percentage points—consistent with the pattern observed in Shinn et al. (2023) that WebShop is less amenable to verbal feedback approaches due to the need for precise search queries and extensive exploration. At N=1, Retroformer achieves 43% vs. Reflexion's 42% and ReAct's 42%—essentially no improvement, suggesting that a single self-reflection does not meaningfully change the agent's web browsing behavior.
SAC baseline: The online RL baseline performs poorly across all environments. In HotPotQA, SAC achieves 27% regardless of N, substantially below even ReAct (34–40%). In AlfWorld, SAC achieves 58.95–59.7%, below ReAct (62.69%) and dramatically below Retroformer (93–100%). In WebShop, SAC achieves 30%, slightly below ReAct (33–42%). These results confirm that continuous-action RL applied to text embeddings is not competitive with language-native approaches for these environments—the credit assignment challenge requires the semantic understanding that verbal reflection provides.

What these numbers establish: Across all three environments, Retroformer with gradient-based fine-tuning of the retrospective model consistently outperforms Reflexion, which uses the same frozen actor LLM and the same information (trajectory history) but generates reflections from a frozen LLM. The performance gaps are largest in AlfWorld (where the frozen Reflexion agent suffers from persistent failure modes like action loops that the reinforced reflections specifically address) and smallest in WebShop (where the underlying challenge may be more about exploration than credit assignment). The SAC results demonstrate that the improvement is not simply from "applying RL" to the problem—the specific combination of verbal reflection with gradient-based optimization is what matters.

Learning Speed: Faster Improvement in Early Trials

Figure 4 (HotPotQA) and Figure 6 (AlfWorld, WebShop) plot success rate as a function of episode ID (trial number), revealing the learning trajectory within a task as the agent accumulates self-reflections. The key pattern is that Retroformer's advantage over Reflexion is most pronounced in early trials—the agent learns faster because its first reflection is already of higher quality.

In HotPotQA (Figure 4), the Retroformer (GPT-4, r=4) curve starts at approximately 51% at trial 1 (after a single reflection) and rises to approximately 54% by trial 4. Reflexion (GPT-4) starts at approximately 46% and rises to approximately 52%. The gap at trial 1 is roughly 5 percentage points; by trial 4 it narrows to roughly 2 points. This pattern suggests that the retrospective model's fine-tuned reflections provide an immediate advantage (better diagnosis on the first failure), but that further reflections add incrementally similar value for both methods. The ReAct baseline is flat at approximately 40% across all episodes (no learning occurs, as expected since it does not use prior trial feedback).

In AlfWorld (Figure 6a), the pattern is more dramatic: Retroformer with GPT-3 and r=4 rises from approximately 95% at trial 1 to 100% by trial 3, while Reflexion (GPT-3) rises from approximately 76% to only approximately 84% after 3 trials. The Retroformer curve is nearly saturated after a single reflection, while Reflexion shows a slower, incomplete climb. With GPT-4, both Retroformer curves (r=1 and r=4) reach 100% by trial 2, while Reflexion plateaus at approximately 85%. This demonstrates that the reinforced retrospective model's reflections are not just "better" in a generic sense but specifically target the failure modes (e.g., stuck in loops, examining instead of using objects) that prevent Reflexion from solving the remaining ~15–24% of tasks.

In WebShop (Figure 6b), the learning curves are nearly flat for all methods—Retroformer (GPT-4) hovers around 43–46% across trials, Reflexion around 42–44%, and ReAct around 42%. The minimal slope indicates that self-reflections, whether from frozen or fine-tuned models, provide limited benefit for web browsing tasks. As the paper acknowledges, "this limitation was also observed in (Shinn et al., 2023) as web browsing requires a significant amount of exploration with more precise search queries." The flat curves suggest that the failure modes in WebShop (imprecise search queries, difficulty navigating product pages) may require fundamentally different interventions than post-hoc verbal diagnosis.

Effect of LoRA Rank on Retrospective Model Capacity

Table 2 reports Retroformer with two LoRA configurations: r=1 (0.53M trainable parameters) and r=4 (2.25M trainable parameters). The comparison serves as a rough capacity ablation:

HotPotQA (GPT-4, N=4): r=1 achieves 53%, r=4 achieves 54%—essentially identical. With GPT-3, r=1 achieves 53%, r=4 achieves 54%—again nearly identical.
AlfWorld (GPT-4, N=1): r=1 achieves 95.62%, r=4 achieves 97.76%—a small advantage for higher rank. With GPT-3, r=1 achieves 93.28%, r=4 achieves 97.76%—a 4.5-point gap that narrows to zero at N=3 (both 100%).
WebShop (GPT-4, N=4): r=1 achieves 36%, r=4 achieves 36% at N=1; r=1 achieves 45%, r=4 achieves 46% at N=4—marginal differences.

The pattern suggests that the credit assignment task—diagnosing failure root causes and proposing corrective plans—can be learned with very few parameters (0.53M, representing 0.015% of the base Llama-7b model). The marginal benefit of quadrupling the LoRA rank is small and inconsistent, appearing primarily in AlfWorld with GPT-3 at N=1, where the additional capacity may help the retrospective model better understand the diverse set of possible action sequences in the physical environment. The paper does not explore whether further increasing the rank (e.g., full fine-tuning) would yield additional gains, which leaves open the question of whether the LoRA bottleneck limits the retrospective model's ability to capture more complex credit assignment patterns.

Effect of Actor Model Scale (GPT-3 vs. GPT-4)

Table 2 disaggregates results by both the actor LLM and the retrospective model configuration, enabling analysis of how the stronger base actor interacts with the reinforced reflections:

HotPotQA: With GPT-3 actor, Retroformer (r=4) achieves 54% at N=4; with GPT-4, also 54%. The ceiling appears to be the same regardless of actor capability. At N=1, GPT-4 with Retroformer (r=4) achieves 51% vs. GPT-3's 48%—a 3-point gap that disappears with more retries. Reflexion shows a similar pattern: GPT-4 (52%) outperforms GPT-3 (50%) by 2 points at N=4. The 14-point gap between ReAct GPT-3 (34%) and ReAct GPT-4 (40%) suggests GPT-4 is a substantially stronger zero-shot reasoner, but the benefit of reflections (frozen or fine-tuned) compresses this gap—the agent with GPT-3 plus good reflections nearly matches the agent with GPT-4 plus mediocre reflections.
AlfWorld: The actor scale matters more. ReAct GPT-3 achieves 62.69% vs. GPT-4's 77.61%—a 15-point gap. Reflexion narrows this: GPT-3 achieves 84.33% vs. GPT-4's 85.07% (at N=3), nearly closing the gap. Retroformer eliminates it entirely: both GPT-3 and GPT-4 achieve 100% at N=3, and at N=1, GPT-4 with r=1 (95.62%) is only slightly ahead of GPT-3 with r=4 (97.76%—actually higher, suggesting the rank matters more than actor scale at this point). This implies that in AlfWorld, the primary bottleneck for GPT-3 is not reasoning capability but susceptibility to specific failure modes (action loops, incorrect object targeting) that good reflections can correct—once those are addressed, GPT-3 and GPT-4 are equally capable of completing the relatively constrained physical tasks.
WebShop: GPT-4 with ReAct (42%) significantly outperforms GPT-3 (33%)—a 9-point gap that persists across all methods. Retroformer GPT-4 (45–46%) and Reflexion GPT-4 (44%) maintain a lead over Retroformer GPT-3 (36%) and Reflexion GPT-3 (35%). Unlike AlfWorld, the stronger actor provides a persistent advantage in web browsing—reflections cannot close the capability gap. This is consistent with the interpretation that WebShop failures stem from exploration and search precision challenges that reflections do not address, rather than from identifiable credit assignment errors.

Environment-Specific Analysis

HotPotQA: The reinforced retrospective model provides better credit assignment and actionable insights. Figure 5 provides the qualitative evidence supporting the quantitative gains in Table 2. The specific example—the "Teen Titans spinoff series" question—demonstrates the failure mode that Retroformer addresses: the agent found the correct information (Tara Strong's filmography, her role in Lollipop Chainsaw, her voice work for Teen Titans and Teen Titans Go!) but submitted an answer that includes both "Teen Titans" and "Teen Titans Go!" when only the spinoff ("Teen Titans Go!") was asked. The frozen model's reflection fails to identify this specific error—it recommends searching for Lollipop Chainsaw and Tara Strong's filmography, which the agent already did—and merely rephrases the prior action sequence as a proposed plan. The reinforced model's reflection correctly identifies the root cause ("I failed to find the Teen Titans spinoff series... I should have searched Tara Strong and Teen Titans or Teen Titans Go! specifically") and proposes a focused corrective action ("focus on spinoff series and find the answer in the previous actions and observations with the search results of Tara Strong"). This is not a generic improvement in language quality—it is a specific improvement in credit assignment that directly addresses the actual failure mechanism.

The broader implication is that the HotPotQA improvement (from 50% to 54% with GPT-3 at N=4) likely comes from addressing a particular class of failures: multi-hop reasoning tasks where the agent successfully retrieves all relevant information but fails to synthesize it correctly—submitting answers that are too broad (both original and spinoff), too narrow (missing one component of a compound answer), or factually incorrect due to misreading a retrieved passage. The reinforced reflections diagnose these specific synthesis errors, while frozen reflections provide generic re-planning advice that does not target the actual mistake.

AlfWorld: The reinforced retrospective model prevents repetitive action loops. The AlfWorld example in Appendix E.1 and Figure 7 reveals the primary failure mode in this environment: the agent enters an infinite action loop where it repeatedly issues the same ineffective command (e.g., examining a stoveburner four times instead of heating the mug on it). This is a classic credit assignment problem: the agent does not realize that examine does not progress toward the goal, and without feedback, it continues the loop until the maximum number of actions is exhausted.

The frozen model's reflection (shown in the AlfWorld prompt example in Appendix E.1) identifies the problem—"I was stuck in a loop in which I continually examined stoveburner 1 instead of heating mug 1 with stoveburner 1"—and proposes the correct corrective action. However, the fact that Retroformer with fine-tuned reflections still substantially outperforms Reflexion (100% vs. 84.33% with GPT-3 at N=3) suggests that the frozen model sometimes fails to identify these loops, or identifies them imprecisely (e.g., recommending "try a different action" without specifying which action), or fails to generate reflections that actually prevent loop recurrence. The fine-tuned retrospective model, trained on AlfWorld-specific trajectories with reward signals, learns to consistently detect loops, identify which specific action should have been taken instead, and phrase the reflection in a way that the actor LLM reliably follows.

The scale of improvement—from 84.33% to 100% with GPT-3, and from 85.07% to 100% with GPT-4—indicates that the remaining ~15% of AlfWorld tasks that Reflexion cannot solve are precisely those where loop detection and correction fail. Retroformer's retrospective model, having been trained on loop-containing trajectories with positive and negative ratings, has learned to generate reflections that preempt these loops in all cases.

WebShop: Verbal reflection is fundamentally limited for web browsing. The minimal improvement across all methods in WebShop—Retroformer (45–46%) vs. Reflexion (44%) vs. ReAct (42%) with GPT-4 at N=4, and Retroformer (36%) vs. Reflexion (35%) vs. ReAct (33%) with GPT-3—indicates a ceiling on what post-hoc verbal diagnosis can achieve in this environment. The paper hypothesizes that "web browsing requires a significant amount of exploration with more precise search queries" (Section 5.3), and the experimental data supports this: even the best reflections do not substantially change the agent's search behavior. Unlike HotPotQA, where the error is often a specific synthesis mistake that a reflection can directly address ("search for spinoff, not original"), or AlfWorld, where the error is an identifiable action loop ("use stoveburner, don't examine it"), WebShop failures may be more diffuse—poor initial search queries that return irrelevant products, difficulty navigating product options, or price/budget mismatches that a reflection cannot correct because the underlying search problem has not been solved.

The SAC baseline's poor performance (30%) further suggests that WebShop's challenge is not primarily about credit assignment or learning from rewards in the way Retroformer addresses it—even with online RL that updates action representations continuously, the agent cannot substantially improve. This is a valuable negative result: it delineates the boundary of where verbal feedback approaches (both frozen and reinforced) are effective. The paper does not explore alternative WebShop-specific interventions (e.g., training a search query refinement module) that might address the exploration bottleneck more directly than post-hoc trajectory diagnosis.

Ablation Studies and Robustness Checks

LoRA rank (r=1 vs. r=4): Table 2 shows that increasing the number of trainable parameters from 0.53M (r=1) to 2.25M (r=4) produces small and inconsistent improvements. In HotPotQA with GPT-4 at N=4, r=1 achieves 53% vs. r=4's 54%—a 1-point gain. In AlfWorld with GPT-3 at N=1, r=1 achieves 93.28% vs. r=4's 97.76%—a 4.5-point gain that is the largest observed difference. In WebShop, there is essentially no difference. This suggests that the credit assignment task learned by the retrospective model is relatively low-dimensional—identifying failure patterns (loops, overbroad answers, forgotten goals) and proposing corrective actions does not require the full representational capacity of a 7B-parameter language model. The slightly larger gains with r=4 in AlfWorld may reflect the greater diversity of action sequences in the physical environment (six task types with different object and location vocabularies) requiring marginally more capacity to encode. The paper does not report results for higher ranks (e.g., r=8, r=16) or full fine-tuning, leaving open whether a capacity ceiling exists above r=4.

Actor temperature (T=0 for all methods): Appendix C.1 notes that all experiments use temperature zero for the actor LLM, stating that "setting a higher temperature value can encourage exploration but it can obscure the impact of the proposed approaches, making it difficult to compare against existing baselines with T=0." This is a deliberate design choice rather than an ablation, but it has important implications: the deterministic actor ensures that any behavioral change between trials is caused by the appended reflection, not by random sampling variation. A natural ablation—not performed in the paper—would test whether Retroformer's advantage over Reflexion increases or decreases with nonzero actor temperature. If the advantage is larger at higher temperatures (because good reflections steer a more stochastic actor toward correct actions), that would strengthen the case for Retroformer in real deployments where temperature >0 is standard. If the advantage persists similarly, it confirms that the reflection quality improvement is the operative mechanism independent of action sampling stochasticity.

ReST $^{EM}$ revision model: This ablation from Appendix K is described in the prior sections and is noted here for cross-reference. The attempt to optimize the revision model with ReST $^{EM}$ degraded sequential revision performance substantially, suggesting sensitivity to training methodology. This negative result reinforces the value of Retroformer's offline, rating-based approach over online policy iteration for the reflection component.

Critical Assessment

Does Retroformer demonstrate that gradient-based optimization of the reflection model improves over frozen verbal feedback?

The experimental evidence supports this claim, but with important qualifications about the magnitude and generality of the improvement. In HotPotQA, the improvement over Reflexion at N=4 is 2–4 percentage points (GPT-3: 54% vs. 50%; GPT-4: 54% vs. 52%). On a 100-question test set, this corresponds to 2–4 additional correct answers. The paper does not report confidence intervals or statistical tests, so it is not possible to assess whether this difference is statistically reliable or within the range of sampling variance. The improvement is more convincing in AlfWorld (100% vs. 84–85%, a 15–16 point gain that represents complete task saturation vs. incomplete performance) and least convincing in WebShop (1–2 point gain that could easily be noise). The claim of "gradient-based optimization improves over frozen verbal feedback" is therefore conditionally supported: strongly in environments where credit assignment errors are the dominant failure mode and where frozen reflections are unreliable (AlfWorld), moderately in environments where frozen reflections partially work and fine-tuning adds marginal value (HotPotQA), and weakly in environments where verbal feedback itself is of limited utility regardless of its quality (WebShop).

What is actually being measured? A critical subtlety: the comparison between Retroformer and Reflexion is not a pure test of "gradient-based optimization vs. frozen LLM." Retroformer also differs from Reflexion in using a separate, specialized model (LongChat-7b) for reflection rather than the same frozen LLM (GPT-3/4) used for action generation. The Retroformer retrospective model is prompted with trajectory-and-return information and few-shot examples of good reflections—a different architecture and prompt than Reflexion's self-reflection mechanism. The gradient-based optimization improves this already-different reflection generator, not the same reflection generator as Reflexion. An apples-to-apples comparison would train Retroformer's retrospective model without PPO (just supervised fine-tuning on positive-rating samples) and compare that to Reflexion—this would isolate the effect of using a separate reflection model vs. the same model, separate from the effect of gradient optimization. The paper does report this partially by noting that the frozen (pre-trained, un-fine-tuned) LongChat model produces the uninformative reflections in Figures 1 and 5, but the quantitative performance of this un-fine-tuned Retroformer variant is not reported in Table 2—the table only shows Retroformer with LoRA (which implies PPO fine-tuning) vs. Reflexion.

Does Retroformer demonstrate that learning transfers across tasks and environments?

The paper states that the replay buffer stores reflections "across tasks and environments" (Section 4.1) and that the agent "not only exploits lessons learned over failed trials in the current task, but also explores by learning from success in other related tasks." However, the training procedure in Appendix C.1 describes separate data collection for each environment: 3,383 HotPotQA reflections, 523 AlfWorld reflections, and 267 WebShop reflections. It does not state that the retrospective model is trained on a combined dataset from all three environments, nor does it report a cross-environment transfer experiment (e.g., training on HotPotQA and testing on AlfWorld). The reported results in Table 2 appear to be trained and evaluated within each environment separately. The claim of cross-task generalization is therefore not directly tested in the reported experiments—it is a conceptual property of the replay buffer architecture but is not empirically validated.

Does Retroformer solve the expensive difficulty estimation problem?

The paper does not use difficulty estimation—this observation from the reference example does not apply. However, Retroformer has an analogous cost that is not accounted for: the offline data collection stage requires running the base policy for three trials on thousands of training tasks, generating multiple reflection alternatives per task, and rolling out each alternative to compute differential ratings. For HotPotQA, this means ~3,000 tasks × 3 trials × (2 reflection alternatives × 1 rollout each) = up to 18,000 episodes of environment interaction just to collect training data. This cost is not included in any efficiency comparison with Reflexion (which requires zero training). The paper presents Retroformer as achieving higher success rates at the same number of evaluation retries, but the total cost (training + evaluation) is substantially higher. An honest accounting would compare Reflexion's zero-training deployment against Retroformer's training cost amortized over the number of tasks the agent will eventually solve—for a deployment that handles thousands of tasks, the training cost may be negligible; for a deployment handling a handful of tasks, the training cost may dominate and Retroformer would be less efficient. The paper does not perform this amortization analysis.

Missing experiments that would strengthen the paper:

Single-task fine-tuning ablation: Does the retrospective model need diverse training tasks, or could it achieve similar performance by fine-tuning only on reflections from the specific test tasks? If the latter, the replay buffer's cross-task sampling is not contributing to generalization, only to data quantity.
Reflection quality human evaluation: The paper provides qualitative examples (Figures 5, 7) showing that reinforced reflections are better, but no systematic human study rates reflection quality on dimensions like diagnostic accuracy, actionability, and clarity. Such a study would validate that the PPO optimization improves reflection content in ways that humans can perceive, not just in ways that produce higher episode returns.
Number of training tasks ablation: How many training tasks are needed? The paper uses 3,000 HotPotQA tasks—would 300 suffice? How does performance scale with training data? This is crucial for practical deployment where environment-specific training data may be scarce.
Cross-environment transfer: Train retrospective model on HotPotQA, evaluate on AlfWorld (or vice versa). Does the credit assignment skill transfer across environments with different action spaces and reward structures?
Ablation on the rating scheme: Compare the differential rating $G_{k,i+1} - G_{k,i}$ against alternative ratings: absolute return $G_{k,i+1}$ only, human-labeled reflection quality, or binary success of the next episode. This would validate the causal effect interpretation that the paper relies on.
Comparison with in-context learning instead of fine-tuning: Instead of PPO fine-tuning LongChat-7b, provide the frozen retrospective model with more few-shot examples of good reflections. At what point does in-context learning catch up to fine-tuning? This would help distinguish whether the benefit comes from having any training signal (which few-shot examples also provide) or specifically from gradient-based optimization.
Ablation on deterministic actor (T=0): Repeat key results with T=0.3, T=0.7, T=1.0 for the actor. Does the rating scheme remain valid when action sampling introduces noise unrelated to reflection quality? The paper's reliance on T=0 to isolate the reflection's causal effect is a significant practical limitation—most deployed agents use nonzero temperature for diversity.

Weaknesses in the experimental design:

Small test sets with no statistical reporting. The 100-question HotPotQA test set means each percentage point corresponds to a single question. A 4-point gap between Retroformer (54%) and Reflexion (50%) at N=4 represents 4 additional correct answers. Without confidence intervals, it's unclear whether this gap is statistically reliable or could arise from the particular 100 questions selected. The AlfWorld results (100% vs. 84–85%) are more convincing due to the large absolute gap and ceiling effect, but the WebShop results (1–2 point differences) are almost certainly not statistically significant.
The SAC baseline is weak and not well-tuned. SAC's poor performance (27% on HotPotQA vs. ReAct's 34%) suggests the embedding-based action representation and continuous RL approach is fundamentally mismatched to these environments, making SAC an uninformative baseline for "does RL help?" A stronger RL baseline would be one that uses the same retrospective model architecture as Retroformer but optimizes it with a different algorithm (e.g., REINFORCE, A2C, or online PPO without the offline data collection stage) to test whether the specific offline PPO + reward model pipeline is necessary.
No comparison with non-verbal RL approaches that are state-of-the-art for these environments. The paper compares only against Reflexion and ReAct, both language agent baselines. For AlfWorld, behavioral cloning or RL approaches that directly learn action policies from environment rewards exist (Shridhar et al., 2021) and may achieve comparable or better performance. The paper does not position Retroformer relative to these non-LLM baselines, nor does it discuss whether the verbal reflection approach offers advantages (interpretability, transfer) that compensate for potentially lower asymptotic performance.
The improvement in WebShop is not convincingly demonstrated. The 1–2 point gap between Retroformer and Reflexion could be noise. The paper's own acknowledgment that "the improvements may be limited" and that "verbal feedback approach... is not an optimal method for this environment" undercuts the claim that Retroformer "considerably outperforms baselines" (from the abstract). The abstract's language overclaims relative to the WebShop results specifically.
The training cost is substantial and unreported. The offline data collection involves thousands of environment interactions, and the three-stage training pipeline (supervised fine-tuning, reward model training, PPO) requires hyperparameter tuning and compute. The paper does not report the total GPU-hours or API costs for data collection. A practitioner reading the paper cannot easily assess whether the ~2-4% improvement in HotPotQA and the saturated AlfWorld performance justify this cost compared to simply using Reflexion (zero training) or spending the equivalent budget on more retry attempts with Reflexion (e.g., Reflexion with N=8 might match Retroformer with N=4 at lower total cost).

What the experiments genuinely demonstrate: The AlfWorld results provide compelling evidence that fine-tuning a retrospective model on environment-specific trajectories with RL can solve credit assignment problems that a frozen LLM cannot consistently handle—the jump from 84% to 100% and the qualitative examples of loop-breaking reflections are convincing. The HotPotQA results suggest a moderate but real improvement that may be practically meaningful if the training cost can be amortized over many tasks. The WebShop results demonstrate the boundary condition: when the task itself requires capabilities that verbal diagnosis cannot improve (precise search queries, exploration of product space), Retroformer provides minimal benefit over Reflexion, and neither verbal feedback approach substantially outperforms the no-feedback ReAct baseline. This boundary characterization—though not emphasized by the authors—is one of the paper's most valuable empirical contributions because it delineates where future work should focus: on exploration mechanisms for WebShop-like tasks rather than credit assignment mechanisms for HotPotQA/AlfWorld-like tasks.

6. Limitations and Trade-offs

The Training Data Collection Cost Is Substantial and Unaccounted For

The assumption or constraint. Retroformer's three-stage training pipeline (offline data collection → reward model learning → PPO fine-tuning) requires generating thousands of reflection samples by running the base policy for multiple trials on training tasks, then rolling out alternative reflections to compute differential ratings. The paper reports collecting 3,383 HotPotQA samples, 523 AlfWorld samples, and 267 WebShop samples (Appendix C.1). The data collection protocol in Algorithm 1 specifies that for each unsuccessful task, the retrospective model generates two alternative reflection responses (at temperature 0.9), and each alternative is rolled out for one additional episode to compute $\Delta G_{k,i} = G_{k,i+1} - G_{k,i}$ . This means that beyond the initial rollout (which all methods require), Retroformer's training demands roughly two additional episodes per unsuccessful training task just to generate pairwise comparison data, on top of the initial 3-trial rollouts used to collect base trajectories. The paper does not report the total number of environment interactions, API calls, or GPU-hours consumed during training.

The consequence. The headline comparison in Table 2—"Retroformer achieves 54% vs. Reflexion's 50% at N=4 on HotPotQA"—reports evaluation performance given equal evaluation retries, but completely ignores the training cost that Retroformer incurs and Reflexion does not. Reflexion requires zero training: it uses the frozen GPT-3/4 model for both action generation and reflection, with no data collection, no reward model training, and no PPO fine-tuning. A practitioner deciding whether to deploy Retroformer over Reflexion therefore faces an unmeasured tradeoff: is the 2–4 percentage point improvement in HotPotQA (or the 15–16 point improvement in AlfWorld) worth thousands of additional environment interactions during training? For a deployment that handles millions of tasks, the amortized training cost per task may be negligible. For a deployment handling a few hundred tasks, the training cost may exceed the cost of simply giving Reflexion more evaluation retries—for instance, Reflexion with N=8 might achieve comparable or better performance than Retroformer with N=4 at lower total environment interaction cost. The paper provides no analysis to help practitioners make this judgment.

What evidence exists in the paper. The paper reports the number of reflection samples collected (3,383 for HotPotQA, 523 for AlfWorld, 267 for WebShop in Appendix C.1) but does not convert these into total environment episodes. For HotPotQA: 3,000 training tasks × 3 base rollouts = 9,000 episodes. For unsuccessful tasks, 2 reflection alternatives × 1 additional rollout each means up to ~2 × (number of unsuccessful tasks) additional episodes. These additional episodes are multiplicative with the number of trials (N=3) and involve API calls to the cloud-hosted actor LLM (GPT-3/4), which have monetary costs that are not estimated. The paper does not report total API costs, wall-clock time, or compute budget for training.

Mitigation status. The paper does not acknowledge this cost in the main text or attempt to amortize it. Appendix C.1 reports the dataset sizes and training hyperparameters but does not discuss the efficiency tradeoff versus simply running Reflexion with more retries. The authors do not propose cheaper data collection strategies (e.g., using fewer training tasks, shorter rollouts, or single-reflection alternatives instead of pairwise comparisons) nor do they report an ablation showing how performance scales with training data size—leaving open the question of whether 300 HotPotQA training tasks would suffice for near-peak performance, which would substantially reduce the training burden.

The Differential Rating Scheme Requires a Deterministic Actor — Limiting Practical Applicability

The assumption or constraint. The paper's central innovation—using $\Delta G_{k,i} = G_{k,i+1} - G_{k,i}$ as a rating signal for reflection quality—fundamentally depends on the assumption that "the injected randomness that leads to differences in returns... are mostly from the reflection responses" (Section 4.2). This assumption is satisfied only because the paper explicitly sets the actor LLM temperature to zero (T=0, top-p=1) across all experiments, as stated in Appendix C.1: "In all experiments, we set the temperature of actor LM as zero, i.e., T=0 and top p=1 to isolate the randomness of LM from the effects of reflections." With T=0, the actor is deterministic: given the same prompt, it always produces the same action sequence. Therefore, the only difference between trial $i$ and trial $i+1$ on the same task is the appended reflection $y_{k,i}$ , making $\Delta G$ a valid causal measure of the reflection's effect.

The consequence. In any realistic deployment where the actor operates at nonzero temperature—which is standard practice for most LLM applications to encourage diversity, avoid repetitive outputs, and enable exploration—the rating scheme's validity breaks down. If the actor samples actions stochastically, the performance difference $\Delta G_{k,i}$ between two consecutive trials could arise from random variation in action sampling rather than from the reflection's quality. A reflection that is actually unhelpful might appear to produce improvement because the actor happened to sample better actions in the next trial (lucky exploration). Conversely, a genuinely insightful reflection might be penalized because the actor sampled worse actions despite good advice (unlucky exploration). The training signal becomes noisy in proportion to the actor's temperature, and the paper provides no analysis of how much noise the PPO training can tolerate before the learned reward model becomes unreliable.

This is not a theoretical concern—it directly constrains practical adoption. Most deployed language agents use nonzero temperature (e.g., T=0.3 to T=1.0) to balance exploitation and exploration, avoid getting stuck in deterministic loops, and produce diverse candidate solutions for best-of-N selection. If Retroformer's training pipeline requires T=0 to produce valid rating signals, it cannot be trained on data from a realistically deployed agent—it must be trained in a separate, artificial deterministic regime and then deployed with the hope that the learned reflection skills transfer to the stochastic setting. The paper does not test this transfer.

What evidence exists in the paper. The paper provides no ablation varying the actor temperature during training or evaluation. All results in Table 2, Figure 4, and Figure 6 use T=0. The authors acknowledge this choice in Appendix C.1: "We acknowledge that setting a higher temperature value can encourage exploration but it can obscure the impact of the proposed approaches, making it difficult to compare against existing baselines with T=0." This acknowledges the motivation for T=0 (clean comparison) but does not acknowledge the consequence (that the rating scheme's validity relies on this choice). The paper does not discuss whether a nonzero actor temperature would degrade the reward model's ability to distinguish good reflections from bad ones, or whether alternative rating schemes (e.g., averaging over multiple stochastic rollouts per reflection, or using a learned baseline to subtract expected stochastic variation) could address this limitation.

Mitigation status. Not addressed. The paper does not propose any technique for extending the rating scheme to stochastic actors, such as importance sampling corrections, multi-rollout averaging, or learned value functions that estimate expected returns conditional on the reflection. The limitation is not listed in the conclusion or future work sections—it is an implicit constraint that the paper's experimental design enforces but does not discuss as a limitation. Any practitioner attempting to apply Retroformer to a deployed agent with standard sampling temperatures would need to solve this problem independently or accept a potentially degraded training signal.

The Method Is Evaluated on Only Three Benchmarks with a Single Model Family

The assumption or constraint. All experiments use GPT-3 (text-davinci-003) and GPT-4 as the frozen actor LLMs, LongChat-7b-16k as the retrospective model base, and three specific benchmarks: HotPotQA (multi-hop QA with Wikipedia search), AlfWorld (text-based embodied task completion), and WebShop (web browsing for product purchase). The paper states in Section 4 that the actor model is assumed to be "a frozen LLM whose model parameters are inaccessable (e.g., OpenAI GPT)" and that the retrospective model is "a smaller, local language model that can be fine-tuned under low-resource settings (e.g., Llama-7b)." While these are exemplary instances, the paper does not test whether Retroformer's advantages generalize to other actor LLM families (Claude, Gemini, Llama-3, Mistral), other retrospective model architectures, or other benchmark domains.

The consequence. Several aspects of the findings could be specific to the tested configuration. The quality of the learned reward model and the PPO-fine-tuned retrospective model depends on the distribution of the actor's failure trajectories—different actor LLMs may produce qualitatively different errors, and a retrospective model trained on GPT-3 trajectories may not transfer to GPT-4 or to models from other families. The AlfWorld results are particularly striking (100% vs. 84% for GPT-3 with Reflexion), but this environment has a constrained action space and relatively stereotyped failure modes (action loops, targeting wrong objects)—it is unclear whether similarly dramatic improvements would transfer to environments where failures are more diverse and less patternable, such as open-ended code generation, multi-turn dialogue, or creative writing tasks. The WebShop results, where Retroformer provides minimal improvement over Reflexion across all configurations, already serve as an existence proof that the method's effectiveness is environment-dependent. Without testing on additional benchmarks, the paper cannot characterize which environment properties predict Retroformer's effectiveness—whether it is the action space structure (discrete vs. open-ended), the reward sparsity, the typical failure mode type (credit assignment vs. exploration), or the actor model's baseline capability level.

Furthermore, the retrospective model's base architecture (LongChat-7b-16k) was chosen for its 16,000-token context window, which is necessary to ingest full episode trajectories as reflection prompts. The paper does not test whether a shorter-context model (e.g., standard Llama-7b with 2,048 tokens) could achieve comparable performance with trajectory summarization, or whether a larger retrospective model (e.g., Llama-13b or Llama-70b) would produce substantially better reflections. The LoRA rank ablation (r=1 vs. r=4, Table 2) shows diminishing returns from increased capacity, but this only characterizes the low-rank adaptation regime, not the scaling behavior of the base model itself.

What evidence exists in the paper. The paper presents results on exactly three benchmarks (Table 2, Figures 4 and 6) with exactly two actor LLM variants (GPT-3 and GPT-4) and exactly one retrospective model base (LongChat-7b-16k). No cross-model-family experiments are reported (e.g., using Claude as the actor while training the retrospective model on GPT-3 trajectories, or vice versa). No additional benchmarks are tested. The paper does not discuss whether the choice of LongChat-7b-16k was ablated against other base models.

Mitigation status. The paper does not claim generalization to other model families or benchmarks, but it also does not explicitly acknowledge the single-model-family limitation. The authors position Retroformer's agnostic nature as a strength—"a flexible plug-in module for various types of cloud-based LLMs" (Section 6)—but this claim is aspirationally stated rather than empirically validated. The conclusion suggests that "our approach is not limited to enhancing the retrospective model alone; it can be applied to fine-tune other components" but does not address whether the current evaluation provides sufficient evidence to support broader deployment across model families.

No Combination of PRM Search with Revisions — the Complementary Mechanisms Are Never Tested Jointly

The assumption or constraint. Retroformer operates on a single reflection per failed trial—one retrospective model generates one self-reflection, which is appended to the actor prompt for the next attempt. The paper does not explore whether generating multiple candidate reflections, scoring them with the learned reward model, and selecting the best one (best-of-n at the reflection level) would yield improvements beyond the single-reflection approach. Nor does the paper explore whether reflections could be composed with other test-time improvement strategies, such as majority voting across multiple action trajectories, or iterative refinement where the actor explicitly revises its plan based on the reflection before executing actions.

The consequence. The paper's reported performance represents a lower bound on what a combined approach could achieve. The retrospective model generates a single reflection deterministically during evaluation (the online execution uses "best-of-n sampler, with the scores evaluated by the learned reward model from RLHF pipeline" as stated in Section 4.2, but the quantitative impact of best-of-n sampling on reflections is not separately reported). If generating and scoring multiple reflections could further improve performance—analogous to how best-of-N sampling improves action selection—then Retroformer's headline numbers understate the method's potential. Conversely, if best-of-n reflection sampling provides no benefit (suggesting the reward model cannot reliably rank reflection quality), this would reveal a fundamental limitation of the rating-based training approach.

Additionally, the paper studies Retroformer in a setting where the actor attempts the same task sequentially with accumulated reflections, but it does not test whether combining reflections with parallel strategies—generating multiple independent action trajectories in each trial, selecting the best via majority vote or verifier, and then generating a single reflection based on the best trajectory—would outperform the purely sequential approach. This is a missed opportunity because parallel sampling can compensate for the deterministic actor's lack of exploration while the retrospective model benefits from seeing the most successful trajectory rather than potentially suboptimal individual attempts.

What evidence exists in the paper. The paper describes best-of-n sampling for reflection generation in Section 4.2 ("In online execution, we use best-of-n sampler... for generating better retrospective responses in each trial") but does not report an ablation comparing single-reflection vs. best-of-n reflection performance. All results in Table 2 appear to use the best-of-n approach during evaluation (since this is the described online execution protocol), but the effect of the best-of-n selection step is not isolated from the effect of the PPO fine-tuning. It is therefore impossible to determine whether the reported improvements come primarily from the fine-tuned model generating better individual reflections, or from the best-of-n selection (with the learned reward model) picking better reflections from a set of candidates that includes both good and bad ones. An ablation that disables best-of-n and uses only a single deterministic reflection from the fine-tuned model would clarify this.

Mitigation status. Not addressed. The paper does not report the number of candidate reflections generated in the best-of-n step, the temperature used for reflection sampling, or the performance difference between single-reflection and best-of-n reflection. The interaction between reflection-level best-of-n and action-level best-of-N is unexplored. The paper's focus on establishing the basic Retroformer framework leaves these natural extensions as future work, but their absence means the reported results cannot be attributed cleanly to gradient-based optimization vs. inference-time sampling strategy.

Hard Problems Where the Base Model Fundamentally Lacks Capability Show No Improvement

The assumption or constraint. Retroformer assumes that the actor LLM possesses the underlying capability to solve the task and that failures arise from identifiable credit assignment errors that a reflection can diagnose and correct. The paper does not claim that reflections can create capabilities the base model lacks—it positions the retrospective model as improving the agent's ability to avoid mistakes it is already capable of avoiding with proper guidance, not as expanding the frontier of what the model can do.

The consequence. This assumption creates a hard capability ceiling: for tasks where the actor LLM's base pass@1 is effectively zero—where it lacks the factual knowledge, reasoning depth, or tool-use proficiency to ever produce the correct answer regardless of prompting—Retroformer provides no benefit over any other method. The paper's results indirectly reveal this ceiling in multiple places. Most tellingly, the WebShop results (Figure 6b, Table 2) show that Retroformer, Reflexion, and ReAct all cluster within a narrow 33–46% range across all configurations and retry counts, with Retroformer's maximum improvement over ReAct being only 4 percentage points (GPT-4, N=4: 46% vs. 42%). This near-flat scaling suggests that the majority of WebShop failures arise from the actor's inability to formulate precise search queries or navigate complex product pages—capabilities that a post-hoc textual reflection cannot impart—rather than from credit assignment errors that a good reflection could correct. Similarly, in HotPotQA, the hardest questions (those requiring multi-step reasoning across obscure Wikipedia articles, akin to the difficulty bin 5 problems in the reference example) likely see minimal improvement because the actor cannot retrieve or synthesize the necessary information even with perfect guidance.

This limitation has direct practical implications: Retroformer is not a substitute for improving the base actor model or for mechanisms that expand the agent's capabilities (e.g., better retrieval, more sophisticated action spaces, external verification). A practitioner who deploys Retroformer hoping to solve tasks that are fundamentally beyond the actor's reach will be disappointed, regardless of how many reflective retries are provided.

What evidence exists in the paper. The paper does not explicitly analyze performance as a function of task difficulty (no difficulty binning analogous to the reference paper's Figure 3 right panel). The WebShop results in Figure 6b serve as indirect evidence of the capability ceiling—the flat learning curves across all methods indicate that verbal feedback, whether frozen or reinforced, does not address the core failure modes in that environment. The paper acknowledges this in Section 5.3: "web browsing requires a significant amount of exploration with more precise search queries, if compared with HotPotQA. The results probably indicate that the verbal feedback approach (Reflexion, Retroformer) is not an optimal method for this environment, but our fine-tuning method still proves effective." This statement recognizes the environment-dependent effectiveness but does not characterize which failure modes Retroformer can address vs. which are fundamentally beyond its scope.

Mitigation status. Partially acknowledged but not systematically characterized. The paper notes the WebShop limitation in passing but does not provide a framework for predicting, given a new environment, whether Retroformer will provide substantial, marginal, or negligible improvement. A task difficulty analysis (similar to the reference paper's difficulty quintile breakdown) would reveal whether the HotPotQA and AlfWorld improvements are concentrated in a subset of tasks where the actor has latent capability, and whether a identifiable fraction of tasks in each environment are "unrescurable" by reflections. The paper does not propose combining Retroformer with complementary methods (e.g., search-based exploration, retrieval augmentation, or verifier-guided selection) that might address the capability-ceiling limitation for tasks where credit assignment alone is insufficient.

The 14× Larger Model Baseline Is Weak — the Comparison Does Not Represent a Fair Training-Compute Tradeoff

The assumption or constraint. The paper compares Retroformer against ReAct and Reflexion, both of which use the same frozen actor LLM as Retroformer. The comparison therefore isolates the effect of improving the reflection mechanism while holding the actor constant. However, this comparison does not address the training-inference compute tradeoff that a practitioner actually faces: given a fixed total budget, should I spend resources on (a) fine-tuning a retrospective model with RL and using a smaller/cheaper actor, or (b) using a larger/more-expensive actor with a simpler (or no) reflection mechanism? The paper provides no experiment where the baseline uses a stronger actor model without Retroformer's training overhead—for instance, Reflexion with GPT-4-Turbo or a fine-tuned open-source model vs. Retroformer with GPT-3.5.

The consequence. A practitioner reading the paper cannot determine whether the 2–4 percentage point HotPotQA improvement from Retroformer over Reflexion (both using the same GPT-4 actor) is worth the training cost, or whether simply upgrading from GPT-3.5 to GPT-4 with Reflexion would achieve a larger improvement at lower total cost. The paper's Table 2 shows that switching from GPT-3 to GPT-4 with ReAct improves HotPotQA from 34% to 40% (a 6-point gain)—substantially larger than the 4-point gain from adding Retroformer to GPT-3 (34% → 53–54% at N=4, which requires training). But this comparison confounds two effects (better actor vs. better reflections) and does not control for total cost. A proper cost-controlled comparison would measure: Reflexion with a moderately stronger actor (whose API cost equals Retroformer's training + inference cost) vs. Retroformer with the base actor. Without such an analysis, the paper's claim that gradient-based optimization of the retrospective model is preferable to scaling the actor or using frozen reflections cannot be evaluated on economic grounds.

The paper also does not compare Retroformer against simply giving Reflexion more retries. Since Reflexion requires no training, a practitioner could allocate the budget that would have been spent on Retroformer's data collection and training toward additional Reflexion evaluation episodes. If Reflexion with N=8 achieves 53% on HotPotQA (vs. Retroformer's 54% at N=4), the marginal value of Retroformer's gradient-based training over simply running more frozen-reflection attempts is negligible. This comparison is not reported.

What evidence exists in the paper. The paper reports results for Retroformer and Reflexion at N=1 and N=4 (Table 2) but does not report Reflexion performance at higher retry counts (N=8, N=16) to establish the scaling curve. The only cost analysis is the LoRA parameter count (0.53M for r=1, 2.25M for r=4) and the note that training runs on "a single A100 40GB GPU" (Appendix C.1). Total training time, API costs, and environment interaction counts are not reported. The SAC baseline (Section 5.2) attempts to compare against an alternative RL approach but uses a fundamentally different action representation (continuous embeddings) that makes it uncompetitive (27% vs. 34% for ReAct on HotPotQA), so it does not inform the training-compute tradeoff for language-native methods.

Mitigation status. Not addressed. The paper does not frame Retroformer as a cost-effectiveness intervention, does not report cost-controlled comparisons, and does not discuss the economic tradeoff between training the retrospective model vs. using a stronger actor or more retries. The conclusion (Section 6) positions Retroformer as a "concise and adaptable plug-in module" but does not help practitioners decide when the plug-in is worth the training investment. An honest economic analysis—amortizing training cost over expected deployment volume and comparing against alternative uses of the same budget—would substantially strengthen the paper's practical relevance.

7. Implications and Future Directions

How This Work Changes the Landscape

Retroformer introduces a conceptual reframing rather than a paradigm shift: it demonstrates that the frozen, cloud-hosted actor LLM—traditionally viewed as the agent being optimized—can be treated as a fixed component of the environment dynamics, with a separate, trainable retrospective model serving as the actual policy. This reframing is not merely an implementation trick to work around inaccessible API parameters; it bridges the language agent and reinforcement learning literatures by showing that standard policy gradient methods (PPO) apply directly to the problem of improving agent behavior through prompt refinement, without ever needing to access or backpropagate through the actor model. The mathematical formulation in Equation (2)—composing the frozen actor $\mathcal{L}$ with the environment transition $\mathcal{T}$ into a new dynamics function $\mathcal{T}' = \mathcal{T}(S, \cdot) \circ \mathcal{L}$ —provides a reusable template for applying RL to any language agent component whose outputs influence the actor's prompt.

The paper also resolves a tension in the prior literature between two ostensibly contradictory findings: (1) that verbal self-reflection can help agents learn from failure (Reflexion; Shinn et al., 2023), and (2) that frozen LLMs often generate uninformative or counterproductive self-reflections that fail to identify the actual error (Figure 1, this paper). Retroformer's resolution is that the bottleneck is not in the concept of verbal reflection itself, but in the credit assignment capability of the reflection generator. When reflections are generated by a frozen LLM with no environment-specific training, they can be actively harmful—recommending actions the agent already performed, missing the root cause of failure, or hallucinating irrelevant plans. When the same reflection generator is fine-tuned with gradient-based RL on environment-specific trajectories, it learns to diagnose failures accurately and propose actionable corrections. The 15–16 percentage point gap between Retroformer and Reflexion on AlfWorld (100% vs. 84–85% with GPT-3 at N=3, Table 2) is the strongest quantitative evidence that credit assignment, not action generation, is the critical capability that self-reflective agents need to acquire.

This reframing redirects research attention in two ways. First, it makes training environment-specific diagnosticians more attractive as a research direction—investing in better retrospective models, reward models, and credit assignment mechanisms—while making better general-purpose prompting of frozen models for self-reflection less attractive, since the paper shows that even state-of-the-art frozen LLMs (GPT-4) produce reflections that trail those from a 7B-parameter model fine-tuned on 523 AlfWorld trajectories. Second, it opens the door to applying the full RL toolbox—off-policy algorithms, model-based RL, hierarchical RL—to language agent optimization, since any component that generates text consumed by the actor's prompt can now be viewed as a policy operating in an environment that includes the frozen LLM.

The paper's identification of the differential rating scheme ( $\Delta G_{k,i} = G_{k,i+1} - G_{k,i}$ , Equation 5) as a causal reward signal for open-ended language generation is a methodological contribution with implications beyond self-reflective agents. The principle—that when a frozen deterministic policy's input is augmented with generated text between episodes, the performance delta serves as a valid training signal for the text generator without requiring ground-truth labels—could apply to any setting where a frozen model's behavior can be influenced by generated context and outcomes can be measured. This includes prompt optimization for code generation (where unit tests provide returns), dialogue systems (where user ratings provide returns), and tool-use agents (where task completion provides returns).

Finally, the paper delineates a clear boundary condition that is as informative as its positive results: in WebShop, where failures primarily stem from exploration challenges (imprecise search queries, complex product navigation) rather than credit assignment errors, Retroformer provides minimal improvement over Reflexion (45–46% vs. 44% with GPT-4 at N=4, Table 2) and neither verbal feedback approach substantially outperforms ReAct (42%). This negative result is valuable because it shows that Retroformer's effectiveness is not a function of "applying RL to reflections" in general, but specifically of addressing credit assignment failures—and it implies that environments dominated by exploration bottlenecks require fundamentally different interventions.

Follow-Up Research This Work Enables

Directly predicting task difficulty from the question text to eliminate the expensive offline data collection stage. The paper collects 3,383 HotPotQA training samples, 523 AlfWorld samples, and 267 WebShop samples by rolling out the base policy for three trials each and generating alternative reflections for pairwise comparison. For HotPotQA, this means thousands of environment interactions just to train the retrospective model. A strong follow-up would train a lightweight classifier—perhaps a fine-tuned encoder-only model like DeBERTa or a frozen LLM with a classification head—to predict, directly from the question text, whether the actor LLM is likely to succeed (reflection not needed), fail due to credit-assignable errors (Retroformer will help), or fail due to capability gaps (neither Retroformer nor Reflexion will help). The training data already exists: the paper's offline data collection produces, for each training task, the sequence of returns across retries, which reveals whether reflections improved performance. A classifier that predicts this improvement category from the question text alone could eliminate the need for expensive trajectory rollouts for difficulty estimation, directly reducing the training cost that the paper does not account for. The evaluation would compare the classifier's predicted improvement category against actual Retroformer improvement on held-out tasks, and measure whether deploying Retroformer only on tasks predicted to benefit (rather than all tasks) maintains most of the aggregate performance gain while substantially reducing inference cost.

Cross-environment transfer: does the credit assignment skill learned in one environment generalize to others? The paper trains separate retrospective models for HotPotQA, AlfWorld, and WebShop, but the architecture is identical across environments—the retrospective model receives a trajectory and return, and outputs a diagnosis and corrective plan. An open question is whether credit assignment is a generalizable skill: can a retrospective model trained on HotPotQA trajectories (where failures involve incorrect answer synthesis, overbroad search queries, or misreading retrieved passages) transfer to AlfWorld (where failures involve action loops, wrong object targeting, or sequence errors)? A cross-environment transfer experiment would train the retrospective model exclusively on HotPotQA data, then evaluate on AlfWorld tasks (and vice versa), comparing against both the within-environment Retroformer and the Reflexion baseline. If transfer yields performance between Reflexion and within-environment Retroformer, it suggests that credit assignment has a domain-general component—the model learns how to diagnose failures from diverse trajectories, not just the specific failure patterns of one environment. If transfer yields no improvement over Reflexion, it indicates that credit assignment is environment-specific and that the retrospective model is essentially memorizing common failure patterns rather than learning a general diagnostic capability. This distinction matters for practical deployment: if transfer works, a single retrospective model could be trained once on a diverse corpus of agent trajectories and deployed across many environments without per-environment data collection.

Ablation on the rating scheme: does the differential rating $\Delta G$ outperform simpler alternatives? The paper's rating scheme in Equation (5) defines $r(x_{k,i}, y_{k,i}) = G_{k,i+1} - G_{k,i}$ and argues that this measures the causal effect of the reflection. But three simpler alternatives exist: (a) rate reflections based solely on the absolute return of the next episode $G_{k,i+1}$ (ignoring the baseline $G_{k,i}$ ), (b) rate reflections based on whether the next episode succeeds or fails (binary, discarding partial-credit signals like F1 scores), or (c) use the learned reward model's own score as the rating without any episode rollout (self-supervised). A systematic ablation would train four Retroformer variants, each using one of these rating schemes, on the same HotPotQA training data, and compare both the learned reward model's ability to distinguish accepted/rejected reflection pairs and the final agent success rate. The paper's theoretical argument—that $\Delta G$ isolates the reflection's causal effect—is compelling but untested against simpler alternatives. If the absolute return scheme performs comparably, it would undermine the paper's conceptual contribution about causal reward signals and suggest that any signal correlated with reflection quality suffices. If the differential scheme substantially outperforms alternatives, it validates the causal framing and provides guidance for future work on training text generators from behavioral outcomes.

Scaling the retrospective model: does a larger base model (13B, 70B) produce substantially better reflections, or is 7B sufficient? The paper uses LongChat-7b-16k as the retrospective model base and tests LoRA ranks of r=1 and r=4, finding diminishing returns from increased adapter capacity (Table 2). But the LoRA rank ablations only characterize the low-rank adaptation regime—they do not answer whether the base model's pre-trained capabilities matter. A scaling experiment would fine-tune Retroformer retrospective models based on Llama-7B, Llama-13B, and Llama-70B (all with the same LoRA configuration and training data) and compare reflection quality and agent success rate. If larger models produce substantially better reflections (e.g., Llama-70B Retroformer achieves 60% on HotPotQA vs. Llama-7B's 54%), it would suggest that the credit assignment task benefits from the deeper reasoning capabilities of larger pre-trained models, and that Retroformer's effectiveness will improve as open-source models scale. If larger models produce minimal gains (e.g., all within 1–2 percentage points), it confirms that credit assignment on these trajectory types is a relatively shallow pattern-recognition task that a 7B model can saturate—implying that investment should go into better training data and rating schemes rather than larger retrospective models.

Combining Retroformer with parallel sampling: does generating multiple reflections and selecting the best one via the reward model improve over single-reflection evaluation? The paper mentions using best-of-n sampling for reflection generation during online execution (Section 4.2) but does not report an ablation isolating this effect. A direct experiment would compare three Retroformer evaluation variants on HotPotQA: (a) single deterministic reflection from the fine-tuned model (T=0), (b) best-of-n with 4 candidate reflections scored by the learned reward model, and (c) best-of-n with 8 candidates. This would quantify how much of Retroformer's reported improvement comes from the fine-tuned model's individual reflection quality vs. the inference-time selection mechanism. If best-of-n provides substantial additional gains (e.g., single reflection 52%, best-of-8 56%), it suggests that the reward model can reliably rank reflection quality and that Retroformer's practical deployment should always use best-of-n. If best-of-n provides negligible gains (all variants within 1 point), it indicates that the fine-tuned model's reflections are already near-optimal and that the primary value is in the training, not the inference-time sampling. This distinction has important cost implications: best-of-n increases inference cost multiplicatively, and practitioners need to know whether the cost is justified.

Stress-testing the differential rating scheme under stochastic actor policies (T > 0). The paper's rating scheme assumes a deterministic actor (T=0, Appendix C.1) so that performance deltas can be attributed to the reflection rather than to random action sampling. In any realistic deployment with nonzero temperature, this assumption breaks. A stress-test experiment would repeat the Retroformer training pipeline with actor temperature set to T=0.3, T=0.7, and T=1.0, measuring: (a) how much noise is introduced into the $\Delta G$ signal (variance of ratings for the same reflection across multiple stochastic rollouts), (b) whether the learned reward model's accuracy on pairwise comparison degrades, and (c) whether the final agent success rate degrades compared to the T=0 baseline. If the rating scheme is robust to moderate temperatures (T ≤ 0.3), Retroformer could be deployed with the exploration benefits of stochastic sampling without requiring methodological changes. If performance degrades sharply even at low temperatures, the paper's approach would need modification—for instance, averaging over multiple stochastic rollouts per reflection to reduce rating variance (at increased data collection cost), or using a learned value function to subtract expected stochastic variation from the observed $\Delta G$ . This experiment directly addresses the most significant unacknowledged limitation of the current work.

Practical Applications and Downstream Use Cases

Cost-efficient improvement of API-only language agents without model access. The most direct application of Retroformer is for organizations deploying language agents built on cloud-hosted LLMs (GPT-4, Claude, Gemini) who observe persistent failure patterns but cannot fine-tune the underlying model. For example, a customer support agent that repeatedly misclassifies refund requests as exchange requests, or a code generation agent that consistently forgets to import required libraries—these are credit-assignable errors that Retroformer's retrospective model can learn to diagnose from a modest number of failure trajectories. The key practical advantage is that Retroformer requires no access to the actor model's parameters, weights, or training pipeline—it operates entirely through the prompt interface. The training cost (3,383 trajectory samples for HotPotQA, Appendix C.1) amounts to perhaps a few thousand API calls, which for a high-volume deployment (millions of queries) amortizes to negligible per-query cost. In AlfWorld-like settings where Retroformer achieves 100% success vs. 84% for Reflexion (Table 2), the 16-percentage-point improvement in task completion rate directly translates to reduced failure-handling costs, fewer human escalations, and improved user experience—all without switching to a more expensive actor model or gaining model access.

Automated prompt optimization for specialized enterprise tools and APIs. Retroformer's architecture—a frozen actor LLM plus a trainable component whose outputs influence the actor's prompt—generalizes beyond self-reflection to any setting where a language agent interacts with a tool or API and receives success/failure feedback. Consider an enterprise deployment where an LLM agent generates SQL queries against a company database, with execution success as the reward signal. A retrospective model could be trained on trajectories of failed queries (syntax errors, wrong table names, incorrect JOIN logic) to generate reflections that steer the actor toward correct query patterns. The same pattern applies to agents that call internal APIs (correct vs. incorrect endpoint selection), manipulate documents (valid vs. invalid transformations), or control robotic equipment (successful vs. failed manipulation sequences). In each case, the environment-specific reward signal—query execution success, API response code, transformation validity, task completion—provides the returns needed for Retroformer's differential rating scheme. The retrospective model learns the specific credit assignment patterns of that enterprise environment from a modest number of training trajectories (hundreds, based on the AlfWorld data size of 523 samples achieving 100% success), and the actor model remains the standard cloud-hosted LLM. This is immediately practical because it requires no new infrastructure—just a training loop that collects trajectories, computes ratings, and fine-tunes a 7B-parameter model on a single GPU.

On-device or edge deployment where the actor is frozen but task-specific adaptation is needed. Retroformer's separation between the large, frozen actor model and the small, trainable retrospective model (0.53M trainable parameters with LoRA r=1, representing 0.015% of Llama-7b) enables deployment scenarios where the actor runs in the cloud but the retrospective model runs locally, or where both run on-device but only the retrospective model is updated. For instance, a mobile robotics application using a cloud-hosted LLM for high-level planning could deploy a locally-fine-tuned Retroformer that has learned, from the specific robot's past failures in its specific environment (that particular kitchen layout, those particular object locations), to generate reflections that prevent common errors like targeting the wrong drawer or forgetting to open a cabinet before reaching inside. The 100% AlfWorld success rate with Retroformer (Table 2) vs. 85% with Reflexion suggests that environment-specific fine-tuning can eliminate essentially all routine failure modes, enabling reliable autonomous operation. When the robot is deployed in a new environment, only the small retrospective model needs to be fine-tuned on new trajectory data—the cloud actor remains unchanged, avoiding the cost and latency of cloud model updates.

Data generation for self-improving agent pipelines. Retroformer's retrospective model, once trained, can serve as a high-quality automatic critic for agent trajectories—generating diagnostic reflections that identify errors and propose corrections. These reflections, paired with the original failed trajectories, form a dataset of (mistake, diagnosis, correction) triplets that could be used to fine-tune the actor model itself—if and when actor model access becomes available, or for open-source actor models where fine-tuning is feasible. This is analogous to the self-improvement loops discussed in the reference paper (STaR, ReST $^{EM}$ ), but with the retrospective model serving as an automated error annotator rather than relying on human feedback or simple correctness signals. In domains like AlfWorld where Retroformer achieves 100% success, the retrospective model has effectively learned to perfectly diagnose all failure modes in the training distribution—its reflections on training set failures would be near-oracle error annotations. Using these annotations to fine-tune the actor could bake the corrective knowledge directly into the actor's weights, potentially achieving high success rates without requiring retrospective prompting at inference time. The paper does not explore this direction, but the architecture naturally supports it: once the retrospective model is trained and generating high-quality reflections, those reflections become training data for actor improvement in a virtuous cycle.

When to Prefer This Method

The paper explicitly positions Retroformer against Reflexion as the primary alternative for agents that need to learn from environment feedback without actor model access. The decision rule that emerges from the experimental results (Table 2, Figures 4, 6) and the method's design constraints is:

Prefer Retroformer over Reflexion when:

The environment has identifiable, repeatable credit assignment failures that an LLM can diagnose from trajectory text (action loops in AlfWorld, synthesis errors in HotPotQA). The paper shows gains of 15–16 percentage points in such environments (AlfWorld, Table 2).
Training budget is available for offline data collection and PPO fine-tuning. Retroformer requires collecting hundreds to thousands of reflection samples (523 for AlfWorld, 3,383 for HotPotQA, Appendix C.1) plus reward model and PPO training, but this cost amortizes over deployment volume.
The actor model is cloud-hosted with inaccessible parameters and you cannot fine-tune it directly. Retroformer is specifically designed for this constraint (Section 4.2).
The actor can operate deterministically (T=0) during training, or you have a strategy for handling the noise that stochastic sampling introduces into the rating scheme. The paper does not validate Retroformer with T > 0.

Prefer Reflexion over Retroformer when:

Zero training cost is required and the environment is one where frozen reflections already provide most of the achievable benefit. On HotPotQA, Reflexion achieves 50–52% vs. Retroformer's 53–54% (Table 2)—a 2–4 point gap that may not justify training cost for low-volume deployments.
The environment is dominated by exploration bottlenecks rather than credit assignment errors. In WebShop, Retroformer provides minimal improvement over Reflexion (45–46% vs. 44% with GPT-4, Table 2), and the paper's own analysis suggests "the verbal feedback approach is not an optimal method for this environment" (Section 5.3). In such environments, neither Retroformer nor Reflexion is the right choice—alternative methods focused on search, exploration, or retrieval augmentation should be explored.
The actor model must operate at nonzero temperature for exploration, and you do not have the infrastructure to implement the multi-rollout averaging or baseline subtraction that would be needed to maintain rating signal quality. The paper does not provide solutions for stochastic actors.

Prefer scaling the base actor model (e.g., GPT-3 to GPT-4) over Retroformer when:

The primary failure mode is fundamental capability gaps (the actor lacks the knowledge or reasoning depth to ever produce correct answers) rather than credit-assignable errors. The paper's WebShop results show that even with perfect reflections, the agent cannot overcome exploration and precision limitations—a stronger base model would likely help more than better reflections.
Inference volume is very low such that Retroformer's training cost cannot be amortized. For a deployment handling dozens of tasks, the thousands of training rollouts outweigh any per-task improvement.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

RLCD: Reinforcement Learning from Contrast Distillation for Language Model Alignment2023· rl-training← in-domain predecessor
- Deep reinforcement learning from human preferences2017· rl-training← LLM-judged similar
- Secrets of RLHF in Large Language Models: Part I: PPO2023· rl-training← in-domain predecessor
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes2023· training-methods← topic thread
Deterministic Policy Gradient Algorithms2000· rl-training← similar topic
- Bootstrapped Thompson Sampling and Deep Exploration2000· rl-training← in-domain predecessor
- Fiber: Distributed Computing for AI Made Simple2017· rl-training← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

Agentic Reinforced Policy Optimization2025· rl-training→ LLM-judged similar
- WMPO: World Model-based Policy Optimization for Vision-Language-Action Models2025· rl-training→ LLM-judged similar
- Learning When to Act or Refuse: Guarding Agentic Reasoning Models for Safe Multi-Step Tool Use2026· alignment→ LLM-judged similar
- Complementary Reinforcement Learning2026· rl-training→ LLM-judged similar
- RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models2026· training-methods→ LLM-judged similar
- Geometric-Mean Policy Optimization2025· rl-training→ in-domain predecessor
- Trust Region Policy Optimization2000· rl-training→ similar topic
- Trust Region Policy Optimization2015· rl-training→ similar topic
- GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization2026· rl-training→ similar topic
Gated recurrent neural networks discover attention2023· rl-training→ in-domain predecessor
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality2024· architecture→ LLM-judged similar
- Statistical Rejection Sampling Improves Preference Optimization2023· rl-training→ in-domain predecessor
Deterministic Policy Gradient Algorithms2000· rl-training→ similar topic
- Human-level control through deep reinforcement learning2000· rl-training→ in-domain predecessor
- Fiber: Distributed Computing for AI Made Simple2017· rl-training→ similar topic