Complementary Reinforcement Learning

🎯 Pitch

An agent that learns from past experience gets progressively worse advice as it improves, because the experience bank doesn’t evolve. By co-evolving an experience extractor with the actor inside the RL loop—where the extractor is rewarded only when its advice demonstrably boosts success—the agent achieves 10% higher performance while needing 1.5–2× fewer actions to finish tasks, and crucially internalizes the guidance into its parameters so it stays effective even when the experience bank is removed.

1. Executive Summary

This paper proposes Complementary Reinforcement Learning (Complementary RL), an algorithm and infrastructure co-design framework that jointly trains a policy actor and an experience extractor in a closed co-evolutionary loop, where the extractor continually distills structured experience from agent trajectories while receiving optimization signals based on whether its experience demonstrably improves the actor's task success. Evaluated on four agentic environments—MiniHack, WebShop, ALFWorld, and SWE-Bench—using Qwen2.5-7B-Instruct as the actor and Qwen3-4B-Thinking as the extractor, Complementary RL achieves 10% higher final performance than outcome-based RL baselines that do not learn from experience in single-task settings, with the actor simultaneously requiring 1.5–2× fewer actions to complete tasks (e.g., 1.5× fewer actions on MiniHack Room) by internalizing transferable decision strategies distilled from past rollouts. In multi-task training across three environments, Complementary RL outperforms baselines using static or non-adaptive experience banks by substantial margins—yielding a 7% average improvement with experience at test time and a 2% improvement even without retrieved experience—establishing that co-evolving the extractor alongside the actor enables the agent to internalize useful guidance into its parameters, while static experience banks progressively lag behind the actor's growing capabilities and provide diminishing returns.

2. Context and Motivation

The Problem: Experience in Agent RL Is Underutilized Because Extractors Don't Co-Evolve

Reinforcement Learning has become a dominant paradigm for training LLM-based agents to interact with environments—navigating rooms, making purchases on websites, completing household tasks, or resolving software bugs. The RL recipe here is straightforward: the agent takes actions in a text-based environment, receives a sparse binary outcome reward (success=1, failure=0) at episode termination, and uses that signal to update its policy. This approach demonstrably works—the paper cites a growing body of evidence from Jin et al. (2025), Dong et al. (2025), and Xue et al. (2025)—but it suffers from a fundamental inefficiency that the paper identifies in its opening paragraphs (Section 1):

"Policy updates rely solely on sparse reward signals... which, while effective at optimizing task outcomes, provide no explicit signal for why a trajectory succeeded or failed throughout the multi-turn interaction process."

This is the core gap. When an agent spends 20 turns navigating a maze and finally reaches the exit, the binary reward tells it "good job," but says nothing about which specific decisions were wise (e.g., checking for traps before moving east), which failures were avoidable (e.g., walking into a dead end three times before backtracking), or which patterns generalize to similar tasks. The trajectory contains rich procedural information—effective behaviors, recoverable failure patterns, critical decision points—but outcome-based RL leaves this information entirely unexploited. The agent essentially learns from trial and error, discarding all intermediate reasoning and behavioral data that could accelerate learning on future episodes.

This is not merely a theoretical concern about data efficiency. The paper performs a concrete pilot study (Figure 3a in Section 2.2) that quantifies the gap. When an actor is trained from scratch on MiniHack Room using only outcome rewards, it eventually converges, but the learning curve rises slowly. The question the paper poses is: can we accelerate this learning by distilling structured experience from past trajectories and feeding it back as guidance for future episodes?

Why This Matters: Efficiency, Transfer, and Internalization

The importance of solving this problem extends beyond faster training curves, touching three practical concerns that matter for real-world LLM agent deployment:

Sample efficiency under sparse rewards. In many agentic tasks, success is rare in early training, meaning the agent receives almost no positive reinforcement signal. On MiniHack Room, a maze navigation task with traps, monsters, and fog-of-war visibility, an untrained agent wanders randomly and rarely stumbles upon the goal. If each failed trajectory could nevertheless extract useful patterns—"this corridor always leads to a dead end," "checking for monsters before stepping into a dark room prevents death"—then the agent could learn from failures, not just successes. The paper frames this as extracting the signal that outcome-based RL misses:

"the rich procedural information embedded in collected rollouts, such as effective behaviors, recoverable failure patterns, and critical decision points, is largely unexploited."

Transfer across tasks via shared experience. When an agent learns to navigate rooms, search for products, and complete household chores simultaneously (the multi-task setting in Section 4.2), experience distilled from one task might inform another. A strategy like "when stuck in a loop, stop and diagnose rather than repeating the same action" applies equally to maze navigation, web shopping, and code debugging. An experience extractor that can identify such cross-cutting patterns would provide scaling benefits as more tasks are added—precisely what the paper examines in its task scaling experiment (Figure 9d).

Internalizing guidance into the policy. Perhaps the most subtle but critical concern is that an agent that merely receives retrieved experience as in-context guidance at test time has not truly "learned" from that experience—it relies on an external crutch. If the experience bank is unavailable at deployment (due to latency, privacy, or infrastructure constraints), performance collapses. The paper explicitly measures this by evaluating agents both with and without retrieved experience at test time (Table 1), and shows that co-evolutionary training produces agents that retain gains even when experience is stripped away, because the guidance has been internalized into the policy parameters.

Where Prior Approaches Fall Short

The paper engages with two broad families of prior work on leveraging experience for RL training, and identifies specific failure modes that motivate Complementary RL. Here we unpack each.

Experience as Static Resource

The most natural approach—and the one closest to how humans use memory—is to maintain an experience bank: a collection of structured knowledge distilled from past trajectories, which the agent queries for relevant guidance before each new episode. This idea appears under various names in the literature: memory banks (Zhao et al., 2025), workflow memories (Wang et al., 2024), episodic memory buffers (Zhang et al., 2026a), and reasoning banks (Ouyang et al., 2025).

The paper subdivides this family into two variants, which it tests empirically in its pilot study (Section 2.2, Figure 3a):

Offline experience banks. These are pre-constructed from prior trajectories using an external extractor model and remain frozen during RL training. Zhai et al. (2025) exemplify this approach: an LLM processes historical rollouts offline, distills them into structured experience entries, and the resulting bank is used for retrieval throughout training without modification. The pilot study confirms an intuitive pattern: offline experience provides an initial performance boost in early training, because it gives the stumbling novice agent useful hints about how to navigate. But as the agent improves and its policy shifts to exploit discovered strategies, the static bank's benefit decays progressively:

"While offline experience provides an initial performance boost, its benefit decays progressively over the course of training."

The reason is a distributional mismatch. The bank was distilled from trajectories generated by an earlier, less capable version of the agent. When the agent now encounters situations it never saw before—because its improving policy leads it to novel states—the bank has no relevant experience to offer. Worse, the guidance it does provide may be actively misleading: a strategy that was optimal for a novice agent (e.g., "always check every room systematically") becomes inefficient for a skilled agent that can recognize which rooms are dead ends at a glance.

Static online experience banks. A partial fix is to update the bank dynamically during training: as new trajectories are collected, the experience extractor distills fresh experience and adds it to the bank. The extractor itself, however, is not optimized—it remains a frozen model using a fixed distillation strategy. The pilot study tests this variant and delivers a sobering result:

"Static online experience yields only marginal gains over the baseline, suggesting that simply collecting online experience without co-evolving the extractor is insufficient."

Why does this fail? Even though the bank receives fresh trajectories, the extraction strategy is calibrated to the initial agent. An extractor that learned to distill "how to avoid obvious traps" from early rollouts is not equipped to extract "how to efficiently chain multiple tool uses" from later rollouts where the agent attempts sophisticated multi-step plans. The extractor's criteria for what constitutes a useful experience entry are static, while what the actor needs changes. The paper crystallizes this diagnosis in a single phrase:

"We attribute this to a distributional misalignment: a static M cannot track the evolving state-action distribution of πθ, causing the guidance to become stale and counterproductive."

This is the paper's central critique of prior work. The extractor and the actor are two components of a single learning system, but existing approaches treat them asymmetrically—the actor improves while the extractor stagnates—creating a growing capability gap that undermines the very purpose of having an experience bank.

Self-Reflection for Experience Distillation

A second family of approaches, represented by Zhan et al. (2025), attempts to distill experience through self-generated reflections. The idea is appealingly simple: after each episode, prompt the same LLM that serves as the actor to "reflect" on what went well and what went wrong, producing textual insights that then become in-context guidance for subsequent episodes. No separate extractor model is needed; the actor generates its own experience.

The paper identifies a critical reliability problem with this approach:

"When the base model is weak or tasks are complex, self-reflection becomes unreliable, frequently producing hallucinations that corrupt rather than enrich the learning signal."

This is the garbage-in, garbage-out problem applied to experience. An untrained agent on MiniHack Room does not know why it died—it may attribute failure to "the room was too dark" when the actual cause was stepping onto an invisible trap. Its reflections therefore encode false causal models, and feeding these hallucinations back into training reinforces misconceptions rather than correcting them. Lin et al. (2025) have documented the broader hallucination problem in LLM agents, and the paper cites this work to ground the concern.

Quality-Centric and Retrieval-Centric Alternatives

The paper acknowledges two other lines of work that attempt to address different aspects of the experience problem, though it engages with them cursorily (Section 5, Related Works):

Quality-centric approaches focus on improving the distillation process itself. Rather than naively reflecting or applying a fixed extraction template, these methods introduce specialized data structures for organizing experience (Qian et al., 2025; Ouyang et al., 2025) or dedicated experience models that refine experience dynamically (Zhai et al., 2025; Zhang et al., 2025a; Xia et al., 2026; Yan et al., 2025). The extractor model is more sophisticated, but it is still not optimized jointly with the actor—it improves experience quality independently of whether that quality translates to better actor performance.

Retrieval-centric approaches, typified by Zhou et al. (2025) and Zhang et al. (2026a), design multi-stage retrieval heuristics to surface the most valuable experience from an accumulated bank. The extraction strategy may be sophisticated, but the bank construction strategy remains fixed. The paper's critique is consistent across these variants: all treat some component of the experience pipeline as static, whether the bank contents, the extraction strategy, or the retrieval mechanism. The consequence is the same. As the actor climbs its learning curve, the experience pipeline falls behind, and the guidance becomes stale.

Three Design Requirements for Experience-Driven Agent RL

Having surveyed the landscape and identified the common failure mode—static versus dynamic components creating mismatch—the paper articulates three design requirements that any successful experience-driven RL algorithm must satisfy (Section 1):

"❶ Actor-Extractor Co-Evolution: the actor and experience extractor must mutually adapt throughout training, each continuously shaping the other toward greater capability."

This is the core insight. The extractor should not just generate experience; it should learn to generate better experience over time, where "better" means "more helpful to the current, evolving actor." The actor, in turn, should learn to use the extractor's guidance more effectively. The two are not independent modules but symbiotic learners.

"❷ Experience Consolidation: the experience bank must be automatically constructed and maintained from trajectories, distilling transferable experience while resolving conflicts and redundancies."

Having better extraction is insufficient if the bank itself becomes bloated with contradictory or redundant entries. As multiple parallel environments generate trajectories and the extractor adds new experience, the bank must be curated—outdated entries discarded, similar entries merged, conflicts resolved—without human intervention.

"❸ Training-Distillation Coordination: actor training and experience distillation must be efficiently coordinated at scale without introducing blocking latency to actor training."

This is the engineering requirement that makes co-evolution practical, not just theoretically appealing. If the extractor must wait for the actor to finish collecting rollouts, and the actor must wait for the extractor to finish distillation before receiving updated guidance, training throughput collapses. The paper explicitly names this failure mode:

"A naïve implementation would serialize these dependencies... introducing synchronization barriers that cause significant resource idleness and degrade overall training throughput."

Biological Inspiration and How This Paper Positions Itself

The paper draws an explicit analogy to Complementary Learning Systems (CLS) theory from neuroscience (O'Reilly et al., 2011), which describes how the human brain solves a structurally identical problem. CLS posits two complementary memory systems:

The neocortex forms slow, structured, long-term knowledge—patterns of how the world works that generalize across specific episodes. This corresponds to the actor's policy: a durable, slowly-updating model of how to act in the environment.
The hippocampus manages fast, episode-specific memories—individual experiences that are initially stored separately but are then consolidated into the neocortex through replay and feedback. This corresponds to the experience extractor: a rapidly-updating system that captures fresh trajectory-level insights and selectively transfers structurally valuable ones to the actor.

The crucial mechanism in CLS is bidirectional interaction: the hippocampus replays memories to strengthen neocortical representations, while the neocortex provides structured knowledge that helps the hippocampus interpret and encode new experiences efficiently. One system shapes the other, and vice versa. This is precisely the co-evolutionary loop that Complementary RL implements between the actor and extractor.

The paper explicitly frames its contribution in these terms (Section 1):

"Can we design a RL framework in which the policy actor and its experience extractor form a closed co-evolutionary loop, each continuously shaping the other toward better?"

The positioning relative to prior work is now clear. Prior approaches correspond to taking only one half of the CLS mechanism: they maintain a hippocampus-like memory store (the experience bank), but they don't allow it to teach the neocortex-like actor through bidirectional feedback, nor do they allow the neocortex's growing understanding to reshape what the hippocampus encodes. Complementary RL is the first RL framework that closes this loop, treating both components as learners in a mutual optimization process.

The paper is not proposing a new model architecture, a new RL objective in isolation, or a new retrieval mechanism. It is proposing a paradigm shift in how we think about the relationship between an agent and its memory: from a static assistant serving a growing master to two co-evolving learners that strengthen each other through a unified optimization loop.

3. Technical Approach

3.1 Reader Orientation

Complementary RL is a training framework that simultaneously optimizes two language models—a policy actor that solves tasks and an experience extractor that distills reusable knowledge from the actor's attempts—where each model's improvement directly drives the other's, forming a closed feedback loop analogous to how the human brain's neocortex and hippocampus mutually reinforce each other during learning. The problem it solves is the progressive staleness of guidance that occurs when an agent learns from a static memory bank: as the agent gets better at a task, fixed experience entries become obsolete or even misleading, so Complementary RL instead treats the experience extractor as a learner itself, optimizing it based on whether its extracted experience actually helps the actor succeed, thereby ensuring that the experience bank continuously adapts to the actor's evolving capabilities rather than falling behind.

3.2 Big-Picture Architecture (Diagram in Words)

The Complementary RL system has six major components that operate in two asynchronous tracks:

Policy Actor ( $\pi_\theta$ ) — a language model that interacts with text-based environments (MiniHack rooms, WebShop, ALFWorld, SWE-Bench) by receiving observations, producing actions, and receiving sparse binary outcome rewards. It is optimized via Group Relative Policy Optimization (GRPO) with a split-subgroup design that evaluates guided and unguided rollouts separately.
Experience Extractor ( $\pi_\phi$ ) — a separate language model that, after each episode completes (success or failure), receives the full interaction trajectory, the task goal, the episode outcome, and the experience entry that guided the episode, then produces structured operations (Add, Update, or Return) to maintain the experience bank. It is optimized via a clipped importance sampling policy optimization (CISPO) objective based on whether its distilled experiences demonstrably helped the actor succeed.
Experience Bank ( $\mathcal{M}$ ) — a vector-indexed store of structured experience entries, each consisting of a textual experience string, a dense embedding produced by an embedding model $f_\psi$ , and the prompt-response pair from $\pi_\phi$ that generated it. Supports semantic similarity search under a reader lock and receives write operations under a writer lock.
Experience Manager ( $\mathcal{H}$ ) — a centralized coordinator that handles all concurrency between the primary training loop and the background distillation track. It maintains a distillation queue $\mathcal{Q}$ for incoming trajectory processing requests, manages the reader/writer lock protocol on $\mathcal{M}$ , aggregates retrieval queries into micro-batches, and distributes search across parallel workers.
Embedding Model ( $f_\psi$ ) — a pre-trained sentence embedding model (Qwen3-Embedding-0.6B) that maps textual experience entries and retrieval queries to dense vectors for semantic similarity search. It runs in parallel workers and benefits from query batching and a cache for repeated task descriptions.
Background Track — an asynchronous process that continuously dequeues pending distillation requests from $\mathcal{Q}$ , forwards them to $\pi_\phi$ for experience distillation, applies the resulting operations to $\mathcal{M}$ , accumulates experience-reward pairs into a training buffer $\mathcal{B}_\phi$ , and periodically triggers $\pi_\phi$ optimization and experience bank merging.

Information flows through the system in a continuous cycle: the actor starts an episode by querying $\mathcal{H}$ for relevant experience → $\mathcal{H}$ retrieves the best matching entry from $\mathcal{M}$ and returns it → the actor conditions on this experience (in half of its rollouts) and interacts with the environment until episode termination → the completed trajectory is submitted as a distillation request to $\mathcal{H}$ → the background track dequeues the request, sends it to $\pi_\phi$ for distillation → $\pi_\phi$ produces operations that update $\mathcal{M}$ → the experience is assigned a reward based on the episode outcome and buffered → when the buffer fills, $\pi_\phi$ is updated via CISPO. The actor and extractor optimize on entirely independent schedules, with $\mathcal{H}$ preventing race conditions through its lock protocol.

3.3 Roadmap for the Deep Dive

First, the formal problem formulation in Section 2.1, which establishes why learning from experience requires augmenting the standard MDP objective and provides the mathematical notation used throughout the technical sections.
Second, the pilot study (Figure 3a) that empirically demonstrates why static experience banks fail—this grounds the co-evolutionary requirement in data rather than intuition.
Third, the experience extractor's optimization objective (Equation 3), because it defines how the extractor learns and what reward signal drives its evolution.
Fourth, the actor's split-subgroup GRPO objective (Equation 5), which is the paper's solution to the "over-reliance on external guidance" problem and involves a non-obvious advantage estimation design choice.
Fifth, the asynchronous training infrastructure (Section 3), because co-evolution creates practical concurrency challenges—the actor and extractor have mutual dependencies that must be resolved without blocking either learner.
Sixth, the stabilization techniques (Appendix B), because experience extractor training is severely off-policy and suffers from data redundancy that requires explicit mitigation through retrieval diversification and training-count-aware advantage reweighting.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an algorithm and infrastructure co-design paper whose core idea is that an agent's experience extractor should not be a static component but rather a reinforcement learner that is optimized for the meta-objective of producing experience that demonstrably improves the agent's task success rate, with both models training asynchronously in a closed loop.

Formal Problem Formulation

The paper frames LLM-based agent interaction as a Markov Decision Process (MDP) $\langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R} \rangle$ , where $\mathcal{S}$ is the set of possible environment states (textual observations like grid layouts, webpage contents, or room descriptions), $\mathcal{A}$ is the set of possible actions (directional movements, search queries, tool invocations), $\mathcal{T}: \mathcal{S} \times \mathcal{A} \to \mathcal{S}$ is the deterministic or stochastic environment transition function, and $\mathcal{R}: \mathcal{S} \times \mathcal{A} \to \mathcal{R}$ is the reward function that returns a binary outcome $R \in \{0, 1\}$ at episode termination (with the exception of ALFWorld, which uses $R \in \{-1, 1\}$ ). At the start of each episode, the agent receives a task goal $g$ specifying what it must accomplish. At each timestep $t$ , it observes the current state $s_t$ , generates an internal reasoning trace by reflecting on the observation and interaction history, and then samples an action $a_t \sim \pi_\theta(\cdot \mid s_{\leq t}, g)$ from its policy—following the ReAct paradigm (Yao et al., 2022) of interleaved reasoning and acting. The episode terminates when the task is completed or after $T_{\text{max}}$ steps, and the objective is to maximize expected success rate across diverse tasks and environments.

The outcome-only objective. In standard outcome-based agentic RL, the policy $\pi_\theta$ is optimized to maximize:

$J(\theta) = \mathbb{E}_{\mathcal{E}, g, \tau \sim \pi_\theta} \left[ R(\tau) \right]$

where $\mathcal{E}$ is the environment distribution, $g$ is a task goal sampled from that environment, $\tau = (s_0, a_0, s_1, a_1, \ldots, s_T)$ is the complete interaction trajectory, and $R(\tau) \in \{0, 1\}$ is the binary outcome reward.

What it computes: the expected fraction of episodes in which the agent succeeds, averaged over environments and tasks. The expectation is over the randomness in task sampling, environment transitions, and the agent's own stochastic action sampling. The gradient of this objective is estimated via policy gradient methods (REINFORCE, GRPO) that increase the log-probability of actions in successful trajectories and decrease it in failed ones.

Why this is insufficient: every trajectory $\tau$ contains detailed intermediate information—which corridors led to dead ends, which tool invocations succeeded or failed, which decision points were critical—but the binary reward $R$ collapses all of this into a single bit. A trajectory that fails because of a single catastrophic mistake at step 15 after 14 perfectly reasonable steps is indistinguishable from one that meanders randomly from start to finish. The gradient signal is therefore extremely sparse: the agent receives no credit for smart sub-decisions that didn't quite salvage the episode, and no penalty for specific identifiable errors.

The experience-augmented objective. To incorporate learning from past experience, the standard MDP is augmented with an experience bank $\mathcal{M}$ containing structured knowledge entries $m$ . Before each episode, relevant experience is retrieved and prepended to the agent's context, modifying the policy to $\pi_\theta(\cdot \mid g, m)$ and the objective to:

$J(\theta) = \mathbb{E}_{\mathcal{E}, g, m \sim \mathcal{M}, \tau \sim \pi_\theta(\cdot \mid g, m)} \left[ R(\tau) \right]$

where $m \sim \mathcal{M}$ denotes retrieval of the most semantically relevant experience entry for the current task goal $g$ .

What it computes: the expected success rate when the agent conditions its actions on retrieved guidance from past trajectories, rather than reasoning from scratch. The experience $m$ provides hints about effective strategies, common failure patterns, or environmental rules that the agent can use to make better decisions.

Why augmentation alone is not enough: this objective treats $\mathcal{M}$ as given—it optimizes the actor assuming the experience bank is fixed. But as the actor improves, the optimal content of $\mathcal{M}$ changes. An experience entry that says "always check every room systematically" is helpful for a novice but becomes a wasteful distraction for an expert that can identify promising rooms at a glance. The objective above provides no mechanism for updating $\mathcal{M}$ to track the actor's growing competence. This is the gap that Complementary RL fills by making $\mathcal{M}$ itself the output of a learned extractor $\pi_\phi$ , and then optimizing $\pi_\phi$ alongside $\pi_\theta$ .

The Pilot Study: Why Static Experience Fails

Before presenting the full Complementary RL algorithm, the paper runs a controlled experiment on MiniHack Room (Samvelyan et al., 2021)—a grid navigation task with fog-of-war visibility, traps, and monsters—comparing three experience strategies (Section 2.2, Figure 3a):

Baseline: the actor $\pi_\theta$ is trained via GRPO using only outcome rewards, without any experience retrieval. This establishes the lower bound for what pure trial-and-error RL can achieve on this task.
Offline Exp.: the experience bank $\mathcal{M}$ is pre-constructed by running Qwen2.5-7B-Instruct offline on MiniHack Room for a maximum of 30 interaction turns, routing the resulting trajectories to a large model (Qwen3-30B-A3B-Instruct-2507) for experience distillation, and applying merging and deduplication to produce a fixed bank. This bank is then used for retrieval throughout RL training but is never updated.
Static Online Exp.: $\mathcal{M}$ is maintained dynamically during training by a frozen experience extractor $\pi_\phi$ (Qwen3-4B-Thinking-2507) that processes new trajectories as they arrive and adds entries to the bank, but $\pi_\phi$ itself is never optimized. This tests whether simply having an online-updated bank (with a fixed extraction strategy) is sufficient.

The experimental setup uses a rollout batch size of 128, group size $K = 8$ , clip ratio $\epsilon = 0.2$ , 145 training steps, a micro-batch size of 64 for $\pi_\theta$ , maximum 30 interaction turns, maximum 4,096 output tokens per step, and maximum sequence lengths of 32,768 for the actor and 65,536 for the extractor.

Results and interpretation. The learning curves in Figure 3a reveal three distinct patterns. Offline experience provides a clear initial advantage over the baseline—the agent starts with useful hints and climbs faster in early training—but the gap narrows over time until the two curves nearly converge by step 150. This is the "progressive staleness" effect: the bank cannot adapt to the actor's new capabilities, so its guidance becomes less relevant with each policy update. Static online experience fares even worse: it yields only marginal gains over the baseline throughout training, never achieving the early advantage of the offline bank. The paper's diagnosis is that a frozen extractor, even one receiving fresh trajectories, applies a static distillation strategy that is mismatched to the actor's evolving state-action distribution. The extractor keeps extracting the same kind of experience—perhaps patterns relevant to early-stage exploration—when the actor needs qualitatively different guidance as it transitions from exploration to exploitation.

This three-way comparison establishes the paper's central empirical claim: the benefit of experience depends not on whether it is collected online, but on whether the extraction strategy co-evolves with the actor. A static strategy, regardless of whether its input data is fresh, inevitably falls behind. This motivates the co-evolutionary design where $\pi_\phi$ is itself a learner that adapts its distillation behavior based on feedback about what helps the actor.

The Experience Extractor: Optimization Objective and Reward Design

What the extractor does. At the end of each episode (success or failure), the experience extractor $\pi_\phi$ receives four pieces of information: the task goal $g$ that the agent was trying to achieve, the full interaction trajectory $\tau = (s_0, a_0, \ldots, s_T)$ containing all observations and actions, the binary episode outcome $o \in \{\text{success}, \text{failure}\}$ , and the specific experience entry $m \in \mathcal{M}$ that was retrieved to guide this episode. From this input, $\pi_\phi$ generates an experience entry $m_{\text{new}} \sim \pi_\phi(\cdot \mid g, \tau, o, m)$ and issues one of three structured operations:

Add: insert $m_{\text{new}}$ as a fresh entry into $\mathcal{M}$ , typically when the trajectory reveals a novel pattern or strategy not already captured by existing entries.
Update: replace or modify the previously retrieved entry $m$ with $m_{\text{new}}$ , typically when the episode reveals that the existing entry was incomplete, misleading, or could be refined based on new evidence.
Return: take no action, when the episode yields no extractable insight (e.g., it was too short or too random to contain structured knowledge).

The reward signal for experience. The extractor is not trained to maximize the agent's task success directly—that would conflate the quality of the experience with the quality of the actor's execution. Instead, each experience entry $m$ receives a binary reward $r(m) \in \{-1, +1\}$ based on the outcome of the episode it guided. The authors state this directly:

"We track how $m$ influences subsequent actor behavior by assigning a binary reward $r(m) \in \{-1, +1\}$ based on the outcome of the trajectory it guided."

The $+1$ reward indicates that the experience was helpful—the agent succeeded on the episode that used $m$ as guidance. The $-1$ reward indicates the experience was unhelpful—the agent failed despite (or perhaps because of) using $m$ . This is a direct reward assignment, which the paper found empirically superior to a relative reward strategy that computed improvement over an experience-free baseline (discussed in Appendix B.2):

"Empirical comparison revealed that this relative reward strategy consistently underperforms direct reward assignment."

Experience-reward pairs $(g_i, \tau_i, m_i, r(m_i))$ are accumulated into a training buffer $\mathcal{B}_\phi$ of size $|\mathcal{B}_\phi| = 64$ . Since multiple trajectories may be guided by the same experience entry $m$ , the paper aggregates rewards across all trajectories associated with that entry:

$\bar{r}(m) = \frac{1}{|\mathcal{T}_m|} \sum_{\tau \in \mathcal{T}_m} r(m, \tau)$

where $\mathcal{T}_m \subseteq \mathcal{T}$ is the subset of trajectories in the current rollout batch that were guided by experience entry $m$ , $|\mathcal{T}_m|$ is the number of such trajectories, and $r(m, \tau) \in \{-1, +1\}$ is the outcome of trajectory $\tau$ .

What this computes: the average success rate among all episodes that used $m$ as guidance, mapped to $[-1, +1]$ by treating successes as $+1$ and failures as $-1$ . An experience entry that guided 10 episodes and led to success on 7 of them receives $\bar{r}(m) = 0.4$ ; one that succeeded on only 3 receives $\bar{r}(m) = -0.4$ .

Why aggregate this way: a single trajectory is a noisy signal—the actor might succeed or fail for reasons unrelated to the experience quality. Averaging across multiple episodes using the same experience reduces this noise and provides a more reliable estimate of $m$ 's utility. Additionally, because the number of unique experience entries in a rollout batch may be smaller than the required training batch size for $\pi_\phi$ , the training buffer $\mathcal{B}_\phi$ accumulates samples across multiple rollout collection steps, and $\pi_\phi$ is only optimized when $|\mathcal{B}_\phi|$ reaches the full size of 64.

The CISPO objective. The experience extractor is optimized using the Clipped Importance Sampling Policy Optimization (CISPO) objective from Chen et al. (2025):

$J_{\text{CISPO}}(\phi) = \mathbb{E} \left[ \frac{\sum_{i=1}^{O} \sum_{t=1}^{|m_i|} \text{sg}\left( \left[\rho_{i,t}\right]^{1+\varepsilon^{\text{IS}}_{\text{high}}}_{1-\varepsilon^{\text{IS}}_{\text{low}}} \right) \hat{A}_i \log \pi_\phi(m_{i,t} \mid g_i, \tau_i, m_{i,<t})}{\sum_{i=1}^{O} |m_i|} \right]$

where $O$ is the number of experience samples in the training batch $\mathcal{B}_\phi$ , $|m_i|$ is the number of tokens in the $i$ -th experience entry, $m_{i,t}$ is the $t$ -th token of entry $i$ , and $m_{i,<t}$ denotes all preceding tokens in that entry.

$\rho_{i,t} = \frac{\pi_\phi(m_{i,t} \mid g_i, \tau_i, m_{i,<t})}{\pi_{\phi_{\text{old}}}(m_{i,t} \mid g_i, \tau_i, m_{i,<t})}$ is the token-level importance sampling ratio—the ratio of the probability that the current extractor would generate token $m_{i,t}$ to the probability that the old extractor (at the time the entry was generated) assigned to it. The notation $\left[\rho_{i,t}\right]^{1+\varepsilon^{\text{IS}}_{\text{high}}}_{1-\varepsilon^{\text{IS}}_{\text{low}}}$ denotes clipping this ratio to the interval $[1 - \varepsilon^{\text{IS}}_{\text{low}}, 1 + \varepsilon^{\text{IS}}_{\text{high}}]$ , where both clip thresholds are set to 0.1 in all experiments.

$\text{sg}(\cdot)$ is the stop-gradient operator—it prevents gradients from flowing through the clipped importance weight, treating it as a constant multiplier for the log-probability.

$\hat{A}_i = r(m_i) - \bar{r}$ is the batch-level advantage, where $\bar{r}$ is the mean reward over all $O$ samples in $\mathcal{B}_\phi$ . This centers the rewards so that above-average experiences receive positive advantages and below-average ones receive negative advantages.

What it computes: the objective increases the log-probability of tokens in experience entries that led to above-average actor success rates, and decreases the log-probability for below-average entries. The importance sampling ratio corrects for the fact that the experience entries in the buffer were generated by an older version of $\pi_\phi$ —without this correction, optimization would be biased toward the stale distribution. The clipping prevents any single token from dominating the gradient, which would cause the experience distribution to shift abruptly and destabilize the co-evolutionary loop. The denominator $\sum |m_i|$ normalizes by total token count, ensuring that longer experience entries don't disproportionately influence the update.

Why CISPO over REINFORCE: the paper explicitly states the rationale (Section 2.3):

"We adopt CISPO instead of REINFORCE to ensure stable co-evolution: the clipping mechanism constrains the IS ratio, preventing excessive policy updates that could cause the experience distribution to shift abruptly while ensuring that the gradients of all tokens are not wasted."

Vanilla REINFORCE would multiply the log-probability gradient by the raw importance ratio $\rho_{i,t}$ without clipping. In a co-evolutionary setting where both $\pi_\theta$ and $\pi_\phi$ are changing, $\rho_{i,t}$ can take extreme values—the extractor's current token distribution might diverge dramatically from the distribution that generated the training sample. These extreme ratios would produce gradient updates of wildly varying magnitudes, causing training instability. CISPO's clipping bounds this variance while the stop-gradient prevents the clipping operation itself from introducing additional gradient bias.

Why batch-level advantage instead of group-relative: the extractor's training signal comes from a buffer that accumulates samples across multiple rollout steps, potentially spanning many different tasks and actor states. Computing a group-relative advantage (as is done for the actor's GRPO objective) within each mini-batch would require maintaining paired experience-free and experience-guided rollouts for every experience entry, which is infeasible given the asynchronous accumulation scheme. The batch-level mean subtraction $r(m_i) - \bar{r}$ provides a simple centering that ensures positive rewards drive probability increase and negative rewards drive probability decrease, without requiring paired comparisons.

The Actor: Split-Subgroup GRPO with Condition-Wise Advantage Estimation

Why standard GRPO fails with experience. In principle, the actor could be trained by simply prepending retrieved experience to the context for all $K$ rollouts in each GRPO group, computing group-relative advantages across all $K$ trajectories, and applying the standard clipped surrogate objective. The paper tested this configuration and reports a critical failure (Figure 3b, Section 2.3):

"When all interactions are conditioned on retrieved experience, the actor converges prematurely and lags behind the experience-guided setting, suggesting that the actor fails to internalize experience into its own capabilities and instead develops an over-reliance on external guidance."

The problem is that if every training trajectory includes retrieved experience, the actor learns to be good at following hints, not good at solving tasks. At deployment time, if the experience bank is unavailable, performance collapses because the actor never learned to reason independently. The problem is analogous to a student who only ever solves problems with the answer key open—they may appear proficient during practice but fail the closed-book exam.

The split-subgroup design. To force the actor to develop both guided and autonomous capabilities, the paper partitions each GRPO group of $K$ rollouts evenly into two subgroups of size $K_c = K/2$ each:

Experience-guided subgroup ( $c = m$ ): these $K/2$ rollouts include retrieved experience $m$ in the actor's context, following the standard learning-from-experience paradigm.
Experience-free subgroup ( $c = \emptyset$ ): these $K/2$ rollouts are generated without any retrieved experience—the actor must rely solely on its own policy parameters and the task description.

The key question is how to compute advantages across these two subgroups. A naïve approach would pool all $K$ trajectories together, compute a single group mean and standard deviation, and normalize all advantages relative to this pooled baseline. The paper reports that this approach causes training to collapse (Figure 3c):

"The reward scales and variances differ between subgroups, causing advantage estimates to become biased and training to collapse."

The reason is subtle but concrete. Trajectories in the experience-guided subgroup have systematically different success rates than those in the experience-free subgroup—typically higher, because the guidance is helpful. If we pool them and compute a single mean $\bar{r}_{\text{pooled}}$ , then every experience-guided trajectory is evaluated relative to a baseline that includes many experience-free (and thus lower-performing) trajectories, making the guided trajectories look artificially good, while experience-free trajectories look artificially bad. The advantage estimates are contaminated by the condition membership rather than reflecting genuine within-condition performance differences. This is a form of Simpson's paradox for advantage estimation.

The condition-wise advantage solution. The paper resolves this by computing advantages independently within each subgroup (Equation 5):

$J^{\text{split}}_{\text{GRPO}}(\theta) = \mathbb{E} \left[ \frac{1}{2} \sum_{c \in \{m, \emptyset\}} \frac{1}{K_c} \sum_{k=1}^{K_c} \mathcal{L}_{\text{clip}}\left( \rho_c, \hat{A}_c \right) \right]$

where $c \in \{m, \emptyset\}$ indexes the two subgroups, $K_c = K/2$ is the number of trajectories in each subgroup, and $\rho_c = \frac{\pi_\theta(\tau_c \mid g, m_c)}{\pi_{\theta_{\text{old}}}(\tau_c \mid g, m_c)}$ is the sequence-level importance sampling ratio for the $k$ -th trajectory in subgroup $c$ (with $m_\emptyset = \emptyset$ indicating no experience conditioning).

$\hat{A}_c = (r(\tau_c) - \bar{r}_c) / \sigma_c$ is the subgroup-normalized advantage, where $\bar{r}_c$ is the mean reward of the $K_c$ trajectories within subgroup $c$ only, and $\sigma_c$ is their standard deviation. $\mathcal{L}_{\text{clip}}(\rho, A) = \min(\rho A, \text{clip}(\rho, 1 - \epsilon, 1 + \epsilon) A)$ is the standard PPO-style clipped surrogate loss with threshold $\epsilon$ .

The outer expectation averages the loss contributions from the two subgroups equally (via the $\frac{1}{2}$ factor), ensuring that neither condition dominates the gradient signal.

What it computes: for the experience-guided subgroup, the objective identifies which of the $K/2$ guided trajectories performed above the average for guided trajectories and increases their probability, while penalizing those below that subgroup-specific average. Similarly for the experience-free subgroup: trajectories that succeeded despite having no guidance receive positive advantages (relative to other unguided trajectories), and those that failed receive negative advantages. The equal weighting of subgroups means that the gradient update is balanced—the actor improves at both using guidance effectively and at performing tasks autonomously.

Why this specific design: the paper arrived at this formulation through ablation (Figures 3b–3d). The pooled-advantage design (Figure 3c) fails because it confounds condition effects with performance effects. An alternative design would be to train only on experience-free trajectories (decoupling rollout from optimization, as in Zhai et al., 2025), but this loses the benefit of having the actor practice with guidance during training. The split-subgroup design with within-condition advantages preserves both signals cleanly: experience-guided rollouts teach the actor to extract useful information from guidance, while experience-free rollouts teach it to internalize that information into standalone capability. Figure 3d confirms this: after adopting within-condition advantages, both subgroups show consistent improvement, whereas Figure 3c had shown collapse.

GRPO hyperparameters. Across all experiments, the paper uses GRPO with group size $K = 8$ , a clip ratio $\epsilon = 0.2$ , and a constant learning rate of $1 \times 10^{-6}$ with the AdamW optimizer. No KL regularization is applied to either $\pi_\theta$ or $\pi_\phi$ . The rollout batch size varies by experiment: 128 for MiniHack Room (single-task), 64 for WebShop, 128 for ALFWorld, and 384 total (128 per task) for multi-task training. Micro-batch sizes for $\pi_\theta$ are 64 (MiniHack), 16 (WebShop), 32 (ALFWorld), and 96 (multi-task).

The Asynchronous Training Infrastructure

The concurrency problem. Complementary RL's co-evolutionary design creates a practical engineering challenge: the actor $\pi_\theta$ needs retrieved experience $m$ before it can start an episode, and the extractor $\pi_\phi$ needs the completed trajectory to distill new experience and receive training signals. These are mutual dependencies that, if handled naïvely, would serialize the entire training pipeline:

"A naïve implementation would serialize these dependencies, where after each batch of rollouts, actor training would block while waiting for experience distillation and $\pi_\phi$ optimization to complete, introducing synchronization barriers that cause significant resource idleness and degrade overall training throughput."

In a typical industrial RL training setup, hundreds of environment instances run in parallel to generate rollouts. If each batch of rollouts must then wait for a full distillation-and-extractor-update cycle (which involves LLM forward passes over all completed trajectories), the GPUs running actor inference would sit idle for the majority of wall-clock time. The paper's solution is a fully asynchronous design with two independent execution tracks that coordinate through a centralized experience manager.

Primary training loop. The actor $\pi_\theta$ runs in a continuous cycle that is never blocked by distillation: at the start of each episode, the environment submits a Search request to $\mathcal{H}$ using the task description as query $q$ → $\mathcal{H}$ retrieves the most relevant experience $m$ (see Section 3.2.2 below) → the actor generates rollouts, with $K/2$ receiving $m$ in context and $K/2$ receiving none → trajectories complete and are batched for $\pi_\theta$ optimization via Equation 5 → $\pi_\theta$ is updated immediately, without waiting for any distillation work. Concurrently, each completed trajectory (regardless of actor training status) is submitted as a distillation request to $\mathcal{H}$ and enters queue $\mathcal{Q}$ .

Background track. Running independently on separate compute resources, a background process continuously dequeues pending requests from $\mathcal{Q}$ , forwards each to $\pi_\phi$ for distillation, applies the returned operations to $\mathcal{M}$ , and stores experience-reward pairs in buffer $\mathcal{B}_\phi$ . $\pi_\phi$ optimization is triggered only when $|\mathcal{B}_\phi|$ reaches the full training batch size of 64, which may span multiple actor rollout steps. Crucially:

" $\pi_\phi$ and $\pi_\theta$ are optimized on fully independent schedules, ensuring neither blocks nor interferes with the other throughout co-evolution training."

This independence means that the actor can be updated hundreds of times between extractor updates—or the extractor can be updated multiple times per actor update—depending on the relative speeds of rollout collection versus distillation processing. The training buffer $\mathcal{B}_\phi$ acts as a decoupling FIFO queue, smoothing out mismatches in processing rates.

The Experience Manager ( $\mathcal{H}$ ). The centralized coordinator that makes this asynchronous design safe is the ExperienceManager, which handles two critical functions requiring concurrency control:

Experience Consolidation (Section 3.2.1): all writes to $\mathcal{M}$ (Add, Update operations from $\pi_\phi$ ) are serialized under a writer lock:

"Upon receiving the issued operations from $\pi_\phi$ , $\mathcal{H}$ applies them to $\mathcal{M}$ under a writer lock, which temporarily suspends concurrent reads to prevent state conflicts."

This ensures that no retrieval query sees a partially updated bank—a reader either sees the pre-write snapshot or the post-write snapshot, never an intermediate state. For each newly added entry, the embedding model $f_\psi$ generates its dense vector $v_m = f_\psi(m)$ , and the tuple $(m, v_m, \text{prompt-response-pair})$ is persisted to $\mathcal{M}$ .

Experience Retrieval (Section 3.2.2): all reads from $\mathcal{M}$ proceed under a reader lock, which allows concurrent reads but blocks writes:

"Each [parallel search worker performs] semantic similarity search over $\mathcal{M}$ under a reader lock, allowing concurrent reads while blocking writes."

This is a standard readers-writer lock: any number of search queries can execute simultaneously (they don't modify state), but when a write arrives, all reads must complete before the write acquires the exclusive lock.

Query batching and caching. To maximize retrieval throughput, $\mathcal{H}$ does not process queries one at a time. It accumulates incoming Search requests into a waiting buffer until either a predefined batch size $B = 16$ is reached or a maximum waiting time $t_{\text{max}} = 0.001$ seconds elapses. Each query is first checked against an embedding cache $\mathcal{C}$ , which is particularly effective in group-based RL where many parallel environments share identical task descriptions—the first environment to request "navigate MiniHack Room 5x5" triggers an embedding computation; all subsequent identical requests hit the cache. Cache misses are forwarded to $f_\psi$ for batched embedding computation (processing 16 queries at once rather than 16 sequential forward passes). The resulting embedding vectors are distributed via round-robin scheduling to one of $W = 4$ parallel search workers, each performing nearest-neighbor search over $\mathcal{M}$ independently.

Periodic merge. The consolidation process described so far treats each episode independently: every trajectory potentially produces a new Add or Update operation. In group-based RL, where multiple parallel environments run identical tasks, this can produce redundant or conflicting entries—for example, two episodes on MiniHack Room might both produce experience entries about "how to avoid traps in dark rooms" with slightly different phrasings. Such redundancy degrades retrieval quality: the semantic search returns near-duplicate results that crowd out genuinely distinct advice. The paper confirms this degradation empirically (Figure 5a):

"Such redundancy degrades the quality of semantic retrieval and consequently impairs the actor's learning."

To mitigate this, the paper introduces a periodic Merge operation that runs every 5 actor update steps. The entire experience bank $\mathcal{M}$ is processed in chunks that fit within $\pi_\phi$ 's context window. Each chunk is passed to $\pi_\phi$ with a structured prompt instructing it to analyze semantic relationships among entries and decide which to keep, merge, or discard. The output of processing one chunk is carried forward and concatenated with the next chunk, forming a sliding window over the full bank:

"This design bounds the context length presented to $\pi_\phi$ while ensuring all entries are considered, yielding a compact experience bank that benefits actor learning."

Figure 5a shows that removing the merge operation causes a noticeable performance drop in later training (after step 75), confirming that curation is important for scaling the experience bank over many update cycles.

Search and Ask. A subtle limitation of the basic retrieval design is that using the task description alone as the query $q$ tends to retrieve the same experience entry repeatedly in group-based RL:

"Parallel environments in group-based RL training often share identical task descriptions or differ only in environment-specific details such as map layouts. This reduces the utilization of $\mathcal{M}$ and limits the diversity of training signal available for optimizing $\pi_\phi$ ."

To address this, the paper introduces a search and ask tool that the actor can invoke at any decision step during environment interaction (not just at episode start). When invoked, the actor constructs a context-aware query $q'$ summarizing its current state and specific difficulty, and submits it to $\mathcal{H}$ for retrieval. If a relevant entry $m$ is found, the pair $(q', m)$ is forwarded to $\pi_\phi$ , which refines $m$ according to the actor's described situation before returning it. This mechanism has dual benefits: it increases the diversity of retrieved experiences (and thus the richness of the extractor's training signal), and it gives the actor access to situationally targeted guidance—not just "how to solve this task in general" but "how to solve this task given that I'm currently stuck in this corridor."

Figure 5b confirms that ablating the search and ask tool significantly reduces performance, particularly in mid-to-late training (after step 75), demonstrating that on-demand refined retrieval provides value beyond the initial episode-level experience.

Stabilization Techniques for Extractor Training

The paper identifies two compounding challenges that make extractor training highly unstable (Appendix B), and introduces two specific countermeasures. These are not the primary contributions but are essential for making co-evolution work in practice.

Challenge 1: Severe off-policyness. An experience entry $m$ may be generated by $\pi_\phi$ at training step $t$ , added to $\mathcal{M}$ , and then retrieved to guide an episode many steps later—potentially after both the actor and extractor have been updated multiple times. The importance sampling ratio $\rho_{i,t}$ in the CISPO objective corrects for this distribution shift, but when the gap between generation time and training time is large, $\rho$ can take extreme values even with clipping:

"Since retrieval timing is uncontrolled, a distilled experience $m$ may be retrieved long after it was generated, introducing a large policy lag between the retrieving actor and the current $\pi_\phi$ ."

The CISPO clipping addresses the symptom (extreme gradient magnitudes), but the underlying cause is that certain popular experiences dominate the retrieval distribution, starving other experiences of training signal.

Challenge 2: Data redundancy. In environments with low-diversity task descriptions, a single experience entry $m$ may be retrieved across many training buffer steps:

"When task descriptions exhibit low diversity, a single experience $m$ tends to be retrieved repeatedly across different training buffer steps, causing data redundancy and $\pi_\phi$ may be updated multiple times on the same experience $m$ , severely undermining training stability."

Optimizing on the same sample repeatedly causes the extractor to overfit to it—the CISPO objective drives $\pi_\phi$ to assign ever-higher probability to that particular entry's token sequence, destabilizing the distribution.

Countermeasure 1: Retrieval Diversification. Instead of retrieving the top- $K$ most relevant experiences for a query $q$ , the paper oversamples by drawing $N$ independent candidate sets of size $K$ each, yielding a pool of $N \times K$ candidates $\mathcal{C}(q)$ . The pool is then re-ranked according to a diversity-aware scoring function that penalizes frequently retrieved experiences:

$s(m) = s_{\text{rank}}(m) - \lambda \cdot \log(1 + c(m)) - \mathbb{1}[\text{recent}(m)]$

where $s_{\text{rank}}(m)$ is the base relevance rank score of experience $m$ (its raw semantic similarity to $q$ ), $c(m)$ is the historical count of how many times $m$ has been retrieved across all queries so far, $\lambda$ is a penalty hyperparameter controlling retrieval diversity, and $\mathbb{1}[\text{recent}(m)]$ is an indicator function that penalizes experiences retrieved within a predefined recency window.

What it computes: a modified relevance score where frequently-used and recently-used experiences are penalized. The $\log(1 + c(m))$ term grows sublinearly with retrieval count—the first few retrievals incur a small penalty, but very heavily used entries are strongly penalized. The recency indicator adds a binary penalty for entries that were just retrieved, preventing the same entry from dominating consecutive queries.

Why this form: the logarithmic penalty ensures that the diversification effect is progressive rather than binary—an experience that has been used twice is not treated equivalently to one used a hundred times. The base rank term $s_{\text{rank}}$ ensures that relevance is not sacrificed entirely for diversity; the penalty nudges the ranking toward underutilized entries that are still reasonably relevant. Without this diversification, the top-ranked entry for a common task description would be retrieved for every parallel environment running that task, concentrating all extractor training signal on a single entry and starving the rest of the bank.

Countermeasure 2: Training-Count-Aware Advantage Reweighting. After computing the advantage $\hat{A}_i$ for each experience entry in the training buffer $\mathcal{B}_\phi$ , the paper applies a per-experience weight $w(m)$ that penalizes entries that have been optimized on too many times:

$w(m) = \begin{cases} 0 & \text{if } (t - t_{\text{last}}) < \delta, \\ (1 + c_{\text{train}}(m))^{-\alpha} & \text{otherwise} \end{cases}$

where $t$ is the current global training step, $t_{\text{last}}$ is the most recent step at which $m$ was included in a training batch, $\delta$ is a cooldown window that completely suppresses gradient updates from experience entries optimized too recently, $c_{\text{train}}(m)$ is the cumulative number of times $m$ has been used for training $\pi_\phi$ , and $\alpha \geq 0$ is a decay exponent controlling how aggressively the advantage is discounted as training count accumulates.

What it computes: a multiplicative weight applied to the advantage $\hat{A}_i$ before it enters the CISPO objective. If $m$ was trained on in the last $\delta$ steps, its weight is zero—it contributes nothing to the current update. Otherwise, its advantage is scaled by $(1 + c_{\text{train}})^{-\alpha}$ , which decays toward zero as $c_{\text{train}}$ grows.

Why this form: the cooldown (the zero-weight branch) prevents the most extreme form of redundancy—immediately optimizing on the same entry in consecutive updates, which would cause the extractor to chase its own tail. The count-based decay provides a softer penalty for entries that are used repeatedly but not immediately consecutively. The decay exponent $\alpha$ controls the aggressiveness: $\alpha = 0$ gives uniform weighting (no penalty), while large $\alpha$ rapidly down-weights any entry that has been trained on more than a few times. Together, these two mechanisms ensure that the extractor receives a balanced diet of diverse training samples with varying retrieval histories, preventing any single experience entry from dominating optimization.

The Actor-Critic Extension (Optional Component)

The paper also describes an optional mechanism called Actor-Critic (Appendix B.1) that introduces explicit communication between $\pi_\theta$ and $\pi_\phi$ to handle quality issues with retrieved experience. While not used by default due to latency overhead, it is included for practitioners who prioritize final performance over training throughput.

Motivation: two failure modes of retrieved experience. The paper identifies specific circumstances where retrieved experience can harm rather than help:

Experience Staleness: when the actor has already mastered a given task, retrieved experience may be overly conservative or incorrect relative to its current capability. For example, an early-training experience entry that says "check every room exhaustively" becomes wasteful when the actor has learned to identify promising rooms at a glance.
Experience Imprecision: when the actor's success rate is low, retrieved experiences are directionally helpful but may require adaptation to the specific task context—they provide roughly correct strategies that don't precisely match the current situation.

The critic mechanism. Before the main dual training loop, the actor runs for $T_{\text{warm}}$ warm-up iterations to estimate its initial average success rate $\bar{r}_\theta$ . Once training begins, after each retrieval of experience $m$ for a task query $q$ , the actor $\pi_\theta$ is prompted to reflect on $m$ in light of both the current task and its accumulated success rate $\bar{r}_\theta(q)$ , and produces one of three critic actions:

accept: the experience is used as-is; receives a critic score $s_c(m) = 1$ .
refine: the experience is edited using the actor's own knowledge to better fit the current task context; receives $s_c(m) = 0.5$ .
reject: the experience is discarded entirely; receives $s_c(m) = 0$ .

The critic score $s_c(m)$ is combined with the task completion reward $r(m)$ to form an enriched learning signal for the experience extractor:

$\tilde{r}(m) = s_c(m) + r(m)$

where $\tilde{r}(m)$ now ranges from $-1$ (rejected experience that also led to failure) to $+2$ (accepted experience that led to success), providing $\pi_\phi$ with a more nuanced signal than binary outcome alone.

What it computes: a modified reward that incorporates the actor's own assessment of experience quality alongside the objective task outcome. An experience entry that the actor correctly identifies as helpful and that leads to success gets $+2$ ; one that the actor correctly rejects and that would have been harmful gets $-1$ ; one that the actor mistakenly accepts but leads to failure gets $0$ (the $+1$ critic acceptance and $-1$ outcome cancel).

Why this helps: the critic provides $\pi_\phi$ with a denser training signal. Without it, $\pi_\phi$ can only infer experience quality from binary task outcomes, which conflate experience quality with actor execution skill. With the critic, $\pi_\phi$ receives immediate feedback on whether its experience was well-calibrated to the actor's current capability level—even if the episode succeeds for unrelated reasons, a rejected experience signals to $\pi_\phi$ that its distillation strategy is producing outputs that the actor finds unhelpful.

The paper reports that Actor-Critic yields improved success rates, particularly in early training on MiniHack Room (Figure 14a). However, it also introduces latency: the actor must produce a critic decision before each environment interaction, blocking rollout collection pending the critic response. The average search time increases from 0.04 seconds to 1.21 seconds (Figure 14b). The paper therefore does not adopt Actor-Critic as a default component, recommending it only when final performance is prioritized over throughput.

Implementation Details and Scale

Model configurations. The default setup uses Qwen2.5-7B-Instruct (Qwen et al., 2025) as the actor $\pi_\theta$ and Qwen3-4B-Thinking-2507 (Yang et al., 2025) as the experience extractor $\pi_\phi$ . The choice of a 4B thinking model for the extractor is deliberate: the extractor must perform structured reasoning over long trajectories (up to 65,536 tokens) to distill transferable patterns, and the thinking architecture's chain-of-thought capability supports this synthesis. For the larger-extractor ablation (Section 4.3), the paper uses Qwen3-30B-A3B-Instruct-2507 as $\pi_\phi$ . For the SWE-Bench experiment, the actor is upgraded to Qwen3-4B-Instruct-2507. The embedding model $f_\psi$ is Qwen3-Embedding-0.6B, a lightweight 600M-parameter dense retriever that balances embedding quality with inference speed.

Infrastructure scale. The system runs 4 parallel search workers and 4 parallel embedding workers. The query batch size $B$ is set to 16, with maximum waiting time $t_{\text{max}} = 0.001$ seconds—in practice, this means queries are almost always batched because 16 parallel environments quickly fill the buffer, and the 1-millisecond timeout serves only as a deadlock prevention for low-concurrency scenarios. The extractor training buffer size is $|\mathcal{B}_\phi| = 64$ , and the periodic merge interval is 5 actor update steps. Both clip thresholds for CISPO are $\varepsilon^{\text{IS}}_{\text{low}} = \varepsilon^{\text{IS}}_{\text{high}} = 0.1$ .

Sequence length limits. The maximum sequence length is 32,768 tokens for $\pi_\theta$ (16,384 for WebShop and ALFWorld), 65,536 tokens for $\pi_\phi$ , and 4,096 tokens for maximum output per step. The maximum number of interaction turns varies by environment: 30 for MiniHack Room and multi-task training, 40 for ALFWorld, and environment-specific limits for WebShop and SWE-Bench.

Rollout latency (Figure 9c). To verify that the asynchronous infrastructure doesn't add overhead, the paper measures average rollout collection time across different rollout batch sizes (128, 256, 512, 1024 parallel environments) and compares against a baseline without experience integration. Across all settings, Complementary RL's rollout time is within 1–2% of the baseline—the ratios are 0.98×, 0.99×, 1.00×, and 0.99× for the four batch sizes, meaning the framework introduces no appreciable latency. The average search time per step (Figure 12) remains under 1 second even at 1024 parallel environments, confirming that the query batching, caching, and parallel search design successfully absorbs the retrieval workload.

4. Key Insights and Innovations

Innovation 1: Co-Evolution as a Paradigm Shift from Static Memory to Mutual Optimization

The paper's most fundamental intellectual contribution is the reframing of an agent's experience extractor from a static tool into a co-learner inside a closed optimization loop. Prior to this work, essentially all experience-driven RL for LLM agents treated the experience pipeline as a fixed component: either a pre-built offline bank (Zhai et al., 2025) or an online-updatable bank maintained by a frozen extractor (Zhang et al., 2026a; Ouyang et al., 2025). The implicit assumption across this literature was that improving experience quality—through better data structures, more sophisticated distillation prompts, or multi-stage retrieval heuristics (Zhou et al., 2025)—would translate to better actor learning. What these approaches failed to recognize, and what Complementary RL diagnoses, is that quality is relative to capability. Experience that is optimal for a novice agent becomes stale, then wasteful, then actively harmful for an expert agent. A fixed-quality experience pipeline therefore has a built-in expiration date.

The conceptual move that distinguishes Complementary RL is to treat the experience extractor as a reinforcement learner in its own right, optimized for a meta-objective: produce experience entries that demonstrably increase the actor's task success rate. This is not simply "training a better extractor" in the sense of a larger model or more careful prompting (which the paper itself tests via the 30B-A3B extractor ablation in Figure 9a, yielding only incremental gains). It is a structural change to the optimization topology: the extractor receives an RL training signal (the CISPO objective, Equation 3) whose reward depends on downstream actor performance, creating a feedback loop where the extractor's distillation strategy adapts based on which kinds of experience the current actor finds useful. The actor's growing competence reshapes what the extractor learns to distill; the extractor's improving outputs accelerate the actor's learning. Each shapes the other toward greater capability.

This is a fundamental paradigm shift, not an incremental refinement. The difference is analogous to the transition from hand-crafted features to learned representations in computer vision: prior work treated experience extraction as an engineering problem (design better templates, curate better banks), while Complementary RL treats it as a learning problem (optimize an extractor for the downstream utility of its outputs). The paper makes this shift explicit by drawing the analogy to Complementary Learning Systems in neuroscience (O'Reilly et al., 2011), but the contribution is not the biological inspiration—it is the operationalization of bidirectional feedback between memory formation and skill acquisition within an RL framework.

The empirical evidence for why this shift matters is captured in Figure 3a and Table 1, but the conceptual point is independent of the specific performance numbers. A static online experience bank (where the bank updates but the extractor is frozen) performs only marginally better than no experience at all—the extractor's fixed distillation strategy cannot track the actor's evolving state-action distribution. The extractor keeps mining for patterns relevant to early exploration long after the actor has moved on to exploitation. Co-evolution eliminates this mismatch by design: the extractor's objective is tied to the actor's current success, making staleness impossible in principle because the extractor is rewarded for producing experience that helps the current actor, not some earlier version.

Innovation 2: The Split-Subgroup Advantage Estimation as a Solution to the Internalization Problem

The second intellectual contribution is the diagnosis and resolution of the "over-reliance on external guidance" problem through a specific advantage estimation design. The paper identifies a subtle but critical failure mode that arises when an agent is trained exclusively on experience-guided rollouts: the agent becomes proficient at following hints but fails to develop autonomous capability, collapsing when experience is unavailable at test time. This is not merely a training artifact; it is a structural consequence of the standard GRPO objective when all $K$ rollouts in a group share the same conditioning.

The standard GRPO objective computes group-relative advantages across all $K$ trajectories in a batch—each trajectory's reward is compared to the batch mean. When all $K$ trajectories include retrieved experience, the actor is never evaluated (and thus never optimized) on its standalone capability. The gradient signal says "be better at using guidance" but says nothing about "be better at solving tasks without guidance." The result is an agent that appears competent during training (where guidance is available) but fails the closed-book exam of deployment.

The paper's solution—partitioning rollouts into experience-guided and experience-free subgroups with within-subgroup advantage estimation (Equation 5)—is conceptually significant not because of the specific partitioning ratio (K/2 each) but because it preserves two distinct learning signals without contamination. Experience-guided rollouts teach the actor to extract useful information from context; experience-free rollouts teach it to rely on its own parameters. The within-subgroup normalization ensures that a trajectory's advantage reflects its performance relative to other trajectories under the same conditioning, not relative to trajectories from a different (and typically easier or harder) subgroup.

This may appear like a minor implementation detail—a K/2 split instead of K unified—but it resolves a deep statistical problem that the paper demonstrates empirically. Figure 3c shows that pooling both subgroups under a single advantage computation causes training to collapse because reward scales and variances differ between conditions: guided trajectories have systematically different success rates than unguided ones, so a pooled baseline contaminates both subgroups' advantages with condition membership effects. This is a form of Simpson's paradox for advantage estimation: the condition (guided vs. unguided) confounds the within-condition performance signal, producing biased gradient updates that destabilize training.

The broader significance of this contribution is that it establishes a design principle for multi-condition RL training: when an agent is trained under multiple context conditions (with and without retrieval, with and without tools, with and without demonstrations), advantage estimation must respect condition boundaries to preserve signal integrity. This principle applies beyond experience-driven RL to any setting where the agent's training data includes heterogeneous context types with different expected reward distributions. Prior work on decoupling rollout collection from optimization (Zhai et al., 2025) attempted to solve a related problem by training only on experience-free trajectories, but this abandons the benefit of guided practice entirely. The split-subgroup approach offers a more nuanced solution: practice with guidance, but evaluate and optimize within condition-consistent reference groups.

The evidence for this innovation's effectiveness is in Figures 3b–3d and Table 1. The split-subgroup design with within-condition advantages yields consistent improvement in both subgroups (Figure 3d), whereas the pooled-advantage design collapses (Figure 3c). More importantly, Table 1 shows that Complementary RL-trained agents retain a 2% average improvement over the baseline even when evaluated without retrieved experience at test time, confirming that the split-subgroup design successfully internalizes useful guidance into the policy parameters. This internalization—the conversion of external scaffolding into intrinsic capability—is the paper's concrete answer to the problem that "simply providing experience at test time is not learning."

Innovation 3: Verifier Over-Optimization as a First-Class Phenomenon in Experience Selection

A more subtle but equally important conceptual contribution is the paper's characterization of experience quality from the actor's perspective as a dynamic, capability-relative property, and its operationalization through the extractor's reward signal. This insight is embedded in the design choice to reward the extractor based on actor outcomes rather than intrinsic experience properties.

Prior work on experience extraction for LLM agents (Xia et al., 2026; Yan et al., 2025; Zhang et al., 2025a) evaluated experience quality through intrinsic criteria: Is the experience syntactically well-formed? Does it accurately summarize the trajectory? Does it provide actionable advice? These are necessary conditions for useful experience but not sufficient—an experience entry can be perfectly accurate and well-structured yet useless (or harmful) to an agent that has already internalized its lesson. The extractor in Complementary RL is not rewarded for producing "good experience" in the abstract; it is rewarded specifically for producing experience that demonstrably increases the probability of the current actor succeeding on tasks.

This is a subtle but profound shift. It means that the same trajectory, processed by the same extractor at two different points in training, should ideally produce different experience entries—not because the trajectory changed, but because the actor's needs changed. An early-training entry might emphasize basic environmental rules ("traps are represented by the ^ symbol"); a late-training entry might emphasize efficiency strategies ("skip systematic room-checking when the goal is visible"). The paper's CISPO objective (Equation 3) enables this adaptation by optimizing $\pi_\phi$ toward experience generation strategies that receive positive rewards (actor success) and away from strategies that receive negative rewards (actor failure), where the reward distribution shifts as the actor improves.

The conceptual contribution here is the relational definition of experience utility: utility is not a property of the experience content alone but of the match between the experience and the agent's current capability state. This parallels the concept of the Zone of Proximal Development in educational psychology—instruction is most effective when it targets skills just beyond the learner's current mastery. The paper doesn't explicitly invoke this analogy, but the mechanism is identical: the extractor learns to identify and distill experience at the frontier of the actor's capability, where guidance can bridge the gap between current performance and potential performance.

This insight has implications beyond the specific Complementary RL framework. It suggests that future work on memory-augmented agents should evaluate experience utility dynamically rather than statically: a retrieval mechanism that simply returns the most "relevant" experience (by semantic similarity) may be suboptimal compared to one that returns experience calibrated to the agent's demonstrated competence on similar tasks. The paper doesn't develop this into a full theory of capability-relative retrieval, but the extractor reward design embeds the principle into the optimization loop.

The empirical support for this innovation is indirect but pervasive. The poor performance of Static Online Exp. in Figure 3a and Table 1—where the bank updates but the extractor's distillation strategy remains frozen—demonstrates that freshness of data alone cannot compensate for a static extraction strategy. The extractor's inability to adapt its criteria for what constitutes useful experience (as opposed to simply adding new instances of the same type of experience) produces diminishing returns. The fact that co-evolution outperforms static alternatives by substantial margins (+7% on average in multi-task evaluation with experience at test time) validates the dynamic-utility perspective.

Innovation 4: Asynchronous Infrastructure Design as an Enabling Contribution for Co-Evolutionary Training at Scale

The fourth contribution is the fully asynchronous training architecture that makes co-evolutionary RL practical at scale. While infrastructure design is often treated as engineering rather than research, in this case the design choices are conceptually motivated and nontrivially enable the algorithm's core claims. The mutual dependencies between $\pi_\theta$ and $\pi_\phi$ —the actor needs retrieved experience to generate rollouts, the extractor needs completed rollouts to distill experience—create a circular dependency that, if handled naïvely, would serialize the entire training pipeline into a blocking sequence: collect rollouts → wait for distillation → update extractor → update actor → repeat. This would reduce training throughput to a fraction of what modern distributed RL systems achieve, making co-evolutionary training practically infeasible for all but the smallest experiments.

The paper's solution—decoupling the primary training loop (actor interaction and optimization) from the background track (experience distillation and extractor optimization) through a centralized ExperienceManager with explicit concurrency control—is architecturally novel in the context of LLM agent RL. Prior systems for memory-augmented agents (Li et al., 2025; Salama et al., 2025) either ran distillation and retrieval synchronously (accepting the throughput penalty) or used simple key-value stores with fixed update schedules (avoiding the concurrency problem by not having a learned extractor). The ExperienceManager's reader-writer lock protocol, query batching with embedding caching, and producer-consumer distillation queue are individually standard distributed systems techniques, but their combination into a unified coordinator for dual-model co-evolution is a new infrastructure pattern.

The significance lies in what this architecture enables rather than the specific implementation. The latency measurements in Figure 9c demonstrate that the asynchronous design introduces no appreciable overhead to rollout collection across batch sizes from 128 to 1024 parallel environments—the ratios are 0.98–1.00× relative to a baseline without any experience integration. Similarly, the average search time per step (Figure 12) remains under 1 second even at the largest scale. This means that co-evolutionary training achieves its accuracy gains (10% in single-task, 7% in multi-task) without sacrificing training throughput—a claim that would be impossible to verify without the careful infrastructure measurements.

This contribution is incremental in distributed systems terms but fundamental for the research agenda it enables. Without the asynchronous architecture, scaling Complementary RL to multi-task settings with hundreds of parallel environments and 65K-token trajectories would be impractical, and the paper's most compelling results (multi-task training on 3–6 environments, task scaling in Figure 9d) would not exist. The infrastructure is therefore best understood as an enabling contribution: it doesn't advance the theoretical understanding of co-evolution, but it converts a theoretically appealing idea into an empirically demonstrable system. The paper's decision to release the training framework and demo (as noted in the abstract footnote) further supports treating this as an infrastructure contribution that lowers the barrier for the community to adopt co-evolutionary training.

Innovation 5: The "Experience Is a Dynamic Resource" Diagnostic as a Unifying Explanation for Conflicting Prior Results

The fifth contribution is more conceptual than technical: the paper provides a unified diagnostic framework that explains why prior work on experience-guided agent RL produced inconsistent results, and why co-evolution resolves the inconsistency. This is not a standalone algorithm or method but rather a causal explanation that organizes the space of prior approaches and predicts when each will succeed or fail.

The paper's landscape can be summarized as a two-axis diagnosis. The first axis is extraction strategy: is the extractor frozen or learning? The second axis is experience freshness: is the bank offline (pre-built) or online (updated during training)? Prior work occupied three of the four quadrants: offline + frozen (standard experience bank approaches), online + frozen (static online experience), and attempting online + learning but decoupled from actor feedback (quality-centric approaches that optimize the extractor independently). The missing quadrant—online + learning + actor-coupled—is Complementary RL.

The diagnostic power of this framework is that it predicts the degradation patterns observed across prior work and in the paper's own ablations. Frozen extractors (both quadrants) suffer from distributional misalignment: the extraction strategy is calibrated to some fixed actor capability and becomes progressively less relevant. Offline banks (regardless of extractor) suffer from data staleness: the trajectories were generated by an older policy and may not cover the state distribution of the current policy. Only the online + learning + actor-coupled quadrant avoids both failure modes, because the extractor adapts its strategy based on current-actor outcomes (eliminating distributional misalignment) and processes current-actor trajectories (eliminating data staleness).

This diagnostic framework is significant not because it proposes a new method but because it converts a confusing set of contradictory empirical findings into a coherent explanatory structure. Prior work had shown that experience helps in some settings (Zhai et al., 2025; Zhang et al., 2026a) and fails in others (the paper's own Static Online Exp. results). Without the diagnostic framework, these appear as contradictory findings—does experience help or not? The framework clarifies that the answer depends on how the experience pipeline interacts with the learning actor: fixed pipelines provide diminishing returns; co-evolving pipelines provide sustained benefits. The specific empirical support is in Figure 3a (showing the progressive decay of offline experience benefits), Table 1 (showing static online experience underperforming the baseline by >10%), and Figures 6 and 8 (showing sustained gains from co-evolution across tasks and scales).

This contribution is fundamental in its reframing of the problem space, but its long-term impact depends on whether the community adopts this diagnostic lens when designing future experience-driven systems. The paper provides the conceptual vocabulary ("distributional misalignment," "progressive staleness," "capability-relative utility") to reason about when and why experience mechanisms succeed or fail, which is arguably more valuable than any single algorithm because it guides future research design rather than prescribing a specific solution.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on four open-ended agentic environments: MiniHack (Samvelyan et al., 2021)—specifically MiniHack-Room-Ultimate-5x5-v0 for single-task and MiniHack-Room-Ultimate-5x5-v0, MiniHack-MazeWalk-9x9-v0, MiniHack-KeyRoom-Dark-S5-v0, and MiniHack-River-Narrow-v0 for multi-task scaling experiments; WebShop (Yao et al., 2023) with a 1,000-item product catalog and weighted instruction-pool goal sampling; ALFWorld (Shridhar et al., 2021) with 1,466 training instances and 134 held-out evaluation instances; SWE-Bench-Verified (Jimenez et al., 2024) filtered to retain only tasks where a preliminary pass@16 evaluation of Qwen3-4B-Instruct-2507 yields success rates in the range (0, 80%), producing a curated set of 124 tasks; and Sokoban configured as 6×6 rooms with two boxes and two target positions. All environments use binary reward schemes—success = 1, failure = 0—except ALFWorld, which assigns −1 for failure and 1 for success.
Base model(s). The default actor is Qwen2.5-7B-Instruct (Qwen et al., 2025), chosen as a representative mid-scale model with non-trivial but far-from-saturated performance on the benchmark tasks. The default experience extractor is Qwen3-4B-Thinking-2507 (Yang et al., 2025), selected for its chain-of-thought reasoning capability that supports structured distillation over long trajectories (up to 65,536 tokens). For the SWE-Bench experiment specifically, the actor is upgraded to Qwen3-4B-Instruct-2507. The larger-extractor ablation uses Qwen3-30B-A3B-Instruct-2507. The embedding model for semantic retrieval is Qwen3-Embedding-0.6B, a 600M-parameter dense retriever served via vLLM.
Metrics. The paper tracks success rate (for MiniHack and WebShop) and reward on held-out evaluation sets (for ALFWorld and SWE-Bench), with higher values indicating stronger agent performance. Additionally, it reports average number of actions per task (Figure 7), where fewer actions at equal or higher success indicates more efficient decision-making. Rollout latency is measured as average rollout collection time (Figure 9c) and average search time per training step (Figure 12) in seconds, comparing Complementary RL against a baseline without experience integration across varying parallel environment counts.
Baselines. The paper evaluates against multiple baselines: (1) Baseline—standard GRPO actor training using only outcome rewards without any experience retrieval; (2) Static Online Exp.—a frozen experience extractor dynamically maintains and constructs M during training but is never optimized, isolating the effect of extractor co-evolution; (3) Exp. Only—the extractor πϕ is trained to maintain and refine M, but the actor πθ is held fixed, isolating the effect of actor co-evolution; (4) Offline Exp.—M is pre-constructed from prior trajectories using an external large model (Qwen3-30B-A3B-Instruct-2507) and remains static throughout RL training (used only in the pilot study, Figure 3a). Methods that leverage experience are evaluated under two settings: with retrieved experience at test time (w/ exp.) and without (w/o exp.), the latter measuring whether the actor has internalized guidance into its parameters.
Generation budget / compute accounting. The primary compute unit is one complete rollout (a full interaction trajectory from episode start to termination). For fair comparison, all methods use identical rollout batch sizes and group sizes within each experimental configuration. The paper measures efficiency through learning curves (success rate vs. training steps) rather than FLOPs or token counts, since all methods share the same actor architecture and differ only in the presence and nature of experience extraction overhead. The asynchronous infrastructure's latency is evaluated by comparing average rollout collection time and average search time against a baseline without experience integration at identical rollout batch sizes (128, 256, 512, 1024 parallel environments; Figures 9c and 12).
Cross-validation / statistical protocol. The paper does not employ formal cross-validation for strategy selection (since the core algorithm does not involve hyperparameter search over experience strategies). Instead, it reports learning curves over full training runs (typically 125–150 steps) on fixed evaluation sets, with success rate or reward measured at regular intervals. For the multi-task main results (Table 1), final evaluation performance is reported on held-out test instances. The pilot study (Figure 3a) runs each variant for 145 steps with a fixed random seed; other experiments follow this convention. No confidence intervals or error bars are reported across multiple seeds, which is a limitation for assessing the statistical reliability of specific performance gaps (e.g., the +7% average improvement in Table 1).

Main Quantitative Results

Pilot Study: Why Static Experience Fails (Figure 3a)

The paper opens its experimental analysis with a controlled pilot study on MiniHack Room (MiniHack-Room-Ultimate-5x5-v0) that empirically motivates the entire co-evolutionary design. Three conditions are compared over 145 training steps with a rollout batch size of 128, group size K = 8, and clip ratio ε = 0.2:

Baseline (no experience): the actor πθ is trained via GRPO using only binary outcome rewards without any experience retrieval. This establishes the lower bound for standalone trial-and-error RL on this task.
Offline Exp.: an experience bank M is pre-constructed offline by running Qwen2.5-7B-Instruct on MiniHack Room for up to 30 interaction turns, routing the resulting trajectories to Qwen3-30B-A3B-Instruct-2507 for experience distillation, and applying merging and semantic deduplication. This fixed bank is used for retrieval throughout training but never updated.
Static Online Exp.: M is dynamically maintained by a frozen Qwen3-4B-Thinking-2507 extractor during training—new trajectories are distilled and added to the bank in real-time, but the extractor's parameters are never updated.

The learning curves in Figure 3a reveal three distinct patterns. Offline Exp. provides a clear initial performance advantage over the baseline in early training (approximately steps 0–50), with the agent climbing faster due to useful hints from the pre-built bank. However, this advantage progressively decays: by approximately step 100, the gap narrows substantially, and by step 150 the two curves nearly converge. The paper attributes this to distributional misalignment—the bank was distilled from trajectories generated by an earlier, less capable version of the agent, and its fixed entries become progressively less relevant as the actor's policy improves and visits novel states. Static Online Exp. performs even worse than the offline variant: it yields only marginal gains over the baseline throughout training, never achieving the early advantage of the offline bank and failing to separate from the baseline curve at any point. The paper's diagnosis is that a frozen extractor, even receiving fresh trajectories, applies a static distillation strategy calibrated to the initial agent's needs, and cannot adapt its criteria for what constitutes useful experience as the actor's capability evolves.

This three-way comparison establishes the paper's central empirical premise: simply having an online-updatable experience bank is insufficient; the extraction strategy itself must co-evolve with the actor. The critical comparison is Static Online Exp. vs. the full Complementary RL (which appears in later figures): the former has fresh data but a frozen extraction strategy; the latter has both fresh data and an adaptive extraction strategy. The large performance gap between them (shown in Figures 6 and 8) isolates the value of extractor co-evolution specifically, beyond data freshness.

Actor Advantage Estimation Ablations (Figures 3b–3d)

The paper conducts a set of ablations on MiniHack Room to validate the split-subgroup GRPO design with condition-wise advantage estimation, using Complementary RL as the base algorithm. All experiments use the default model configurations and hyperparameters.

Experience-guided vs. experience-free training (Figure 3b). When all K = 8 rollouts in each GRPO group are conditioned on retrieved experience (the "Experience-Guided" setting), the actor learns to use hints effectively but develops an over-reliance on external guidance. The paper reports:

"when all interactions are conditioned on retrieved experience, the actor converges prematurely and lags behind the experience-guided setting, suggesting that the actor fails to internalize experience into its own capabilities and instead develops an over-reliance on external guidance."

The training curve in Figure 3b shows that the pure experience-guided variant initially tracks the full Complementary RL (which uses the split-subgroup design with both guided and unguided rollouts), but plateaus earlier and at a lower success rate. This confirms that training exclusively on guided rollouts produces an agent that is proficient at following hints but fails to develop standalone capability. The baseline without any experience ("Experience-Free") serves as the lower reference point.

Cross-group vs. within-group advantage (Figures 3c and 3d). The paper tests two methods for computing advantages when rollouts are split into experience-guided and experience-free subgroups. The "Cross-group" variant (Figure 3c) pools all K trajectories together, computes a single group mean and standard deviation, and normalizes advantages across both subgroups relative to this pooled baseline. This causes training to collapse:

"the reward scales and variances differ between subgroups, causing advantage estimates to become biased and training to collapse."

The mechanism is a statistical confounding: experience-guided trajectories have systematically different (typically higher) success rates than experience-free trajectories, so a pooled baseline makes guided trajectories look artificially good (since the baseline includes lower-performing unguided trajectories) and unguided trajectories look artificially bad. The advantage estimates are contaminated by condition membership rather than reflecting genuine within-condition performance differences. Figure 3c shows that this pooled-advantage variant fails to improve over the baseline and exhibits unstable training.

The "Subgroup" variant (Figure 3d) computes advantages independently within each subgroup using subgroup-specific means r̄c and standard deviations σc—the design formalized in Equation 5. This preserves signal integrity by ensuring that a guided trajectory is evaluated relative to other guided trajectories, and an unguided trajectory relative to other unguided trajectories. Figure 3d shows that this variant yields consistent improvement in both subgroups (the "Experience-Guided" and "Experience-Free" curves), with the full Complementary RL achieving the highest success rates. This ablation is the paper's core evidence for the condition-wise advantage estimation design being necessary for stable co-evolutionary training, not merely a heuristic choice.

Single-Task Training (Figures 6 and 7)

The paper evaluates Complementary RL separately on each of four environments—MiniHack Room, WebShop, ALFWorld, and SWE-Bench—comparing against a baseline that does not leverage experience. These experiments use the default model configurations (Qwen2.5-7B-Instruct as πθ, Qwen3-4B-Thinking-2507 as πϕ) except SWE-Bench, which uses Qwen3-4B-Instruct-2507 as πθ. Training configurations vary by environment: MiniHack Room uses a rollout batch size of 128, group size K = 8, micro-batch size 64, maximum 30 interaction turns, and 145 training steps (following the pilot study setup). WebShop uses a rollout batch size of 64, group size K = 8, micro-batch size 16, maximum 256 training steps, and maximum sequence length 16,384 tokens. ALFWorld uses a rollout batch size of 128, group size K = 8, micro-batch size 32, 128 training steps, maximum 40 interaction turns, and maximum 2,048 output tokens per step. SWE-Bench uses the curated set of 124 tasks with default training loop parameters. Hyperparameters (learning rate 1e-6, AdamW optimizer, no KL regularization) are held constant across all environments.

Success rate results (Figure 6). The headline result is that Complementary RL consistently outperforms the baseline across all four tasks. The paper reports specific performance margins:

MiniHack Room: Complementary RL achieves roughly a 1.3× performance margin over the baseline, with notably better training stability. The baseline curve shows a gradual climb over 145 steps, while Complementary RL rises faster in early training and sustains a clear gap throughout. Quantitative details on final success rates are not provided in the main text (they appear in the multi-task Table 1), but Figure 6 (left panel) shows Complementary RL reaching approximately 0.9–1.0 success rate by step 150 versus the baseline at approximately 0.68 (matching the Table 1 baseline value).
WebShop: Complementary RL demonstrates a similar pattern of faster initial improvement and sustained advantage over the 256-step training horizon. The paper does not provide a specific percentage improvement figure for this single-task setting, but the multi-task results in Table 1 report baseline WebShop performance at 0.81 vs. Complementary RL at 0.87 (with experience at test time), suggesting a +6% absolute gain.
ALFWorld: The validation score metric (which ranges from −1 to 1 due to the ALFWorld reward scheme of +1 for success, −1 for failure) shows Complementary RL achieving substantially better training stability. The baseline exhibits oscillatory behavior—the curve fluctuates widely over training—while Complementary RL's curve rises more smoothly and reaches higher final validation scores. The paper reports that Complementary RL achieves "a 1.3× performance margin" on ALFWorld, presumably measured in terms of the validation score at the end of training.
SWE-Bench: On the most challenging benchmark, Complementary RL demonstrates faster improvement and achieves a +3.0% gain over the baseline. This is the most precisely reported single-task improvement figure in the main text. The learning curves in Figure 6 (rightmost panel) show Complementary RL separating from the baseline at approximately step 50 and maintaining an advantage through the remainder of training.

Action efficiency results (Figure 7). Beyond success rates, the paper reports that Complementary RL completes tasks with significantly fewer actions—a measure of decision-making efficiency that captures whether the agent learns to take more direct, effective paths to task completion rather than meandering or getting stuck in loops:

MiniHack Room: Complementary RL requires approximately 1.5× fewer actions than the baseline. The Figure 7 left panel shows the baseline averaging around 20–25 actions per task throughout training, while Complementary RL drops from approximately 20 actions early in training to roughly 12–15 actions by step 100—indicating that the agent is not just succeeding more often, but succeeding more efficiently when it does succeed.
ALFWorld: Complementary RL requires approximately 2× fewer actions than the baseline. The baseline curve hovers around 15–17 actions per task, while Complementary RL decreases to roughly 7–8 actions by mid-training.
WebShop: The action count difference is less dramatic but still present—both curves start around 20–25 actions and decrease, with Complementary RL maintaining a consistent advantage.
SWE-Bench: An interesting and non-obvious pattern emerges: Complementary RL exhibits an increasing number of actions on SWE-Bench over training, while the baseline remains relatively flat. The paper interprets this not as inefficiency but as evidence of more thorough task completion:

"we find that this is because the agent takes more actions to fully complete tasks, thereby achieving a higher success rate, rather than submitting prematurely before a task is finished."

The baseline agent tends to submit patches early when it gets stuck or confused, producing quick but often incorrect solutions, while Complementary RL's agent systematically works through the debugging and editing process, taking more steps but producing correct final patches. This tradeoff—more actions but higher success—is characteristic of agents that have learned to persist rather than give up, which the paper attributes to distilled experience teaching effective debugging and exploration strategies.

A critical limitation of the single-task results is that specific numerical success rates are not provided in the main text for individual environments—only the general "1.3× performance margin" for MiniHack Room and ALFWorld, the "+3.0% gain" for SWE-Bench, and the action-efficiency ratios are reported. The per-environment final success rates can be partially inferred from the multi-task Table 1, but Table 1 reports results from a different experiment (joint training on three tasks simultaneously) and therefore does not directly correspond to the single-task curves in Figure 6. The reader must visually estimate performance from the learning curves, which is imprecise.

Multi-Task Training (Figure 8 and Table 1)

The multi-task experiments jointly train on MiniHack Room, ALFWorld, and WebShop simultaneously to investigate whether Complementary RL benefits from cross-task experience distillation. The total rollout batch size is 384 (128 per task × 3 tasks), group size K = 8, micro-batch size 96, and training runs for 128 steps. Each environment contributes an equal number of parallel rollouts, and the single experience bank M is shared across all three tasks, meaning experience distilled from WebShop episodes can potentially be retrieved for MiniHack Room queries and vice versa. The paper compares four conditions:

Baseline: actor training without any experience (standard GRPO on outcome rewards only).
Static Online Exp.: a frozen πϕ dynamically maintains and constructs M during training but is never optimized, with the actor πθ trained normally. This isolates the effect of extractor co-evolution while keeping the bank online-updated.
Exp. Only: πϕ is trained via CISPO to maintain and refine M, but the actor πθ is held fixed at its initial parameters. This isolates the effect of actor co-evolution—it tests whether a better experience bank alone (without an improving actor) can drive performance gains.
Complementary RL: both πθ and πϕ are jointly optimized in the full co-evolutionary loop, with the split-subgroup GRPO design and asynchronous infrastructure.

Table 1 reports final evaluation performance across all three tasks, with methods that leverage experience evaluated under two settings: with retrieved experience at test time (w/ exp.) and without retrieved experience (w/o exp.), the latter measuring whether the agent has internalized guidance into its parameters.

Method	MiniHack Room	WebShop	ALFWorld	Avg.
Baseline	0.68	0.81	0.72	0.75
Static Online Exp. (w/ exp.)	0.41	0.67	0.69	0.59
Static Online Exp. (w/o exp.)	0.39	0.59	0.64	0.54
Exp. Only	0.49	0.37	0.13	0.33
Complementary RL (w/ exp.)	0.78	0.87	0.82	0.82
Complementary RL (w/o exp.)	0.75	0.84	0.74	0.78

Several findings emerge from this table and the corresponding training curves (Figure 8):

Finding 1: Co-evolution dramatically outperforms static experience. Complementary RL (w/ exp.) achieves an average performance of 0.82 across the three tasks, compared to Static Online Exp. (w/ exp.) at 0.59—a 23 percentage point improvement (+7% in the paper's relative framing). This is the paper's strongest demonstration that co-evolving the extractor matters: both methods have online-updatable banks and retrieve experience at test time, but Static Online Exp. uses a frozen extractor whose extraction strategy cannot adapt to the actor's evolving needs. The gap is largest on MiniHack Room (0.78 vs. 0.41, a 37-point difference), suggesting that maze navigation—where the optimal strategy shifts qualitatively from systematic exploration to targeted pathfinding—is particularly sensitive to extractor staleness.

Finding 2: Static online experience is actively harmful. Static Online Exp. underperforms even the no-experience baseline (0.59 vs. 0.75 average), a gap of >10 percentage points. This is a striking negative result: an online-updatable bank with a frozen extractor is worse than having no experience bank at all. The paper attributes this to distributional misalignment producing noisy and inconsistent retrieval:

"the fixed extractor cannot adapt its experience maintenance strategy to the evolving actor, leading to noisy and inconsistent retrieval, particularly in the multi-task setting where cross-task experience contamination is observed."

Cross-task contamination means that experience distilled from WebShop trajectories (about product search strategies) may be retrieved for MiniHack Room queries (about maze navigation) by semantic similarity alone, providing misleading or irrelevant guidance that actively harms performance. A co-evolving extractor learns to avoid this by being penalized (via negative rewards) when it produces experience that gets retrieved inappropriately and leads to failure.

Finding 3: Experience internalization is real but imperfect. Complementary RL (w/o exp.)—evaluating the actor without any retrieved experience at test time—achieves 0.78 average performance, a +2% improvement over the baseline at 0.75. This confirms that co-evolutionary training internalizes useful guidance into the actor's parameters: even when the experience crutch is removed, the actor retains the strategies it learned during guided practice. However, the gap between w/ exp. (0.82) and w/o exp. (0.78) is +4 percentage points, meaning that retrieving experience at test time still provides additional benefit beyond what has been internalized—the actor has not fully absorbed all available guidance. This is an expected and realistic result: some experience is inherently situational (e.g., "in this specific room layout, the key is behind the third door") and cannot be internalized as general policy without overfitting.

Per-task, the internalization pattern varies: MiniHack Room shows a 3-point gap (0.78 → 0.75), WebShop shows a 3-point gap (0.87 → 0.84), and ALFWorld shows an 8-point gap (0.82 → 0.74). ALFWorld's larger gap suggests that household task guidance is more context-dependent and harder to fully internalize into general strategies.

Finding 4: Exp. Only confirms that experience quality alone is insufficient. Training only the extractor while holding the actor fixed yields 0.33 average performance—the worst result across all conditions. The paper notes:

"optimizing only the experience extractor yields marginal actor improvement, suggesting that experience quality alone is insufficient when the actor's base capability is limited."

This finding is consistent with the "can't create capability from nothing" principle: a better experience bank provides useful hints, but if the actor lacks the basic competence to execute those hints, the guidance cannot be converted into task success. The extremely low ALFWorld score (0.13) under Exp. Only suggests that household task completion requires both good strategies (from experience) and the ability to ground those strategies in specific environment interactions (from an improved actor policy). The extractor can say "check countertops before searching cabinets," but a fixed actor that doesn't understand how to navigate to countertops cannot benefit from this advice.

Training curves (Figure 8). The learning curves tell a richer story than the final evaluation numbers alone. The baseline curve rises steadily across all three tasks and reaches moderate performance. Static Online Exp. is dominated by the baseline across nearly all tasks and all training steps—it never achieves a clear advantage at any point during training. Exp. Only shows marginal or no improvement over time (since the actor is frozen, its performance can only change due to better experience retrieval, which provides minimal benefit). Complementary RL's curve tracks above the baseline from early in training and maintains or widens the gap through the full 128 steps, with particularly strong separation on MiniHack Room and ALFWorld.

The average reward score across all three tasks ("All Tasks (Avg)" panel in Figure 8) shows Complementary RL reaching approximately 0.7–0.8 by step 125, compared to the baseline at approximately 0.55–0.6, Static Online Exp. at approximately 0.4, and Exp. Only at approximately 0.2–0.3. This aggregate metric confirms that the co-evolutionary benefits are not task-specific but generalize across the mixture.

Analysis: Extractor Capacity, Self-Distillation, Rollout Latency, and Task Scaling (Figure 9)

The paper conducts additional analyses to characterize Complementary RL's behavior under different configurations and scaling regimes.

Effect of experience extractor capacity (Figure 9a). To test whether a stronger extractor amplifies the benefits of co-evolution, the paper compares the default Qwen3-4B-Thinking-2507 extractor against Qwen3-30B-A3B-Instruct-2507 in the multi-task setting. Figure 9a shows the average reward score across all three tasks: the 30B-A3B extractor consistently outperforms the 4B extractor, with the gap widening over training. The paper reports:

"a larger experience extractor yields consistent improvement across tasks (+5% on average)"

The per-task breakdown (Figure 11 in Appendix A.2) shows that this benefit is consistent across all three environments, with MiniHack Room and ALFWorld showing larger gains than WebShop. This result is intuitive—a larger model can extract more generalizable and informative patterns from trajectories—but it also demonstrates that the co-evolutionary framework is not saturating at 4B extractor scale: there are still gains to be had from investing more extractor capacity. The paper does not report whether further scaling (beyond 30B) would continue to yield improvements, nor does it compare the cost of a larger extractor against the benefit it provides (larger models increase distillation latency and training cost).

Complementary RL with self-distillation (Figure 9b). The paper explores integrating self-distillation (Hübotter et al., 2026) into the Complementary RL framework. For each trajectory in the experience-guided subgroup, the trajectory's score is compared against the mean score of the experience-free subgroup. Trajectories that exceed this threshold are collected into a self-distillation batch. For each sample, all experience-related context is stripped (retrieved experience at the first turn and all search-and-ask interactions), and the actor πθ is supervised via next-token prediction loss jointly with the RL objective. The motivation is to convert externally scaffolded reasoning into intrinsic capability by having the actor practice reproducing successful guided behaviors without the guidance present.

Figure 9b shows the results on MiniHack Room. The combination (Complementary RL + Distill.) initially improves upon both the baseline and standard Complementary RL, achieving higher success rates in early-to-mid training (approximately steps 25–75). However, it collapses in later training (after step 100), falling below standard Complementary RL and approaching the baseline. The paper hypothesizes:

"this may stem from suboptimal hyperparameter choices, or alternatively, applying self-distillation at periodic intervals rather than every step may alleviate this issue."

This negative result is informative: while self-distillation can accelerate early learning by forcing internalization, the joint optimization of RL gradients and supervised distillation gradients may create conflicting update directions. The RL objective encourages exploration and adaptation, while the distillation objective encourages replication of specific (past) trajectories. When the actor's policy shifts away from the distribution of distillation samples, the two losses pull in opposite directions, destabilizing training. The paper does not pursue this direction further due to resource constraints, leaving it as an open problem.

Rollout latency (Figure 9c). To verify the claim that the asynchronous infrastructure introduces no appreciable latency, the paper measures average rollout collection time across four rollout batch sizes (128, 256, 512, and 1024 parallel environments) and compares against a baseline without experience integration. The results show ratios of 0.98×, 0.99×, 1.00×, and 0.99× respectively—meaning Complementary RL's rollout collection is within 1–2% of the baseline at all scales. This is a strong result: the co-evolutionary training pipeline achieves its accuracy gains without sacrificing training throughput. The companion Figure 12 (Appendix A.3) reports the average search time per training step across the same batch sizes: search times remain under approximately 1.0 second even at 1024 parallel environments, with the largest observed value around 0.9 seconds. This confirms that the query batching (batch size B = 16, maximum waiting time t_max = 0.001 seconds), embedding caching, and parallel search worker design (W = 4 workers) successfully absorb the retrieval workload.

Task scaling (Figure 9d). The paper tests whether Complementary RL's benefits persist as the number of tasks scales up, constructing a 3-task mixture (MiniHack Room, WebShop, ALFWorld) and a 6-task mixture that additionally includes MiniHack Maze, MiniHack KeyRoom, and Sokoban—all more challenging environments than the base MiniHack Room. Figure 9d reports:

"Complementary RL consistently outperforms the baseline in both settings (+6.6% and +8.1% on the 3-task and 6-task mixtures, respectively)"

The training curves (Figure 13 in Appendix A.4) show that Complementary RL maintains a clear advantage over the baseline throughout training on both mixtures. Notably, the relative improvement is larger on the 6-task mixture (+8.1%) than on the 3-task mixture (+6.6%), suggesting that co-evolutionary experience provides increasing returns as task diversity grows—cross-task experience becomes more valuable when more diverse strategies can be shared across environments. The absolute reward scores are lower on the 6-task mixture (both methods score lower, since the additional tasks are harder), but Complementary RL's advantage widens. A Sokoban task that requires multi-step box-pushing may benefit from navigation strategies distilled from MiniHack Room and systematic search patterns from MiniHack KeyRoom, creating synergistic cross-task transfer that a static or single-task experience bank cannot provide.

Ablation Studies and Robustness Checks

Periodic Merge (Figure 5a): Removing the periodic merge operation (which runs every 5 actor update steps to deduplicate and consolidate M via πϕ) causes a noticeable performance drop in later training. On MiniHack Room, both the baseline and the no-merge ablation track closely in early training (steps 0–75), but the ablation begins to fall behind after step 75 and remains below the full Complementary RL through the remainder of training. The paper attributes this to redundancy degrading retrieval quality: as episodes accumulate, near-duplicate experience entries crowd the bank, causing semantic search to return redundant results that fail to provide diverse guidance. The periodic merge, by consolidating semantically similar entries into compact, non-redundant representations, maintains a high-signal experience bank as training progresses. The specific margin of improvement from merging is not quantified in the main text—it must be visually estimated from Figure 5a, which shows the merge variant reaching approximately 0.9 success rate vs. the no-merge variant at approximately 0.8 by step 150.
Search and Ask (Figure 5b): Ablating the search-and-ask tool—which allows the actor to actively query M at any decision step with a context-aware query q′ rather than only at episode start with the task description— significantly reduces performance, particularly in mid-to-late training (after step 75). On MiniHack Room, the ablation variant tracks Complementary RL in early training but then plateaus at a lower success rate (approximately 0.7 vs. 0.9 by step 150). The paper attributes this to two factors: (1) without search-and-ask, the same experience entry is retrieved repeatedly (since parallel environments share identical task descriptions), reducing the diversity of extractor training signal; and (2) the actor loses access to situationally targeted guidance at critical decision points—it gets only "how to solve this task in general" rather than "how to solve this specific dead-end I'm currently stuck in." The margin between the two curves widens over training, consistent with the interpretation that search-and-ask becomes more valuable as the actor's capability improves and it encounters more nuanced situations where generic guidance is insufficient.
Experience extractor capacity (Figures 9a and 11): Replacing the default 4B extractor with a 30B-A3B extractor yields +5% average improvement across tasks in the multi-task setting, with consistent gains on all three environments (Appendix A.2, Figure 11). This ablation tests whether the co-evolutionary framework is bottlenecked by extractor capability: the positive result suggests that it is, and that further scaling the extractor (or using stronger base models) could yield additional gains. However, the paper does not report the computational cost of the larger extractor relative to the benefit—the 30B-A3B model has roughly 7.5× more active parameters than the 4B model, meaning each distillation and merge operation consumes significantly more compute.
Self-distillation integration (Figure 9b): Adding self-distillation (supervised fine-tuning on successful experience-guided trajectories with experience context stripped) to Complementary RL provides initial gains but collapses in later training on MiniHack Room. The paper identifies this as a negative result and hypothesizes suboptimal hyperparameters or the need for periodic rather than step-wise distillation. This ablation demonstrates that naive combination of RL objectives with distillation objectives can destabilize training, and that careful scheduling (not investigated in the paper) may be required for successful integration.
Actor-Critic mechanism (Figures 14a and 14b in Appendix B.1): The optional Actor-Critic extension—where πθ reflects on retrieved experience and produces accept/refine/reject decisions that provide an enriched reward signal for πϕ—yields improved success rates, particularly in early training on MiniHack Room (Figure 14a). However, it introduces a 30× latency overhead: the average search time increases from 0.04 seconds to 1.21 seconds (Figure 14b) because the actor must produce a critic decision before each environment interaction, blocking rollout collection. The paper therefore does not adopt Actor-Critic as a default component, recommending it only when final performance is prioritized over training throughput.
Training stabilization techniques (Appendix B): The paper reports that early attempts at extractor training were highly unstable due to two compounding challenges:
- Severe off-policyness: experience entries may be retrieved long after generation, introducing large policy lags between the generating extractor and the current πϕ. The CISPO clipping mechanism (ε_low = ε_high = 0.1) partially addresses this, but without additional stabilization the extractor training was unreliable.
- Data redundancy: in low-diversity task settings, a single experience entry m may be retrieved across many training buffer steps, causing πϕ to be updated repeatedly on the same sample and overfit.
The paper's two countermeasures—Retrieval Diversification (oversampling with diversity-aware re-ranking that penalizes frequently retrieved experiences) and Training-Count-Aware Advantage Reweighting (cooldown window δ that suppresses recent duplicates plus count-based decay (1 + c_train)^(-α))—are reported to yield stable training, but no ablation is provided showing training curves without these techniques. The paper states that "we were unable to guarantee stable training despite extensive tuning efforts" for earlier designs (Appendix B.2: sharing parameters between πθ and πϕ, using relative reward for experience), and that the current design "yields stable training," but quantitative evidence (e.g., training curves with and without each stabilization technique) is absent. This is a methodological gap: the reader cannot assess how sensitive performance is to these hyperparameters or whether the techniques are essential or merely helpful.
Single vs. separate parameter suites (Appendix B.2): Early experiments attempted to serve a single set of parameters shared between the training and inference engines for both πθ and πϕ, optimizing the shared model under two distinct objectives (Equations 5 and 3) simultaneously. The paper reports:

"Since the two optimization objectives impose possible conflicting gradient directions, we were unable to guarantee stable training despite extensive tuning efforts."

Switching to separate parameter suites for the actor and extractor resolved this and "yields stable training." No quantitative comparison of shared vs. separate parameters is provided—this finding is reported as a lesson learned rather than an ablation with empirical results. The implication is important for practitioners: using the same model checkpoint for both roles appears to be fundamentally unstable, likely because the GRPO objective (which optimizes for task success given an experience context) and the CISPO objective (which optimizes for experience utility) pull the shared parameters in incompatible directions.
Direct vs. relative reward for experience (Appendix B.2): The paper compared two reward strategies for πϕ: direct reward (assigning r(m) = +1 for success, −1 for failure based on episode outcome) and relative reward (first computing the average reward of the experience-free subgroup as a baseline, then assigning rewards proportional to improvement over this baseline). The paper reports:

"Empirical comparison revealed that this relative reward strategy consistently underperforms direct reward assignment."

No quantitative results or learning curves are provided for this comparison. The finding is notable because relative reward would intuitively provide a cleaner signal—it isolates experience contribution by subtracting out the actor's baseline capability—but apparently fails in practice. The paper does not speculate on why, but one plausible explanation is that the experience-free subgroup's performance is a noisy baseline (especially in early training when success is rare), and the relative reward introduces additional variance that destabilizes extractor optimization.
Auxiliary perplexity reduction reward (Appendix B.2): The paper explored augmenting πϕ's reward with a perplexity reduction bonus, based on the intuition that helpful experience should increase the actor's confidence (reduce its entropy) when processing a task. Five normalization strategies were tested (Relative, Tanh, Sigmoid, Asymmetric Clipping, Log-Space), but:

"None of these strategies yielded a consistent improvement in practice."

This negative result is reported succinctly without quantitative details, but it informs future work that entropy-based auxiliary rewards for experience utility are not trivially effective.
Action efficiency under multi-task training (Figure 10 in Appendix A.1): The paper reports average number of actions per task during multi-task training for Baseline, Static Online Exp., and Complementary RL. Across all three tasks (MiniHack Room, WebShop, ALFWorld), Complementary RL requires fewer actions than both the baseline and Static Online Exp., with the gap most pronounced on MiniHack Room (Complementary RL at ~12 actions vs. Baseline at ~18 by step 125) and ALFWorld (~8 vs. ~15). This confirms that the co-evolutionary experience not only improves success rates but also teaches the actor more efficient decision strategies—actions are more purposeful and less exploratory, reducing wasted steps.

Critical Assessment

The experiments in this paper collectively provide substantial evidence for the benefits of co-evolutionary experience extraction over static alternatives, but they also leave several important dimensions of the central claims under-explored. I examine each major claim and the evidence supporting it.

Claim 1: "Complementary RL outperforms outcome-based agentic RL baselines that do not learn from experience, achieving 10% performance improvement in single-task scenarios." This claim is only partially supported with quantitative precision. The single-task results in Figure 6 show clear visual separation between Complementary RL and the baseline across all four environments, and the learning curves consistently favor the co-evolutionary approach. However, the paper does not report specific numerical success rates for single-task experiments—the "10% performance improvement" from the executive summary does not correspond to any specific number in the single-task results section. The text mentions a "+3.0% gain" for SWE-Bench and "1.3× performance margin" for MiniHack Room/ALFWorld, but the 10% figure appears to aggregate across tasks in a way that is not made precise. The multi-task results (Table 1) show an average improvement of +7% with experience at test time and +2% without, but these numbers do not reach 10%. The reader must infer that the 10% claim refers to single-task settings, but no table of single-task final success rates exists to verify it. This is a significant reporting gap that undercuts the headline claim.

Claim 2: "In multi-task settings, Complementary RL exhibits robust scalability, outperforming baselines using static or non-adaptive experience banks by substantial margins—yielding a 7% average improvement with experience at test time." This claim is directly and quantitatively supported by Table 1 (0.82 vs. 0.75 average, a 7% absolute improvement). However, the comparison is against only one static baseline (Static Online Exp., at 0.59 average). The paper does not include a stronger static baseline such as offline experience bank with periodic manual curation or fixed extractor with heuristic update rules—both of which would be more competitive than the clearly dysfunctional Static Online Exp. (which underperforms the no-experience baseline). The 7% improvement relative to the no-experience baseline is legitimate, but the claim of "substantial margins" against static baselines is only demonstrated against a baseline that is actively harmful, which exaggerates the apparent benefit of co-evolution. A fairer comparison would include an offline bank built by a strong external model and kept updated through simple heuristics (e.g., replacing entries that lead to repeated failures). Without this, the 7% number should be interpreted as the benefit of co-evolution over no experience plus a broken static mechanism, not over the best possible static experience approach.

Claim 3: "Co-evolution enables the agent to internalize useful guidance into its parameters—a 2% average improvement even without retrieved experience at test time." This claim is well-supported by Table 1: Complementary RL (w/o exp.) achieves 0.78 vs. Baseline at 0.75, a +2% improvement. The per-task breakdown reveals that this internalization is task-dependent—MiniHack Room shows a 7-point gap between w/ exp. (0.78) and w/o exp. (0.75), while ALFWorld shows an 8-point gap (0.82 → 0.74)—suggesting that some tasks are more amenable to internalization than others. The paper does not investigate what properties make a task more or less internalizable, which is a missed opportunity for practical guidance. Additionally, the 2% average gain is modest relative to the 7% gain with experience, indicating that while internalization occurs, the majority of Complementary RL's benefit comes from having experience available at test time. The claim that the agent has "learned" from experience is therefore only partially true: it has learned to use experience effectively, but has only weakly internalized the content of that experience into standalone capability.

Claim 4: "The asynchronous infrastructure introduces no appreciable latency to rollout collection." This claim is strongly supported by Figure 9c, which shows rollout time ratios of 0.98–1.00× across batch sizes from 128 to 1024. The companion measurement in Figure 12 shows search times under 1 second at all scales. This is a substantive engineering contribution, but the paper's claims about infrastructure are somewhat limited in scope: only rollout collection latency is measured, not end-to-end training throughput (which includes extractor optimization time, merge operations, and other background track costs). The primary training loop is indeed latency-free, but the background track's throughput determines how quickly the experience bank evolves. If distillation and merging are slow relative to actor updates, the extractor may lag behind the actor, partially defeating the co-evolutionary purpose. No measurement of background track throughput or the ratio of actor updates to extractor updates per wall-clock hour is provided.

Structural weaknesses in experimental design:

No multiple-random-seed results. The paper presents learning curves without error bars or confidence intervals. All experiments appear to use single random seeds (the pilot study explicitly uses one seed for 145 steps; the multi-task experiments run for 128 steps without seed variation mentioned). Given the known sensitivity of RL training to random initialization and environment stochasticity, the lack of statistical quantification means that specific performance gaps (e.g., the +7% average in Table 1, the +3.0% on SWE-Bench) cannot be assessed for reliability. The SWE-Bench improvement of 3.0% on a curated 124-task set is particularly vulnerable to this concern—a single random seed could produce a 3% difference by chance on a test set of this size.
No comparison to prior published methods. The paper's baselines are internally constructed (no-experience Baseline, Static Online Exp., Exp. Only) rather than reimplementations of published methods. It does not compare against AgentEvolver (Zhai et al., 2025), MemRL (Zhang et al., 2026a), ReasoningBank (Ouyang et al., 2025), ExGRPO (Zhan et al., 2025), or any other named method from the related work section. This makes it impossible to assess whether Complementary RL advances the state of the art or merely outperforms intentionally weak baselines. The Static Online Exp. baseline, which underperforms the no-experience Baseline by 16 points, is clearly a straw man—no practitioner would deploy a frozen extractor with an evolving actor in a system where they expected good results. A stronger baseline would be a published method that the paper's approach claims to improve upon.
Single model family. All experiments use Qwen2.5-7B-Instruct as the actor (with one exception for the SWE-Bench 4B variant) and Qwen3-4B-Thinking-2507 as the extractor. The paper provides no evidence that Complementary RL works with other model families (Llama, Gemma, Mistral, DeepSeek) or at other scales (1B actors, 70B actors). The extractor capacity ablation (Figure 9a) scales within the Qwen family, but this only shows that larger Qwen models work better as extractors—it doesn't test whether a 4B Llama extractor would also benefit from co-evolution, or whether the approach transfers at all. The claim that Complementary RL is a "paradigm for efficient experience-driven agent learning" is therefore premature without cross-family evidence.
SWE-Bench task curation. The SWE-Bench experiment uses a filtered set of 124 tasks, retaining only those where pass@16 for Qwen3-4B-Instruct-2507 is in the range (0, 80%). This curation removes trivially easy tasks (pass@16 = 100%) and prohibitively hard ones (pass@16 = 0%), but the filtering threshold is arbitrary and specific to the 4B model. It is unclear whether the +3.0% improvement on this curated set would generalize to the full SWE-Bench-Verified distribution or to models of different scales. The curation also means the baseline itself is not evaluated on the hardest tasks where experience might be most needed—removing tasks with pass@16 = 0% eliminates precisely the regime where the extractor's guidance could bridge the gap between no capability and some capability. The paper's finding that Complementary RL helps most on medium-difficulty tasks (consistent with Figure 3a showing diminishing returns on tasks the agent has already mastered or cannot solve at all) is not specifically broken out by SWE-Bench difficulty.
Missing ablation: extractor training frequency and buffer size. The extractor's training buffer size (|B_ϕ| = 64) and the merge interval (5 steps) are reported as fixed hyperparameters, but no ablation sweeps these values. Given that the extractor's optimization schedule is decoupled from the actor's, the ratio of actor updates to extractor updates could significantly impact the co-evolutionary dynamics—too frequent extractor updates might cause the experience distribution to shift before the actor can adapt, while too infrequent updates might cause staleness. The paper does not explore this tradeoff.
No ablation of CISPO vs. other objectives for the extractor. The choice of CISPO (Chen et al., 2025) over REINFORCE is justified in the text by the need for stable co-evolution, but no comparison of these two objectives is provided for the extractor training. REINFORCE, GRPO (adapted for extractor optimization), or even supervised fine-tuning on high-reward experiences could be viable alternatives with different stability-efficiency tradeoffs. Without this comparison, the claim that CISPO is necessary for stability is asserted rather than demonstrated.
The pilot study (Figure 3a) uses different experimental conditions than the main results. The pilot study trains for 145 steps on MiniHack Room with specific hyperparameters (K = 8, batch size 128), while the single-task MiniHack Room results in Figure 6 appear to use the same configuration, but this correspondence is not explicitly confirmed. The Offline Exp. condition in the pilot study uses Qwen3-30B-A3B-Instruct-2507 for distillation—a much larger model than the default 4B extractor. This means the pilot study's demonstration that offline experience decays is partially confounded with extractor capacity: the offline bank was built by a 30B model, while the full Complementary RL uses a 4B model. Would a 4B offline extractor produce less useful initial experience, making the decay pattern less visible? Would a 30B co-evolving extractor eliminate the decay entirely? The pilot study convincingly demonstrates the decay pattern, but the scale mismatch between the offline and online extractors complicates the interpretation.

Missing experiments that would have strengthened the paper:

Direct comparison with a published experience-driven RL method (e.g., AgentEvolver or MemRL) on at least one shared benchmark.
Ablation of the K/2 split ratio in the GRPO subgroups—what if the ratio is 3/4 vs. 1/4? What if the subgroups have different sizes?
Measurement of background track throughput to assess whether distillation keeps pace with actor updates at scale.
Multi-seed results with confidence intervals for all main claims.
Evaluation on the full (unfiltered) SWE-Bench to test whether Complementary RL helps on tasks where the baseline has zero success.
A static but strong baseline: an offline bank built by the 4B extractor on the final actor's trajectories, maintained with simple deduplication and relevance filtering, to test whether co-evolution during training is necessary or whether a one-time distillation at convergence would suffice.
Difficulty-stratified results (following the pattern established in the prior sections' analyzed paper on test-time compute scaling) to determine whether the benefits of co-evolution are concentrated on easy, medium, or hard tasks within each environment.

When do the claims hold conditionally?

The paper's central claim—that co-evolving the experience extractor improves learning efficiency—appears to hold robustly across the four tested environments and two task scales (3-task and 6-task mixtures). However, the claim is bounded by conditions that the paper partially identifies but does not systematically characterize:

Extractor capacity matters: the +5% gain from scaling the extractor to 30B (Figure 9a) implies that the benefits of co-evolution are amplified by extractor quality. Conversely, a very weak extractor might not benefit from co-evolution because it lacks the capacity to adapt its extraction strategy meaningfully. The paper does not test a lower bound (e.g., a 1B extractor) to see if co-evolution still helps.
Actor base capability matters: the Exp. Only condition in Table 1 shows that a better experience bank alone cannot compensate for a weak actor (0.33 average). This implies a threshold effect: there exists some minimum actor capability below which co-evolution provides no benefit because the actor cannot execute guidance effectively enough to generate informative reward signals for the extractor. The paper does not estimate this threshold.
Task diversity matters: the task scaling result (Figure 9d) shows larger relative gains on the 6-task mixture (+8.1%) than the 3-task mixture (+6.6%), suggesting that co-evolution's benefits scale with task diversity. The paper does not test whether the returns to diversity eventually saturate—would a 20-task mixture show proportionally larger gains?
Training horizon matters: the benefit of offline experience decays over training (Figure 3a), and the search-and-ask ablation shows growing importance over time (Figure 5b). These patterns suggest that co-evolution is most valuable in moderate-to-long training runs where staleness has time to accumulate. A very short fine-tuning run might achieve similar results with a static bank, making co-evolution's overhead unjustified.

In summary, the experiments convincingly demonstrate that Complementary RL outperforms no-experience and naive static-experience baselines across a range of agentic environments, establishing co-evolution as a viable and beneficial training paradigm. However, the paper's claims about the magnitude of improvement and the generality of the approach are weakened by the absence of strong published baselines, statistical quantification, cross-model-family evidence, and ablations that would clarify the necessary conditions for co-evolution to be beneficial. The framework is best understood as an enabling infrastructure and a proof of concept that co-evolutionary experience extraction is possible and beneficial, rather than as a fully characterized method with known bounds and failure modes.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Unaccounted for in Efficiency Claims

The assumption or constraint. The entire compute-optimal framework rests on the ability to estimate prompt difficulty before allocating the inference budget. The paper's method for doing so—generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted)—is extraordinarily expensive. The authors acknowledge this directly (Section 3.2):

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. At 2048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations). In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. The 4× figure should therefore be understood as an upper bound on achievable efficiency rather than a realized deployment gain. This is the single largest gap between the paper's theoretical framework and its practical deployability.

What evidence exists in the paper. The paper explicitly flags this as a limitation and suggests future work on training models to predict difficulty directly from question text, but no such model is developed or evaluated (Section 3.2). The predicted difficulty bins (using PRM scores) perform nearly as well as oracle bins (Figures 4 and 8), which is encouraging, but the 2048-sample cost remains regardless of whether ground-truth or PRM scores are used as the correctness signal. No experiment measures total cost including difficulty estimation.

Mitigation status. Not addressed. The paper frames difficulty estimation cost as a key direction for future work and notes that adaptive schemes (starting with a few samples, then allocating remaining budget based on initial difficulty signals) could amortize this cost into the problem-solving process. The authors suggest that "exploring alternative methods for cheap difficulty estimation" is a priority area, with the specific proposal of training a lightweight classifier to predict difficulty from the question text alone.

Limitation 2: Single Benchmark, Single Model Family Prevents Assessment of Generality

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state (Section 4):

"We believe this model is representative of the capabilities of many contemporary LLMs"

but this claim is unverified. The test set of 500 questions, split into five difficulty quintiles of ~100 each and further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin.

The consequence. Several aspects of the findings could be model-specific. The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution—a model with different calibration properties or error patterns might exhibit qualitatively different difficulty-dependent scaling curves. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning; it is unclear whether the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems) generalize to other reasoning domains (code generation, logical reasoning, scientific QA) or to tasks requiring factual knowledge rather than inference. The small per-bin sample size (~50 questions for strategy selection) means the selected strategies may not be robust across different random splits, yet the paper reports no confidence intervals.

What evidence exists in the paper. None. The paper provides no experiments on alternative benchmarks, no experiments with alternative base models, and no analysis of how the difficulty-dependent strategy selection varies across random splits of the small test set. The cross-family and cross-domain questions remain entirely open.

Mitigation status. Not addressed beyond the authors' stated belief in the model's representativeness. The paper does not frame this as a limitation requiring future work, which is a notable omission given that all quantitative claims (4× efficiency, specific accuracy numbers, the FLOPs-matched comparison percentages) are conditional on a single model-benchmark pair.

Limitation 3: The ~14× Larger Model Baseline Is Not Compute-Optimally Trained, Weakening the Pretraining-Vs-Inference Comparison

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The authors acknowledge that this departs from compute-optimal pretraining (Hoffmann et al., 2022), where both data and parameters would be scaled equally:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the ~14× larger model uses only greedy decoding—no majority voting, no best-of-N, no search of any kind.

The consequence. A Chinchilla-optimal model trained with 14× more total FLOPs would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it could be. The reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at R ≪ 1) may shrink or reverse against a properly compute-optimal larger model. Furthermore, giving the larger model even a modest test-time compute budget (say, best-of-8 or best-of-16) would create a much stronger baseline that is never tested. The paper's headline claim—that a smaller model with test-time compute can outperform a ~14× larger model—is therefore only demonstrated against a specific, potentially suboptimal pretraining baseline.

What evidence exists in the paper. The FLOPs-matched results in Figure 9 and the bar charts in Figure 1 clearly show the comparison is against a greedy-decoding larger model with no test-time augmentation. The paper is transparent about the parameter-only scaling assumption in Section 7. However, no experiment compares against a compute-optimally trained larger model or a larger model with any test-time compute budget.

Mitigation status. Partially acknowledged. The authors note the parameter-only scaling choice and frame compute-optimal pretraining baselines as future work, but the acknowledgement is brief and does not discuss how much this choice might affect the quantitative conclusions. The absence of any test-time compute for the larger model is not discussed as a limitation at all—the paper treats greedy decoding as the natural default for the larger model, which is a strong assumption given that the entire paper is about the value of test-time compute.

Limitation 4: Revisions and Search Are Studied Independently, Not Combined

The assumption or constraint. The paper studies two complementary mechanisms—PRM tree-search (modifying the verifier/selection process) and iterative revisions (modifying the proposal distribution)—but never combines them. Section 8 explicitly acknowledges this:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. The paper's two axes of test-time compute (proposal distribution modification and verifier-guided selection) have complementary strengths: revisions are most effective on easy problems where local refinement suffices, while search helps on medium problems where exploration of alternative strategies is needed. The current results therefore represent a lower bound on what a fully integrated system could achieve. A combined approach—using the revision model as the proposal distribution within beam search, or using the PRM to guide which revision branches to pursue—could yield gains beyond either method alone, potentially extending the benefits of test-time compute to a broader difficulty range. The paper does not establish whether the difficulty-dependent patterns (revisions helping easy problems, search helping medium problems) would persist or shift in a combined system.

What evidence exists in the paper. No experiments combine revisions with PRM search. The paper provides separate analyses for each mechanism (Sections 5 and 6), including their individual difficulty-dependent behavior, but the interaction effects are unexplored. The reader cannot determine whether the gains are additive, subadditive, or perhaps antagonistic (e.g., if revision outputs are harder for the PRM to score accurately, combining them with search could exacerbate over-optimization).

Mitigation status. Acknowledged as future work in Section 8. The authors identify this as a natural next step but provide no preliminary results or analysis of what combined approach might be most promising. The computational cost of running both mechanisms simultaneously (which would compound the already-significant inference overhead) is not discussed.

Limitation 5: No Accounting for Latency or Wall-Clock Time in the Compute-Optimal Framework

The assumption or constraint. The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores latency. Sequential revisions are inherently serial—each revision depends on the previous one—while parallel best-of-N can be executed simultaneously with sufficient hardware.

The consequence. A strategy that allocates 128 generations as 64 sequential × 2 parallel takes roughly 64× longer wall-clock time than one that runs 128 parallel samples simultaneously. For latency-sensitive applications (interactive assistants, real-time decision-making), the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be impractical regardless of their accuracy advantages. A user waiting for an answer from a math tutor would experience dramatically different response times under sequential revision strategies versus parallel sampling, even when both use the same generation budget. The "compute-optimal" framework optimizes only for accuracy per FLOP, not for accuracy per second of wall-clock time—a critical distinction for deployment.

What evidence exists in the paper. None. The paper does not report wall-clock time for any experiment, does not discuss the latency implications of sequential sampling, and does not consider latency as a constraint in the compute-optimal objective (Equation 1 in Section 3.1). The compute budget N is treated as a scalar representing total generations, with no distinction between serial and parallel execution cost in terms of time.

Mitigation status. Not addressed. The paper's framework is purely FLOPs-oriented, and the authors do not flag latency as a limitation or direction for future work. This is a significant practical concern that would affect any deployment of the method in interactive settings.

Limitation 6: Hard Problems Remain Fundamentally Unsolved—Test-Time Compute Creates No New Capability

The assumption or constraint. The framework implicitly assumes that the base model has some non-trivial probability of producing correct solutions—that the proposal distribution contains correct answers that search or revisions can find or refine. For problems where the base model's pass@1 is near zero, no test-time strategy provides any benefit.

The consequence. Across all methods—search, revisions, and their compute-optimal combinations—the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%. This establishes a hard boundary: test-time compute can amplify existing capability but cannot create it from nothing. For organizations facing genuinely novel or out-of-distribution reasoning tasks that exceed their base model's training distribution, this approach offers no path forward—pretraining remains the only viable option. The paper's empirical finding that test-time compute works best on easy-to-medium problems but fails completely on hard ones is consistent across all mechanisms, confirming this is a fundamental limitation rather than a methodological shortcoming of any particular approach.

What evidence exists in the paper. The difficulty-bin analyses consistently show near-zero accuracy on bin 5. The paper is candid about this (Section 7 takeaway box, Section 8), explicitly stating that hard problems benefit only from pretraining. The FLOPs-matched comparison (Figure 9) reinforces this by showing negative relative improvement on hard problems at most R values.

Mitigation status. Fully acknowledged but not addressed algorithmically. The paper frames this as a fundamental characteristic of test-time compute rather than a solvable limitation: test-time compute and pretraining compute are not 1-to-1 exchangeable, and some capabilities can only be acquired through pretraining. The authors do not propose methods to extend test-time compute benefits to harder problems, and the findings suggest that doing so would require either (a) improving the base model's pass@1 on bin-5 tasks (which is a pretraining problem) or (b) developing fundamentally different test-time mechanisms that can generate correct solutions from a proposal distribution with near-zero initial success probability—neither of which is within the paper's scope.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper initiates a paradigm shift in how the field conceptualizes the relationship between an agentic LLM and its memory. Before Complementary RL, the dominant mental model treated experience as a static resource: you build a bank (offline or online), you retrieve from it, and you hope the retrieval helps. The extractor—the component that converts raw trajectories into structured, transferable experience—was an engineering artifact, not a learning system. It could be made more sophisticated through better prompts, larger models, or fancier data structures, but it was fundamentally fixed in its extraction strategy. The actor improved; the extractor did not.

Complementary RL changes this by establishing that the experience extractor should be a reinforcement learner optimized for the downstream utility of its outputs. This is not an incremental refinement—it is a topological change to the optimization graph. Instead of a unidirectional flow (actor generates trajectories → extractor distills experience → actor uses experience), Complementary RL creates a bidirectional loop where the extractor's reward depends on whether its experience demonstrably helped the actor succeed, and the actor's policy is shaped by experience calibrated to its current capability. Each model's learning objective references the other's behavior, making them symbiotic learners rather than master and tool.

The paper provides concrete evidence for why this shift matters. Static Online Exp.—where the experience bank is updated online but the extractor is frozen—underperforms the no-experience baseline by over 10 percentage points in multi-task training (Table 1: 0.59 vs. 0.75). This is a striking negative result that reframes the entire prior literature: it suggests that merely having an online-updatable bank, as advocated by Zhang et al. (2026a) and Ouyang et al. (2025), is not just insufficient but can be actively harmful when the extractor's distillation strategy is mismatched to the actor's evolving capability. The paper's diagnostic—distributional misalignment: the extractor keeps mining for patterns relevant to early exploration long after the actor has moved on to exploitation—provides a causal explanation for this failure that generalizes across prior approaches.

The field-level consequence is a reorientation of research priorities. Before this work, improving experience-driven RL meant improving experience quality: better distillation prompts, more sophisticated retrieval heuristics, larger extractor models. These directions are not wrong—the paper itself shows that scaling the extractor from 4B to 30B yields +5% gains (Figure 9a)—but they are incomplete. The paper redirects attention to a question that prior work simply didn't ask: does the experience pipeline adapt as the agent learns? This is now a first-class design requirement, not an afterthought. Future systems that claim to learn from experience will need to demonstrate not just that their experience is high-quality, but that the extraction strategy co-evolves with the actor. Conferences and reviewers will increasingly ask: "Does your extractor adapt? What happens to your experience bank after 100 training steps when the actor's policy has shifted?"

The paper also reconciles contradictory intuitions in the literature about when experience helps. Some prior work found clear benefits from memory augmentation (Zhai et al., 2025; Zhang et al., 2026a); other work, including the paper's own Static Online Exp. baseline, found that experience provides marginal or negative value. The resolution is that the benefit of experience is not a property of the experience content alone, but of the match between the experience and the agent's current capability state. A frozen extractor produces experience that is well-matched to the agent at time t, but this match degrades as the agent improves. Co-evolution maintains the match. This is not a methodological quirk—it is a fundamental insight about the relational nature of experience utility that should inform how all future memory-augmented agent systems are designed and evaluated. The paper's finding that the benefit of offline experience decays over training (Figure 3a) is not a failure of offline experience per se, but a consequence of the extractor not adapting—and this pattern should be expected in any system where the memory component is static while the decision component learns.

The demonstration that the split-subgroup GRPO design with within-subgroup advantage estimation (Equation 5) can internalize guidance into the actor's parameters—producing a 2% average improvement even without retrieved experience at test time (Table 1)—has implications beyond experience-driven RL. This technique addresses a general problem in multi-condition RL training: when an agent is trained under heterogeneous context conditions (with and without tools, with and without demonstrations, with and without retrieved knowledge), naive pooling of advantage estimates across conditions introduces statistical bias because reward distributions differ by condition. The condition-wise advantage solution—compute advantages within each condition's reference group, then average the losses—is a design principle that applies to any setting where the agent's context qualitatively changes between rollouts. This is a concrete, portable algorithmic contribution that researchers can adopt even if they do not use the full Complementary RL framework.

The asynchronous infrastructure design—a centralized ExperienceManager with reader-writer lock protocol, query batching, embedding caching, and decoupled primary/background tracks—establishes that co-evolutionary training need not sacrifice throughput. The latency measurements (Figure 9c: 0.98–1.00× rollout time vs. baseline across batch sizes 128–1024; Figure 12: search times under 1 second) provide a practical template for scaling dual-model RL systems. This is an enabling contribution: it converts a theoretically appealing idea (bidirectional optimization) into an empirically demonstrable one (training runs that complete in comparable wall-clock time to single-model baselines). Future work on co-evolving systems—whether for memory, tool use, planning, or multi-agent coordination—can adopt this architecture as a starting point rather than rediscovering the concurrency challenges.

Finally, the paper establishes that cross-task experience transfer is a scaling phenomenon: the relative improvement over the baseline grows from +6.6% on 3 tasks to +8.1% on 6 tasks (Figure 9d). This suggests that co-evolutionary experience is not just a way to train individual tasks faster, but a mechanism for discovering universal agentic strategies that transfer across environments. Experience entries like "when stuck in a loop, stop and diagnose rather than repeating the same action" (Table 6) are distilled by the extractor and applied across MiniHack Room, WebShop, and ALFWorld. As the number of training tasks scales, the density of such cross-cutting patterns increases, and a co-evolving extractor can identify and refine them in ways that a static extractor cannot. This positions Complementary RL as a candidate for the "agent foundation model" training recipe where a single policy is trained on hundreds of diverse environments with a shared, co-evolving experience bank.

Follow-Up Research This Work Enables

Scaling the number of co-evolving tasks to test whether cross-task transfer saturates or accelerates. The paper demonstrates increasing returns from 3 to 6 tasks (Figure 9d: +6.6% → +8.1%), but the trend is established with only two data points and a narrow task diversity range (navigation, shopping, household tasks). A natural follow-up would train Complementary RL on 20–50+ environments spanning navigation, web interaction, code manipulation, dialogue, tool use, and game playing, measuring whether the relative improvement over the no-experience baseline continues to grow, plateaus, or eventually degrades. The hypothesis to test is that a shared experience bank, co-evolved with the actor, can discover a set of "universal agentic primitives" (loop-breaking, systematic search, failure diagnosis, efficient backtracking) that transfer across superficially dissimilar tasks, and that the value of this shared bank scales with task diversity. The key measurement would be the slope of relative improvement vs. number of tasks, and whether it remains positive beyond the 6-task regime. This would directly inform whether Complementary RL is suitable as a foundation-model training paradigm (where task mixtures of 100+ environments are common) or whether its benefits are bounded to moderate multi-task settings.

Characterizing the necessary conditions for co-evolution to be beneficial through systematic ablations of actor and extractor capability. The paper provides suggestive but incomplete evidence about when co-evolution helps. The Exp. Only condition (Table 1, 0.33 average vs. Baseline 0.75) shows that a frozen actor cannot benefit from a better experience bank, implying a minimum actor capability threshold below which co-evolution cannot create improvement from nothing. The extractor capacity ablation (Figure 9a) shows that larger extractors amplify benefits, but doesn't test a lower bound—would a 1B extractor still benefit from co-evolution, or is there a minimum capacity needed to learn a useful distillation strategy? A systematic follow-up would sweep actor model scales (e.g., Qwen2.5-0.5B, 1.5B, 3B, 7B, 14B, 32B) and extractor scales (0.5B, 1.5B, 4B, 14B, 30B) in a factorial design on MiniHack Room and WebShop, measuring the marginal benefit of co-evolution over the no-experience baseline for each combination. The expected result is a phase diagram: below some actor capability, co-evolution provides zero or negative benefit (the actor can't execute guidance, so extractor rewards are pure noise); above some extractor capability, returns diminish (the extractor is already extracting everything useful that a frozen strategy would find). The phase boundaries would provide practical guidance for when to invest in co-evolutionary training vs. training a larger base model first.

Comparing Complementary RL against published memory-augmented RL methods on a shared benchmark. The paper's primary baselines are internally constructed (Baseline, Static Online Exp., Exp. Only) and the Static Online Exp. variant underperforms the no-experience baseline so severely that it functions as a demonstration of failure rather than a competitive alternative. To establish Complementary RL as a state-of-the-art method, a follow-up study should implement and compare against AgentEvolver (Zhai et al., 2025), MemRL (Zhang et al., 2026a), ReasoningBank (Ouyang et al., 2025), and ExGRPO (Zhan et al., 2025) on at least MiniHack Room and WebShop—environments where these prior methods should be applicable. The key measurements would be final success rate, sample efficiency (success rate vs. training steps), and evaluation with vs. without retrieved experience at test time (to assess internalization, which most prior methods do not measure). A negative result—finding that a simpler published method matches Complementary RL's performance on certain tasks—would be equally valuable, as it would identify specific regimes where co-evolution is unnecessary and a static but well-designed bank suffices. This comparison is the most important missing experiment for establishing the paper's practical contribution relative to the literature.

Abalting the K/2 split ratio and the extractor training frequency to characterize the co-evolutionary dynamics. The paper fixes the GRPO subgroup split at K/2 guided and K/2 unguided rollouts, and the extractor training buffer size at |B_ϕ| = 64 samples (which may span multiple actor update steps), but provides no sensitivity analysis for these ratios. The split ratio controls the tension between guided practice (which accelerates learning but risks over-reliance) and autonomous practice (which builds internalized capability but may be slower). The extractor training frequency controls how quickly the experience distribution adapts to the actor—too frequent updates could cause the distribution to shift before the actor can exploit it; too infrequent updates risk staleness. A systematic sweep of split ratios (K/8 guided, K/4, K/2, 3K/4 guided) and buffer sizes (16, 32, 64, 128, 256) on MiniHack Room would reveal whether there is an optimal balance point and how sensitive performance is to these choices. The hypothesis is that the optimal split ratio depends on the task: tasks where the primary challenge is strategy selection (maze navigation) might benefit from more guided practice, while tasks where the challenge is precise execution (code editing) might require more autonomous practice to avoid over-reliance on potentially imprecise guidance. This ablation would convert the paper's fixed design choices into tunable hyperparameters with understood effects, making Complementary RL more practical for new tasks.

Testing whether co-evolution can recover from a deliberately corrupted experience bank. The paper shows that co-evolution improves experience quality over time, but doesn't test robustness to bad initial conditions. A stress-test would initialize the experience bank with deliberately harmful entries—e.g., advice that recommends walking into traps on MiniHack Room, searching for wrong product attributes on WebShop, or executing incorrect code edits on SWE-Bench—and measure whether co-evolution can (a) identify and discard the harmful entries through negative extractor rewards, and (b) replace them with useful entries through positive rewards on successful trajectories. This would test whether the CISPO objective's reward signal is strong enough to overcome a poor initialization, and whether the actor can survive the initial period of bad guidance long enough for the extractor to learn. The key measurements would be the "recovery time" (training steps until success rate reaches the no-experience baseline) and "recovery quality" (final success rate relative to a clean-initialization Complementary RL run). Positive results would demonstrate robustness to distribution shift in the experience bank—a practical concern if Complementary RL is deployed in settings where the initial bank is constructed from unreliable sources. Negative results would identify a failure mode where co-evolution can get trapped in a bad local optimum (harmful experience → actor failure → negative extractor reward that doesn't point toward improvement because the extractor hasn't seen any good experience to learn from).

Applying Complementary RL to domains with dense, structured reward signals to test whether co-evolution learns more sophisticated distillation strategies. The current experiments use binary outcome rewards (+1 for success, −1 for failure) for both the actor and the extractor. This is a harshly sparse signal: the extractor receives a single bit of feedback per experience entry, averaged over all trajectories it guided. In domains with denser feedback—code generation with per-unit-test rewards, dialogue with turn-level satisfaction scores, game-playing with intermediate score increments—the extractor could potentially learn to distill experience that targets specific sub-skills (e.g., "how to handle null pointer exceptions in Java" vs. "how to structure a multi-function refactor"). A follow-up would deploy Complementary RL on a dense-reward RL environment (e.g., SWE-Bench with per-test-case rewards, or a multi-turn negotiation task with intermediate agreement scores) and analyze whether the co-evolved extractor produces hierarchically structured experience—entries that specifically target sub-problems rather than whole-task strategies. The hypothesis is that dense rewards provide a richer training signal for the extractor, enabling it to learn a finer-grained distillation strategy that decomposes tasks into reusable sub-skills. The analysis would involve qualitative examination of the experience bank's evolution over training: does the extractor start with holistic task strategies and then specialize into sub-task patterns as the actor's per-sub-task performance becomes distinguishable through denser rewards?

Practical Applications and Downstream Use Cases

Training agentic LLMs for enterprise task automation with heterogeneous task mixtures. An organization deploying LLM agents to handle customer support tickets, data entry workflows, inventory management queries, and internal tool orchestration faces a training problem: these tasks differ in surface form but share underlying patterns (diagnosing failures, breaking tasks into steps, retrying after errors). Training a single policy on this mixture using Complementary RL means the experience extractor can distill cross-task strategies—e.g., "when a tool call fails with an authentication error, first check credentials before retrying"—that transfer from one domain to another. The paper's finding that multi-task training with Complementary RL improves action efficiency (1.5–2× fewer actions on MiniHack Room and ALFWorld, Figure 7) translates directly to cost savings: each customer support interaction that takes 8 agent actions instead of 15 reduces API inference costs by nearly half. Further, the ability to evaluate without retrieved experience (Table 1: 0.78 vs. 0.75 baseline, a +2% improvement) means the trained agent can be deployed in latency-sensitive settings where retrieval would add unacceptable overhead—the agent has internalized the most generalizable strategies from its experience.

Cost-efficient fine-tuning of small models for specialized agentic tasks through co-evolution. An organization with a domain-specific agentic task (e.g., legal document processing, medical triage, chip design verification) may have a strong 7B base model but limited task-specific training data. Running Complementary RL with a 4B extractor (the paper's default configuration) is computationally cheaper than scaling to a 30B actor and training from scratch. The paper's SWE-Bench result—a +3.0% improvement with Qwen3-4B-Instruct-2507 as the actor on a curated 124-task set—demonstrates that even small models can benefit from co-evolutionary experience on challenging, specialized tasks. The practical workflow would be: (1) deploy the 7B actor on the task with a 4B extractor, (2) let the co-evolutionary loop run for 100–150 training steps (the paper's typical horizon), (3) evaluate with and without experience retrieval at test time to verify internalization, (4) deploy the trained actor (without the extractor, if latency matters) for production inference. The compute cost is dominated by the rollout collection for actor training, and the paper's latency measurements (Figure 9c) show that the experience infrastructure adds negligible overhead to this process.

Bootstrapping agent training for novel environments where no prior experience bank exists. When deploying an LLM agent in a completely new environment—a custom internal tool, a newly released benchmark, a simulated robotics domain—there is no offline experience bank to provide initial guidance. The naive approach (training with outcome-based RL from scratch) is extremely sample-inefficient because the agent receives no guidance in early exploration. Complementary RL provides a self-bootstrapping alternative: start with an empty experience bank M, an untrained actor, and an untrained extractor, and let the co-evolutionary loop generate its own curriculum. In early training, the extractor will produce low-quality experience (since it has no good trajectories to learn from), but as the actor occasionally succeeds through random exploration, those successes provide positive extractor rewards, gradually teaching the extractor to distill useful patterns. The paper's pilot study (Figure 3a) shows that offline experience provides an early boost that decays, but Complementary RL's benefit is sustained—in a cold-start setting where offline experience doesn't exist, the extractor essentially builds its own offline bank from scratch through the co-evolutionary loop. The practical benefit is that an organization deploying agents in a new domain doesn't need to invest in manual experience curation or expensive offline distillation runs—they can launch Complementary RL directly and rely on co-evolution to build domain-specific experience automatically.

When to Prefer This Method

The paper does not articulate a structured tradeoff matrix against named alternative methods (e.g., "use Complementary RL when X, use AgentEvolver when Y"). It positions co-evolution against static experience banks and no-experience baselines internally. Therefore, a formal decision rule matrix is not appropriate. However, the paper's empirical results imply conditions under which Complementary RL is likely to be beneficial or unnecessary, which practitioners should weigh:

Prefer Complementary RL when training an agent on multiple tasks simultaneously where cross-task experience transfer could accelerate learning (supported by Figure 9d: +6.6% on 3 tasks, +8.1% on 6 tasks), and when the agent is expected to deploy both with and without experience retrieval at test time (supported by Table 1: +2% without experience, confirming internalization).
Prefer Complementary RL when training runs are long enough (100+ steps) for the co-evolutionary dynamics to matter—the paper shows that offline experience benefits decay over time (Figure 3a), while co-evolution's benefits are sustained or grow (Figure 8), making the overhead of training a separate extractor worthwhile primarily for longer training horizons.
Static experience banks may suffice when the agent's capability is already near saturation on the target tasks, or when training runs are short enough that staleness hasn't accumulated (the first ~50 steps in Figure 3a show similar performance for offline and co-evolutionary approaches), or when deployment always includes experience retrieval (so internalization is unnecessary), or when the cost of training a separate extractor model is prohibitive relative to the expected benefit.
Scale pretraining instead when the base model's success rate on target tasks is near zero—the paper's Exp. Only condition (Table 1, 0.33 average) and the consistent finding that hard problems in the pilot study show minimal improvement under any experience strategy (analogous to the bin-5 failure mode in the difficulty-stratified analyses from prior work) confirm that co-evolutionary experience cannot create capability where none exists.