Replay across Experiments: A Natural Extension of Off-Policy RL

ArXiv: 2311.15951

🎯 Pitch

Simply mixing data from prior training runs into a new run’s replay buffer—even at a naive 50/50 ratio—dramatically boosts performance on hard exploration and vision-based tasks without any per-task tuning. Surprisingly, this works just as well with low-return data as with expert data, and the gains plateau after only two iterations, making it a cheap, drop‑in accelerator for iterative RL research.

---

1. Executive Summary

This paper proposes Replay across Experiments (RaE), a simple framework that extends the use of experience replay across multiple RL experiments by mixing data from prior training runs with online data at a fixed ratio (a naïve 50/50 split throughout training) to bootstrap exploration and improve asymptotic controller performance. Empirical validation spans locomotion, vision-based manipulation, and the offline RL Unplugged benchmark across multiple algorithms—DMPO, D4PG, CRR, and SAC-X—demonstrating that RaE matches or exceeds state-of-the-art baselines (including AWAC and fine-tuning) without per-task hyperparameter tuning, with particularly striking gains on challenging egocentric-vision tasks. The method achieves performance improvements with as little as 10,000 prior episodes and is robust to mixing low-return data, establishing that the benefit stems from broader state-distribution coverage rather than reliance on expert trajectories, and that gains plateau after two iterative applications on a single domain.

2. Context and Motivation

The Core Problem: Data Efficiency and Asymptotic Performance in Online RL

The fundamental challenge this paper tackles is deceptively simple: in online reinforcement learning, we almost always throw away data between experiments, and this is wasteful. Every time an RL practitioner launches a new training run—whether to tune hyperparameters, test a new algorithm variant, or simply run another random seed—the interaction data collected during that run is typically discarded once the experiment concludes. The next experiment starts from scratch, with a fresh replay buffer and randomly initialized networks, forced to rediscover the same environment dynamics, the same exploration strategies, and the same reward landscapes that prior runs already mapped out.

This matters for two distinct reasons that the paper highlights implicitly throughout its introduction and explicitly in Section 5. First, asymptotic performance: many challenging RL domains—particularly those with high-dimensional observations (e.g., egocentric vision), sparse rewards, or high-dimensional action spaces—fail to reach satisfactory performance even with state-of-the-art algorithms and generous compute budgets. The paper notes (Section 1) that these problems "remain hard to solve with RL" despite algorithmic advances, leading to "poor asymptotic performance, high variance, low data efficiency, and long training times." Second, research velocity: the iterative cycle of running experiments, analyzing failures, adjusting algorithms or hyperparameters, and re-running is the dominant workflow in RL research and practice. If each iteration must pay the full cost of exploration from scratch, the total wall-clock time and compute budget required to make progress on a domain can be prohibitive—particularly in real-world robotics settings where data collection is physically expensive.

The Gap Between Off-Policy Replay and Cross-Experiment Data Reuse

Experience replay (Lin, 1992) already addresses data efficiency within a single experiment by storing transitions in a buffer and resampling them for training, decoupling data collection from policy optimization. This has become "a principal mechanism underlying the stability and data efficiency of off-policy reinforcement learning" (title/abstract phrase). However, as the paper explicitly notes (Section 1):

"It does not, however, address issues around premature convergence of the function approximators (Ash & Adams, 2020), nor does it directly provide a solution to reusing data from prior experiments."

In other words, replay is scoped to a single run's lifetime. When the run ends—whether because it converged, crashed, or hit a wall-clock limit—the buffer is emptied, and the next run begins anew. This represents a conceptual gap: off-policy algorithms are designed to learn from any data regardless of the collecting policy, yet in practice, they are almost never fed data from prior experiments that explored the same or similar environments. The paper argues that this gap is an artifact of workflow conventions, not a fundamental limitation of the algorithms themselves.

Prior Approaches and Their Shortcomings

The paper positions itself against a landscape of prior work that has attempted to reuse data across experiments or to bootstrap online learning with pre-collected data. It identifies several categories, each with limitations that RaE is designed to avoid:

1. Pure offline RL followed by fine-tuning. Methods like CRR (Wang et al., 2020), CQL (Kumar et al., 2020), and others learn policies entirely from fixed, pre-collected datasets. When fine-tuned online, they typically require specialist algorithmic modifications to avoid instabilities when transitioning from offline to online data—instabilities that arise because the data distribution shifts dramatically once the agent begins collecting its own experience. The paper acknowledges this can work ("simpler off-policy approaches can work well with sufficiently diverse data," citing Yarats et al., 2022) but notes it requires choosing the right offline algorithm and hyperparameters. In the paper's own experiments, fine-tuning CRR works well on the densely rewarded Humanoid run task but "perform[s] poorly in the sparsely rewarded manipulator domains"—a failure case RaE avoids (Section 3.3, discussion of Figure 4).

2. Data mixing with algorithmic complexity. A range of prior work mixes offline or demonstration data into online RL, but each introduces additional machinery. Vecerik et al. (2017) combine demonstration data with prioritized replay, 1-step and N-step returns, and L2 regularization. Nair et al. (2018) add a behavior-cloning loss, a Q-filter, and state-reset mechanisms. Singh et al. (2020b) use a three-stage procedure: first distill prior data from skill experts, then mix with data from a scripted controller, then apply CQL. Walke et al. (2022) train separate forward and backward policies optimized independently before combining. Lee et al. (2022) fine-tune ensembles of offline-trained policies using prioritized replay and a learned density ratio to choose the data mixture. Ball et al. (2023) start with a formulation similar to RaE but then add random ensemble distillation, per-environment design choices, and LayerNorm modifications.

The paper's critique of this body of work is pointed and consistent (Section 4):

"While these methods have shown impressive results on a range of domains, the complexity and specificity of various algorithmic assumptions limit their generality."

Each method introduces hyperparameters (Lagrange multipliers for AWAC, density ratio estimation for Lee et al., ensemble sizes for Ball et al.) that require per-domain tuning. Each couples data reuse to a specific algorithmic formulation, making it difficult to drop into an existing RL codebase that uses a different underlying algorithm. RaE's central claim is that this complexity is unnecessary—a simple fixed-ratio data mix works across algorithms, domains, and data quality regimes without tuning.

3. AWAC: a hybrid with its own limitations. The AWAC algorithm (Nair et al., 2020) sits between offline pre-training and online data mixing. It operates in two phases: first, learning entirely offline for a fixed number of pre-training steps, then switching to online learning with a shared replay buffer using a specific algorithmic formulation (similar to CRR exp with a temperature parameter λ). The paper includes AWAC as a baseline and acknowledges its strengths, but notes (Section 3.2) that it introduces domain-dependent hyperparameters including the number of pre-training steps (swept over 25K, 50K, and 100K steps in their experiments) and the Lagrange multiplier λ (swept over 0.3 and 1.0). The paper's empirical results (Figures 3 and 4) show that RaE matches or exceeds AWAC without requiring any such tuning.

4. Network weight resetting as a confound. An important alternative explanation the paper must rule out: if reloading data from a prior experiment involves reinitializing network weights (since each RaE experiment starts fresh), perhaps the benefit comes entirely from weight resetting rather than data reuse. Prior work by Nikishin et al. (2022) showed that periodic network resets can mitigate "primacy bias"—the tendency of early training data to prematurely commit function approximators to suboptimal solutions—and improve learning on challenging domains like Atari. The paper explicitly addresses this (Section 3.2, "Random Weight Resetting" baseline) by including a control experiment where policy, critic, and optimizer weights are reset periodically (at intervals of K, K/10, and K/100 updates, where K is the number of updates used to train the policy that generated the stored data) without reloading any prior data. This teases apart the effect of weight resets from the effect of data reuse.

How This Paper Positions Itself

The paper's positioning is deliberately minimalist: RaE is not a new algorithm but a workflow change. The abstract describes it as "an effective yet simple framework to extend the use of replays across multiple experiments, minimally adapting the RL workflow." Section 2.2 states the core insight plainly:

"The basic insight of this work is that reusing data across multiple experiments in off-policy learning is a very simple but effective way to accelerate training and improve final performance."

The paper explicitly avoids claiming methodological novelty in data mixing. Instead, it claims that simplicity itself is the contribution—that prior work overcomplicated the problem, and that a naïve 50/50 mix of prior and online data, with no algorithmic modifications, no staged training, and no additional loss terms, achieves results competitive with or superior to far more complex approaches. The introduction frames this as a counterpoint to the trend toward increasing algorithmic complexity:

"We find that the simplest approach, mixing prior and online data with a fixed ratio, is particularly effective across a wide range of application scenarios and algorithms."

Several design choices reinforce this minimalist positioning:

• No algorithmic coupling: RaE works with DMPO, D4PG, CRR, and SAC-X—algorithms with fundamentally different policy optimization mechanisms (EM-style optimization for MPO, deterministic policy gradients for D4PG, advantage-weighted regression for CRR, multi-task scheduled control for SAC-X). This demonstrates that the benefit does not depend on any particular algorithmic inductive bias.
• No hyperparameter tuning: The 50/50 mix ratio is used across all main experiments. The paper reports results "using the same hyperparameter set across all of the main results" (Section 3.2) and notes that ablation of the mixing ratio (Table 1) shows robustness—though optimal ratios vary slightly by data regime, the 50% default works well universally.
• No restrictions on data quality: The paper uses all training data from prior experiments, not just high-return trajectories left in the final replay buffer—a key distinction from approaches that rely on curated expert demonstrations. The ablation in Section 3.4 explicitly tests low-return data (early-training trajectories), mixed-return data, and high-return data, finding that "expert data is the least beneficial in both regimes" (Table 1 discussion), which is a striking result that differentiates RaE from demonstration-based approaches.
• No multi-stage training: Unlike AWAC's offline pre-training phase or the multi-stage pipelines of Singh et al. (2020b) and Walke et al. (2022), RaE mixes data from step zero of online training—the agent never trains purely offline.

The Broader Vision: RaE as a "Project-Long" or "Life-Long" Learning Tool

Beyond the immediate experimental results, the paper articulates a broader vision in Section 5 ("Discussion") that positions RaE as more than a one-shot performance booster. The authors frame it as a workflow philosophy applicable throughout the lifecycle of an RL project:

• Lifelong / project-long learning: Rather than discarding data from failed or suboptimal early experiments, store everything and let later runs benefit from the accumulated exploration. "Even low-return data can be useful to boost performance" (Section 5), which fundamentally changes the cost-benefit calculus of running exploratory experiments.
• Multiple source experiments: When developing controllers for a family of related tasks (e.g., all manipulation skills with a specific robot morphology), data collected for one task may provide useful exploration for another, even if the reward functions differ—the shared dynamics and state-space coverage transfer across tasks.
• Multiple hyperparameters and seeds: Large hyperparameter sweeps and random seed ensembles generate substantial data that is typically discarded after analysis. RaE provides a mechanism to aggregate this data into a single high-performing run, effectively extracting value from the exploration already paid for. The paper demonstrates this concretely in Figure 5c, where combining data across random seeds from a high-variance experiment yields robust performance even when individual seeds vary widely in quality.

This framing connects RaE to the broader theme of amortizing exploration cost in RL. In supervised learning, data is a one-time cost; in online RL, data collection is interleaved with learning, and the cost is paid anew for each experiment. RaE argues that by simply saving and reloading data—a trivial infrastructure change relative to the algorithmic complexity of prior methods—the RL community can fundamentally improve the efficiency of research and deployment.

Why This Paper Matters Now

The paper implicitly argues that RaE is timely for two converging reasons. First, RL is transitioning "from a topic of academic study to a practical tool for the generation of controllers across various real-world applications" (Section 1), citing examples from fusion plasma control (Degrave et al., 2022), stratospheric balloon navigation (Bellemare et al., 2020), data center cooling (Lazic et al., 2018), and autonomous driving (Osinski et al., 2020). In real-world settings, data collection is expensive—physically running robots, interacting with physical systems, or running high-fidelity simulations—and discarding data between experiments is simply not viable at scale. Second, as RL algorithms become more complex and hyperparameter-sensitive (a concern the Henderson et al., 2018 reproducibility crisis highlighted), simple, robust methods that "just work" without tuning become increasingly valuable. The paper's closing sentence captures this ethos: "We believe that as our understanding of RL improves and its use as an engineering and control tool becomes more commonplace, simplicity is key to effective integration."

Potential Tensions the Paper Sets Up

Without explicitly framing them as conflicts, the paper establishes several tensions that its experiments are designed to resolve:

• Complexity vs. performance: Do the algorithmic additions of prior methods (prioritized replay, Q-filters, behavior-cloning losses, ensemble distillation) actually contribute to performance, or is data mixing doing most of the work? If RaE matches AWAC and fine-tuning with zero additional complexity, the burden of proof shifts to proponents of more complex methods to demonstrate that their machinery adds value beyond simple data reuse.
• Expert data vs. diverse data: The demonstration-based literature emphasizes the value of expert trajectories. RaE's finding that low-return data can be more beneficial than expert data (Table 1) challenges this assumption and suggests that state-space coverage—having seen diverse transitions, even suboptimal ones—matters more than reward quality.
• Offline-first vs. online-from-scratch: Methods like AWAC and fine-tuning start offline before transitioning online, which introduces a phase-change in the data distribution that can destabilize learning. RaE mixes data from the beginning, avoiding this distribution shift entirely—a design choice that the paper suggests is more natural for off-policy algorithms that should, in principle, handle any data distribution.

3. Technical Approach

3.1 Reader Orientation

The "system" here is not a new piece of software or a novel neural architecture — it is a workflow convention: whenever you launch an off-policy RL experiment, pre-load the replay buffer with all the interaction data from your previous experiments on the same (or related) domain, keep it there at a fixed 50/50 mix ratio with newly collected online data throughout training, and run your standard off-policy algorithm unchanged. The problem it solves is that RL practitioners routinely discard expensive interaction data between experiments, forcing each new run to rediscover environment dynamics and exploration strategies from scratch, which wastes compute and limits final controller performance. The "shape" of the solution is a minimal change to the data pipeline — add a second, persistent replay mechanism that feeds prior-experience transitions into the training loop alongside online data — without touching the policy optimization algorithm, the network architecture, or the hyperparameter schedule.

3.2 Big-Picture Architecture (Diagram in Words)

The RaE system has four components, only one of which is new relative to a standard off-policy RL setup:

1. Persistent offline data store — a long-lived storage system (disk, database, or distributed file system) that accumulates all interaction transitions $(s, a, r, s')$ from every experiment in a project. This is the only new infrastructure component. It replaces the implicit convention of discarding replay buffers at experiment termination.

2. Offline replay sampler — at each training step, a fixed fraction of the mini-batch (50% for all main experiments) is drawn uniformly from this persistent store. This is structurally identical to a standard replay buffer sampler, just pointing at a different data source.

3. Online replay buffer — the standard FIFO or reservoir-sampled buffer that accumulates transitions from the current experiment's policy interacting with the environment. The remaining 50% of each mini-batch is drawn from here.

4. Off-policy RL agent — completely unchanged. This is the policy network, value/critic network(s), and optimization procedure (DMPO, D4PG, CRR, SAC-X, etc.) that consumes the mixed mini-batches and produces parameter updates exactly as it would in a standard single-experiment run.

Information flows as follows: the agent interacts with the environment, pushing transitions into the online buffer (step 1). At each training step, a mini-batch is assembled by concatenating $\text{batch\_size}/2$ samples from the online buffer and $\text{batch\_size}/2$ samples from the offline store (step 2). The agent computes gradients and updates parameters using this mixed batch with its standard loss functions — no auxiliary losses, no constraints, no schedule changes (step 3). When the experiment ends, all transitions in the online buffer (and any still in the environment pipeline) are flushed to the persistent offline store, making them available for the next experiment (step 4). The next experiment reinitializes all network weights and optimizer state from scratch, loads the now-larger offline store, and repeats the cycle.

3.3 Roadmap for the Deep Dive

• First, the formal problem setting and notation (MDP, off-policy learning, experience replay) that RaE operates within, since the method's simplicity is best appreciated against the backdrop of what a standard off-policy training loop already does.

• Second, the RaE mechanism itself — the mixing procedure, the fixed-ratio design, the data management protocol across experiment boundaries, and what distinguishes it from prior data-reuse methods — since this is the paper's sole technical contribution.

• Third, the algorithms RaE is evaluated with (DMPO, SAC-Q, CRR, D4PG), explained at the level of what each algorithm optimizes and how it consumes replay data, since the paper's central claim is that RaE is algorithm-agnostic and this must be justified by showing that these algorithms share no special property that RaE exploits.

• Fourth, the baseline methods (fine-tuning, AWAC, random weight resetting) that RaE is compared against, since understanding what RaE is not — no offline pre-training phase, no auxiliary losses, no staged training — clarifies the design philosophy.

• Fifth, the ablation framework — the data regimes (High Return, Mixed Return, Low Return), the dataset sizes (10K vs 100K episodes), and the mixing ratios (50%, 70%, 80%, 90% online) — since this is where the paper empirically validates the robustness that justifies the minimalist design.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems-and-empirical-methods paper whose core idea is that a trivial modification to the data pipeline of off-policy RL — pre-loading the replay buffer with data from prior experiments and maintaining a fixed mixing ratio throughout training — recovers most or all of the benefits of far more complex data-reuse methods, without introducing algorithmic coupling, hyperparameter sensitivity, or domain-specific tuning.

---

The Standard Off-Policy RL Loop (What RaE Modifies)

Before explaining what RaE adds, it is essential to understand exactly what it changes. The paper assumes a standard off-policy deep RL setup (Section 2.1), which operates as follows.

Environment formalization. The problem is modeled as a Markov Decision Process (MDP) with state space $\mathcal{S}$, action space $\mathcal{A}$, transition probability $p(s_{t+1} \mid s_t, a_t)$, and reward function $r(s_t, a_t)$. The agent's behavior is specified by a deep neural network policy $\pi(a_t \mid s_t; \theta)$ parameterized by $\theta$. The optimization objective is the standard expected discounted return:

$J(\pi) = \mathbb{E}_{\rho_0(s_0), \, p(s_{t+1} \mid s_t, a_t), \, \pi(a_t \mid s_t)} \left[ \sum_{t=0}^{\infty} \gamma^t r_t \right]$

where $\rho_0(s_0)$ is the initial state distribution, $\gamma \in [0,1]$ is the discount factor that trades off immediate versus future rewards, and $r_t = r(s_t, a_t)$ is the scalar reward at timestep $t$.

What it computes: the expected sum of discounted future rewards when the agent starts from the initial state distribution and follows policy $\pi$ thereafter. The expectation is over three sources of randomness: the initial state draw, the environment's stochastic transitions, and the policy's own action sampling.

Why this form: the discounted sum formulation is standard because it ensures the infinite sum converges (for $\gamma < 1$) and because $\gamma$ provides a tunable knob for how far-sighted the agent should be — $\gamma$ close to 1 weights distant rewards heavily, while $\gamma$ close to 0 makes the agent myopic. This formalism underlies essentially all modern deep RL and is not specific to RaE.

The critic (state-action value function). The Q-function $Q(s_t, a_t)$ is defined as the expected discounted return when taking action $a_t$ in state $s_t$ and then following policy $\pi$ for all subsequent steps:

$Q(s_t, a_t) = r(s_t, a_t) + \gamma \, \mathbb{E}_{p(s_{t+1} \mid s_t, a_t), \, \pi(a \mid s_{t+1})} \left[ Q(s_{t+1}, a) \right]$

where the expectation is over the next state from the environment dynamics and the next action from the policy.

What it computes: a recursive decomposition of the expected return — the immediate reward plus the discounted expected value of being in the next state and acting according to $\pi$. This is the Bellman equation, and it is the theoretical foundation for all value-based and actor-critic RL methods.

Why this form: the recursive structure enables bootstrapping — the Q-function can be learned from data by minimizing the temporal difference (TD) error between the left-hand side (current Q-estimate) and the right-hand side (a target computed from observed rewards and next-state Q-estimates). This makes off-policy learning possible, because the target can be computed from any transition $(s_t, a_t, r_t, s_{t+1})$ regardless of which policy collected it.

Experience replay (Lin, 1992). The standard off-policy training loop interleaves two processes:

1. Data collection: The agent executes its current policy (or an exploratory variant) in the environment, producing trajectories of $(s_t, a_t, r_t, s_{t+1})$ transitions. These are stored in a finite-capacity replay buffer $\mathcal{D}_{\text{online}}$, typically implemented as a FIFO queue or a reservoir-sampled buffer that overwrites old data as new data arrives.

2. Training: At each training step, a mini-batch of $B$ transitions is sampled uniformly from $\mathcal{D}_{\text{online}}$. The critic is updated by minimizing the distributional TD error (for the algorithms used in this paper — DMPO, D4PG, CRR — the critic is a categorical distributional Q-function as in Barth-Maron et al., 2018). The policy is updated using the appropriate mechanism for the algorithm (an EM-style optimization for MPO, a deterministic policy gradient for D4PG, advantage-weighted regression for CRR; see below).

The key property of this setup is that the data in $\mathcal{D}_{\text{online}}$ comes from a mixture of past policies — as the policy improves, the buffer contains transitions from older, worse policies alongside newer, better ones. This is what gives off-policy RL its data efficiency relative to on-policy methods (which can only use data from the most recent policy). However, the mixture is always confined to policies from the current experiment. When the experiment terminates, the buffer is discarded.

The interaction problem RaE addresses. Several failure modes arise from this single-experiment scoping (Section 1):

• Premature convergence of function approximators (Ash & Adams, 2020): early in training, the replay buffer contains only data from a nearly random policy. The critic and policy overfit to this narrow distribution, and later exploration — which would expand the buffer to include higher-reward regions — is inhibited because the policy has already converged to a local optimum dictated by the early data.

• Exploration bottlenecks: in sparse-reward or high-dimensional domains, the agent may never discover rewarding behavior within a single experiment's budget, because its initial random exploration does not stumble upon reward. No amount of replay within the experiment can help, because there is no reward signal in the buffer to bootstrap from.

• Per-experiment amortization failure: each experiment independently pays the cost of exploring the state space, even if prior experiments (with different hyperparameters, different random seeds, or earlier algorithm versions) have already mapped out large regions of it. This is particularly painful in real-world robotics, where data collection is physically expensive.

---

The RaE Mechanism: Fixed-Ratio Data Mixing Across Experiments

RaE modifies the standard loop with exactly one change: the mini-batch at each training step is assembled from two data sources rather than one. The paper states this operationally (Section 2.2):

"The only required algorithmic change is the availability of a second replay mechanism that allows replaying prior and online data with a particular fixed ratio throughout the course of training (we use a naïve 50/50 mix of offline and online data for our main results, without optimizing this ratio)."

The persistent offline store. Let $\mathcal{D}_{\text{offline}}$ denote the union of all transitions collected across all prior experiments on the same (or related) domain. Crucially (Section 2.2, footnote):

"We store and reuse all training data across experiments, not just trajectories left in the final replay."

This means $\mathcal{D}_{\text{offline}}$ includes:

• Early-exploration data from the first few thousand episodes of prior runs (low-return, high-entropy transitions).
• Mid-training data from the period when the policy was improving but not yet converged.
• Final-convergence data from the end of prior runs (high-return, on-policy-like transitions).
• Data from failed experiments, crashed runs, hyperparameter variants that performed poorly, and random seeds with high variance.

In other words, no curation, no filtering, no sub-sampling — store everything. This is a deliberate design choice that differentiates RaE from demonstration-based approaches (which only keep high-return or expert trajectories) and from offline RL benchmark protocols (which often sub-sample or rebalance datasets, as in RL Unplugged's procedure of reducing successful episodes by 2/3).

The mixing procedure. At each training step, the agent samples a mini-batch of size $B$ (128 for Locomotion Soccer, 256 for ablation experiments) as follows:

1. Draw $\lfloor B/2 \rfloor$ transitions uniformly at random from $\mathcal{D}_{\text{offline}}$.
2. Draw $\lceil B/2 \rceil$ transitions uniformly at random from the online buffer $\mathcal{D}_{\text{online}}$ (which accumulates transitions from the current policy as usual).
3. Concatenate these two half-batches into a single mini-batch of size $B$.
4. Feed the concatenated batch to the standard algorithm update — compute TD errors for the critic, compute policy updates via the algorithm's usual mechanism, update network parameters via the usual optimizer (AdamW or similar).

The mixing ratio is fixed at 50/50 for all main experiments (Figures 3, 4, 5a-c). The ablation in Table 1 explores ratios of 50%, 70%, 80%, 90%, and 100% online (100% online is learning from scratch), but these are not tuned per-domain — the paper reports 50/50 results for all main experiments regardless of domain, algorithm, or data regime.

What happens at experiment boundaries. When an RaE experiment terminates:

1. All transitions that accumulated in the online buffer $\mathcal{D}_{\text{online}}$ during the experiment are appended to $\mathcal{D}_{\text{offline}}$, making them available for the next experiment.
2. The next experiment initializes with fresh random network weights for policy, critic, and optimizer — there is no weight transfer, no fine-tuning from a pre-trained checkpoint. This is explicit (Section 2.2): "At the beginning of each training run, policy and value-function are re-initialized in line with stand-alone experiments."
3. The next experiment loads the now-larger $\mathcal{D}_{\text{offline}}$ and begins training with the 50/50 mix from step zero.

This means RaE is fundamentally different from fine-tuning: there is no offline pre-training phase, no weight transfer, and no phase change in the data distribution (since offline data is present from the first gradient step). The paper emphasizes this distinction by noting that RaE "does not require any changes with respect to the RL agent itself and is generally agnostic to agent and architecture changes across experiments" (Section 2.2).

The iterative application of RaE. The paper explores what happens when RaE is applied repeatedly (Section 3.4, "Iterative improvement," Figure 5a). The procedure is:

1. Run experiment E0 with some amount of online data collection (e.g., 10,000 episodes).
2. Run RaE iteration 1: load E0's data into $\mathcal{D}_{\text{offline}}$, train with 50/50 mix, collect new online data. This produces a higher-performing policy than E0.
3. Run RaE iteration 2: load E0's data plus RaE iteration 1's data into $\mathcal{D}_{\text{offline}}$, train with 50/50 mix, collect new online data.
4. Continue until a performance plateau is reached.

The paper finds (Figure 5a) that "a small gain in asymptotic performance and speed of learning [occurs] even on the second iteration although a performance plateau is reached on a third iteration." The practical implication (Section 5) is that "in some settings it may be preferable to break a single training run into smaller runs for iterative performance improvements" — rather than one massive training run, do multiple shorter runs where each builds on the data of all previous ones.

Why 50/50 and not something adaptive? The paper does not provide a theoretical justification for the 50/50 ratio; it is an empirical choice validated by the robustness analysis in Table 1. The philosophy is deliberately anti-tuning: the method should work out of the box without per-domain ratio optimization. The ablation in Table 1 shows that while the optimal ratio does vary somewhat by data regime (more online data is beneficial when offline data is scarce — 90% online with only 10K episodes of low-return data achieves 108% of from-scratch performance, while 50% online with 100K episodes of mixed-return data achieves 112%), the 50% default is consistently competitive. The paper explicitly states (Section 3.2): "RaE did not require any tuning and we report results using the same hyperparameter set across all of the main results (50% offline data)."

Why uniform sampling from offline data and not prioritized? Prior work (Vecerik et al., 2017; Lee et al., 2022) uses prioritized replay or density-ratio-based sampling to upweight informative transitions from offline data. RaE uses uniform sampling, which is simpler and has no additional hyperparameters (priority exponent, importance-sampling correction). The paper does not ablate uniform vs. prioritized sampling, but the strong empirical results with uniform sampling suggest that for the domains tested, the diversity of the offline data (coming from all stages of prior training) provides sufficient coverage that uniform sampling works adequately. The finding that "expert data is the least beneficial" (Table 1 discussion) further supports this: if the most valuable transitions for learning are not concentrated in high-return trajectories, then a prioritization scheme that upweights high-return data would be counterproductive.

Why reinitialize weights and not fine-tune? The paper's design choice to reinitialize weights is partly philosophical (demonstrate that data alone drives the benefit, not weight transfer) and partly practical (algorithm-agnosticism — weight transfer assumes the same network architecture across experiments, while RaE is "generally agnostic to agent and architecture changes across experiments"). The "Finetuning with RaE" experiment in Appendix C.2 (Figure 7) shows that combining weight transfer with data mixing ("finetuning with RaE") yields faster initial learning than RaE alone while matching its asymptotic performance, suggesting that the two mechanisms are complementary. However, the paper's main results use weight reinitialization to isolate the effect of data reuse.

---

The Algorithms RaE Operates With: DMPO, SAC-Q, CRR, D4PG

A central claim of the paper is that RaE is algorithm-agnostic. To validate this, the authors evaluate it with four fundamentally different off-policy algorithms across different domains. Understanding how these algorithms differ — and what they share — clarifies why RaE's simplicity is surprising and important.

DMPO (Maximum a Posteriori Policy Optimization) is used for the Locomotion Soccer domains (both state and vision). DMPO, introduced by Abdolmaleki et al. (2018) and described in Appendix B, optimizes the RL objective through an Expectation-Maximization (EM) framework:

• E-step: Given the current Q-function, compute a non-parametric improved policy $q(a \mid s)$ by solving a constrained optimization problem that maximizes expected Q-value while staying close (in KL-divergence) to the current parametric policy $\pi(a \mid s)$:

$\max_q \int_s \mu(s) \int_a q(a \mid s) \, Q(s, a) \, da \, ds \quad \text{s.t.} \quad \int_s \mu(s) \, D_{\text{KL}}(q(a \mid s) \, \| \, \pi(a \mid s)) \, ds < \epsilon$

where $\mu(s)$ is the state distribution from the replay buffer, $\epsilon$ is a KL-constraint threshold that controls how far the improved policy can deviate from the current one.

What it computes: for each state in the batch, find a distribution over actions that maximizes expected Q-value while staying within $\epsilon$ KL-divergence of the current policy. The solution has a closed form:

$q(a \mid s) \propto \pi(a \mid s) \exp\left( \frac{Q(s, a)}{\eta^*} \right)$

where $\eta^*$ is the optimal dual variable (temperature) found by solving the dual to the constrained problem — it effectively controls how much the Q-function can override the prior policy.

Why this form: the exponential weighting by Q-value means actions with higher estimated returns get upweighted exponentially, while the KL constraint prevents the improved policy from collapsing to a point mass at the maximum-Q action (which would be pure exploitation and destroy exploration). The temperature $\eta^*$ adapts automatically: when Q-values are well-separated, $\eta^*$ is small (strong exploitation); when Q-values are uncertain, $\eta^*$ is large (stay close to the prior policy).

• M-step: Fit the parametric policy $\pi_\theta$ to the non-parametric targets $q(a \mid s)$ via weighted maximum-likelihood supervised learning:

$\pi_{n+1} = \arg\max_{\pi_\theta} \sum_j \sum_i q_{ij} \log \pi_\theta(a_i \mid s_j) \quad \text{s.t.} \quad D_{\text{KL}}(\pi_n(a \mid s_j) \, \| \, \pi_\theta(a \mid s_j)) < \epsilon_M$

where $q_{ij}$ is the E-step target probability for action $a_i$ in state $s_j$.

What it computes: maximize the log-probability of actions under the parametric policy, weighted by the E-step target probabilities $q_{ij}$, with an additional KL trust-region constraint on the parametric policy update to prevent destructive parameter updates.

Why this form: the two-step EM procedure decouples the exploration-exploitation tradeoff (handled in the E-step via the temperature $\eta^*$) from the policy representation learning (handled in the M-step via standard supervised learning). This makes DMPO relatively stable compared to pure policy-gradient methods, because the M-step is a convex optimization problem with a well-behaved loss landscape.

DMPO uses a distributional critic (Barth-Maron et al., 2018) with N-step returns (N=5 for locomotion, N=1 for RL Unplugged due to single-step dataset constraints). The distributional critic represents Q-values as a categorical distribution over a discretized value range rather than a scalar, which captures uncertainty and improves stability. The critic update uses the N-step distributional Bellman operator:

$(\mathcal{T}_\pi^N Q)(s_0, a_0) = r(s_0, a_0) + \mathbb{E} \left[ \sum_{n=1}^{N-1} \gamma^n r(s_n, a_n) + \gamma^N Q(s_N, \pi(a_N \mid s_N)) \mid s_0, a_0 \right]$

where the expectation is over N-step transition dynamics and the Q-distribution.

Why N-step returns: single-step returns (N=1) propagate reward information slowly through the value function — it takes many updates for a reward at timestep T to influence the Q-value at timestep 0. N-step returns with N=5 shortcut this by directly incorporating up to 5 future rewards into the target, accelerating credit assignment. This is standard in deep RL and is not specific to RaE.

SAC-Q (Scheduled Auxiliary Control with Q-learning) is used for the Manipulation RGB Stacking domain. Introduced by Riedmiller et al. (2018) and described in Appendix B, SAC-Q builds on a multi-headed network architecture to handle multiple auxiliary tasks. The key components are:

• Multi-headed networks: Both the policy and critic share a common "torso" (early layers of the neural network) with separate output "heads" for each subtask. In the Manipulation domain, subtasks include reaching, lifting, placing, and stacking. The shared torso learns representations that transfer across subtasks, while the per-task heads specialize.

• Scheduler: At fixed intervals during each episode, a scheduler process selects which subtask head to activate. The scheduler itself is learned: it trains a Q-function that optimizes the sequence of tasks to maximize the total return on the main goal task (stacking). This implements a curriculum — the scheduler learns to sequence easier subtasks (reach, grasp, lift) before attempting the harder ones (place, stack).

• Policy and Q-function updates: Under the hood, SAC-Q uses the same MPO update rules as DMPO with a distributional critic. The difference is that the states and actions are now task-conditioned — the policy outputs and Q-values depend on both the state and the active task index.

The key insight for RaE is that the offline data from prior SAC-Q runs contains transitions from all subtasks the scheduler explored, providing broad state-action coverage that helps subsequent runs regardless of which subtask sequence the new run's scheduler chooses.

CRR (Critic Regularized Regression) is used for the RL Unplugged domains and as the offline pre-training component of the fine-tuning and AWAC baselines. Introduced by Wang et al. (2020) and described in Appendix B, CRR is an offline RL algorithm designed to learn from fixed datasets without online interaction. Its policy update is:

$\arg\max_\pi \mathbb{E}_{(s, a) \sim \mathcal{D}} \left[ f(Q, \pi, s, a) \log \pi(a \mid s) \right]$

where $f$ is a non-negative, scalar filter function that increases monotonically with Q-value, and $\mathcal{D}$ is the offline dataset.

What it computes: a weighted maximum-likelihood objective where each datapoint $(s, a)$ is weighted by how good the action $a$ is according to the current Q-function. Actions with high Q-values (likely good actions) get high weight and the policy is encouraged to reproduce them; actions with low Q-values (likely bad actions) get low or zero weight and are effectively ignored.

Why this form: standard behavior cloning (maximizing $\log \pi(a \mid s)$ uniformly) would imitate all actions in the dataset, including suboptimal ones. CRR's Q-filter selectively imitates only the good actions, making it suitable for offline datasets that contain a mix of good and bad behavior. This is crucial for RaE's use case, where $\mathcal{D}_{\text{offline}}$ contains data from all stages of prior training, including early random exploration.

The paper considers two variants of the filter function $f$:

CRR binary uses a hard threshold on the advantage function $A(s, a) = Q(s, a) - \frac{1}{m} \sum_{j=1}^m Q(s, a_j)$ where $a_j \sim \pi(\cdot \mid s)$:

$f := \mathbf{1}[A(s, a) > 0]$

What it computes: a binary filter — weight is 1 if the action is better than the average action under the current policy (positive advantage), 0 otherwise. This discards the bottom half of the data distribution completely.

Why this form: simple, no hyperparameters, and works well when the dataset has a clear separation between good and bad actions. However, it can be brittle when the advantage distribution is continuous and the optimal cutoff is unclear.

CRR exp uses a soft exponential weighting:

$f := \exp(A(s, a) / \beta)$

where $\beta$ is a temperature hyperparameter.

What it computes: a soft continuous weighting — actions with higher advantage get exponentially higher weight, but all actions contribute something (no hard zeros). $\beta$ controls sharpness: small $\beta$ creates a peaked distribution focused on the very best actions; large $\beta$ approaches uniform weighting.

Why this form: the soft weighting is more robust when the optimal cutoff is unclear, but introduces the temperature $\beta$ as a hyperparameter that may need tuning per domain.

The paper uses both CRR binary and CRR exp for the offline and fine-tuning baselines, selecting the best-performing variant per domain (Section 3.2: "we choose the best performing seed from across these runs"). RaE itself does not use CRR's specialized policy update — when RaE is applied with CRR on RL Unplugged, it uses the same CRR update rule but with data from both online and offline sources mixed at 50/50, rather than purely offline data.

D4PG (Distributed Distributional Deterministic Policy Gradients) is used for an ablation experiment (Figure 5b) demonstrating algorithm-agnosticism on the Locomotion Soccer State domain. Introduced by Barth-Maron et al. (2018), D4PG is a deterministic actor-critic algorithm that:

• Uses a deterministic policy $\mu_\theta(s)$ that directly outputs actions rather than a distribution.
• Uses a distributional critic trained with N-step returns (same distributional Bellman operator as DMPO).
• Updates the policy via the deterministic policy gradient: $\nabla_\theta J = \mathbb{E}_{s \sim \mathcal{D}} \left[ \nabla_a Q(s, a) \mid_{a = \mu_\theta(s)} \nabla_\theta \mu_\theta(s) \right]$, which pushes the policy toward actions that maximize the Q-function.

The key difference from DMPO: D4PG's policy update is a simple gradient ascent on Q-values with no KL constraint or EM decomposition. The fact that RaE works with both DMPO (constrained EM-style optimization) and D4PG (unconstrained gradient ascent) supports the claim of algorithm-agnosticism — the benefit of data mixing does not depend on the specific mechanism by which the policy is optimized.

---

Baseline Methods: What RaE Is Compared Against

To establish that the fixed-ratio data mix is sufficient — and that prior methods' additional complexity is unnecessary — the paper compares RaE against three categories of baselines (Section 3.2).

Fine-tuning (CRR → online). This baseline represents the dominant paradigm in offline-to-online RL: first train a policy and Q-function entirely offline on the prior data, then fine-tune online with new data. The procedure:

1. Choose the best offline algorithm and hyperparameters by sweeping across BC (behavior cloning), CRR binary, and CRR exp, training each purely offline on $\mathcal{D}_{\text{offline}}$.
2. Select the best-performing seed from this sweep.
3. Initialize a new online run with the pre-trained policy and Q-function weights.
4. Continue training online, with new data added to the replay buffer alongside the original offline data (standard off-policy learning, no special mixing ratio).

This baseline tests whether pre-training weights offline — which requires selecting the right offline algorithm and hyperparameters — provides benefits beyond what simple data mixing achieves. The paper's results (Figure 3) show that fine-tuning performs similarly to RaE on Manipulation tasks but "perform[s] poorly in the sparsely rewarded manipulator domains" of RL Unplugged (Section 3.3, discussion of Figure 4), while RaE "works well across all settings."

AWAC (Advantage Weighted Actor Critic). Introduced by Nair et al. (2020), AWAC combines offline pre-training with a specific online algorithm. The procedure:

1. Train entirely offline for a fixed number of pre-training steps (the authors prescribe 25,000; the paper additionally sweeps 50,000 and 100,000).
2. After the pre-training phase, switch to online learning with a shared offline-online replay buffer.
3. During online learning, use the AWAC policy update, which is similar to CRR exp:

$\arg\max_\pi \mathbb{E}_{(s, a) \sim \mathcal{D}_{\text{mixed}}} \left[ \exp\left( \frac{A(s, a)}{\lambda} \right) \log \pi(a \mid s) \right]$

where $\lambda$ is a temperature parameter (swept over 0.3 and 1.0 as prescribed), and $\mathcal{D}_{\text{mixed}}$ contains both offline and online data.

What distinguishes AWAC from RaE:

• AWAC has a phase change: pure offline → mixed online. RaE mixes from step zero.
• AWAC has a domain-dependent hyperparameter: the number of pre-training steps. RaE has none.
• AWAC uses a specific algorithmic formulation (exponential advantage weighting) that couples data reuse to the policy update mechanism. RaE does not touch the policy update.

Why this baseline matters: if AWAC's algorithmic machinery is necessary for strong performance when reusing data, then RaE should underperform AWAC. The paper's results (Figures 3 and 4) show RaE at-par or better, suggesting the machinery is not necessary.

Random Weight Resetting. This baseline teases apart the effect of weight reinitialization (which RaE implicitly does at experiment boundaries) from the effect of data reuse. The procedure (Section 3.2):

1. Run a standard online experiment from scratch — no offline data, no mixing.
2. At regular intervals, reset all network weights (policy, critic) and optimizer state to random initialization.
3. Continue training with the same online replay buffer (which is NOT cleared — it continues accumulating data from before and after the reset).
4. Sweep over reset frequencies: every $K$ updates (where $K$ is the number of updates used to train the original data-generating policy), every $K/10$ updates, and every $K/100$ updates. For RL Unplugged (where $K$ is unknown), use 10,000, 100,000, and 1,000,000 updates.

What this isolates: if weight resetting alone (without additional data) drives the benefits observed with RaE, then the random-resetting baseline should match RaE's performance. The paper's results (Figures 3 and 4) show that RaE consistently outperforms random resetting, confirming that data reuse — not weight reinitialization — is the active ingredient. The paper explicitly states this (Section 3.2): "To tease apart the effect of weight resets from the benefit of data reloading, we consider a baseline where all weights (policy, critic and optimizer) are reset every K policy updates."

---

The Ablation Framework: Testing Robustness to Data Quantity, Quality, and Mixing Ratio

Section 3.4 presents a systematic ablation study on the Locomotion Soccer State task designed to answer four questions the paper poses explicitly:

"How much data is required for performance improvements? What kind of data is best for mixing: expert data, early training data or a mix? How sensitive is RaE to the ratio of online to offline data in different data regimes? Is there an advantage of applying RaE repeatedly across experiment iterations?"

Data regimes. The ablation constructs three qualitative categories of offline data, all drawn from a single prior training run that collected 400,000 episodes (the full dataset):

• High Return: data generated only from the end of training, corresponding to "highly rewarding trajectories or 'expert' data." Sampled by recency — take the last $N$ episodes from the run.
• Mixed Return: data sampled uniformly at random throughout the entire training run, corresponding to "a mixed regime with high and low return trajectories."
• Low Return: data generated only from the start of training, corresponding to "early low return data." Sampled by recency — take the first $N$ episodes from the run.

Dataset sizes. For each data regime, two sizes are tested: 10,000 episodes ("lower data regime") and 100,000 episodes (10× larger). This tests whether RaE remains effective when only a small amount of prior data is available.

Mixing ratios. For each combination of data regime and dataset size, the paper varies the fraction of online data in each mini-batch: 50% (the default), 70%, 80%, 90%, and 100% (100% online is the from-scratch baseline, with no offline data). The 100% baseline establishes the reference performance: all results in Table 1 are expressed as a percentage of the asymptotic reward achieved when learning entirely from scratch (100% online, 400,000 online episodes).

What Table 1 tells us. The paper summarizes several patterns from this factorial experiment:

• In the low-data regime (10K episodes): using more online data is beneficial. For Low Return data, 50% online achieves only 51% of from-scratch performance (worse than learning from scratch!), but 90% online achieves 108% — a modest improvement. For Mixed Return data, 50% online achieves 90%, while 90% online achieves 124% — a substantial improvement. This makes intuitive sense: when the offline dataset is small, the agent risks overfitting to it. Using a higher fraction of fresh online data prevents this overfitting.

• In the higher-data regime (100K episodes): a lower online ratio works better. For Mixed Return data, 50% online achieves 112% of from-scratch performance — better than 90% online at 124% for the smaller dataset? Wait, that's inconsistent. Let me re-examine Table 1 more carefully.

Re-reading Table 1:

• Low Return, 10K episodes, 50% online: 51% (worse than scratch)
• Low Return, 10K episodes, 90% online: 108% (better than scratch)
• Low Return, 100K episodes, 50% online: 97% (roughly at scratch)
• Low Return, 100K episodes, 90% online: 108% (slightly better)
• Mixed Return, 10K episodes, 50% online: 90% (worse than scratch)
• Mixed Return, 10K episodes, 90% online: 124% (substantially better)
• Mixed Return, 100K episodes, 50% online: 112% (better than scratch)
• Mixed Return, 100K episodes, 70% online: 110%
• Mixed Return, 100K episodes, 80% online: 106%
• Mixed Return, 100K episodes, 90% online: 110%

The pattern: with 10K episodes, low ratios of online data (50%) can actually hurt performance relative to learning from scratch — the agent overfits to the small offline dataset. Higher online ratios (80-90%) are necessary to see benefits, and the benefits can be substantial (124% of from-scratch performance). With 100K episodes, even 50% online provides benefits (112%), and higher ratios provide similar but not dramatically larger benefits. The paper states (Section 3.4): "In the lower data regime... a mixture with more online data is beneficial. However, as more data becomes available, a lower ratio works better. We hypothesize that using more online data for learning prevents over-fitting to a small set of offline trajectories."

• Low-return data can be surprisingly beneficial. With 10K episodes, Low Return data at 90% online achieves 108% — outperforming from-scratch learning with data that, by definition, contains no successful trajectories. With 100K episodes, Low Return data at 50% online achieves 97% — essentially matching from-scratch performance using only data from the early, pre-success phase of training. The paper highlights this (Section 3.4): "In the lower data regime, low return data tends to be the most beneficial. As the dataset size increases though, a mix of high and low return trajectories provide a greater benefit."

• Expert data is the least beneficial. High Return data (expert trajectories from the end of training) underperforms Mixed Return data in every comparable cell. With 10K episodes at 90% online, High Return achieves 120% vs. Mixed Return's 124%; with 100K episodes at 50% online, High Return achieves 119% vs. Mixed Return's 112%. Wait, actually 119 > 112 — so High Return performs better in the 100K/50% online cell. But the paper states (Section 3.4): "Surprisingly, expert data is the least beneficial in both regimes." Looking at the table more carefully:

At 10K episodes:

• High Return, 50% online: 51%
• Low Return, 50% online: 80%
• Mixed Return, 50% online: 90%
• High Return, 70% online: 80%
• Low Return, 70% online: 101%
• Mixed Return, 70% online: 110%
• High Return, 90% online: 120%
• Low Return, 90% online: 108%
• Mixed Return, 90% online: 124%

At 100K episodes:

• High Return, 50% online: 119%
• Low Return, 50% online: 97%
• Mixed Return, 50% online: 112%
• High Return, 80% online: 98%
• Low Return, 80% online: 126%
• Mixed Return, 80% online: 106%
• High Return, 90% online: 120%
• Low Return, 90% online: 108%
• Mixed Return, 90% online: 110%

The paper's summary that "expert data is the least beneficial" is a generalization — it's true for most cells, but at 100K/50% online, High Return outperforms Low Return. The more robust pattern is that Mixed Return data consistently performs well and that Low Return data is surprisingly useful given that it contains no successful trajectories. The paper's explanation (Section 3.4): "This indicates that the advantage of mixing data may stem from the benefit of having a larger state distribution with mixed rewards."

Why low-return data helps at all. The paper does not provide a mechanistic explanation, but the RL literature offers a plausible one: low-return data from early in training provides broad state-space coverage. Even though the trajectories are suboptimal or reward-free, they show the agent what states exist and how transitions work — information that is valuable for the critic to learn accurate Q-values in regions of state space that the current online policy might not otherwise visit. This is consistent with the finding by Yarats et al. (2022) that "don't change the algorithm, change the data" — the data distribution matters more than algorithmic sophistication for offline RL. RaE leverages this insight by keeping all data, not just expert data.

4. Key Insights and Innovations

Innovation 1: Simplicity Itself as the Core Contribution — the "Null Hypothesis" of Data Reuse

The paper's most intellectually distinctive move is not a new algorithm, architecture, or theoretical result. It is the deliberate reframing of data reuse in RL as a problem that prior work overcomplicates, and the corresponding empirical demonstration that the simplest possible approach — a fixed 50/50 mix of prior and online data with zero algorithmic modifications — matches or exceeds sophisticated state-of-the-art methods across diverse domains and algorithms. This is best understood as establishing the "null hypothesis" for cross-experiment data reuse: before adding prioritized replay, Q-filters, behavior-cloning losses, ensemble distillation, multi-stage training, or learned density ratios, the field should first ask whether a naïve uniform mixture would have sufficed.

This reframing matters because it inverts the burden of proof. Prior work in this space — Vecerik et al. (2017), Nair et al. (2018), Singh et al. (2020b), Walke et al. (2022), Lee et al. (2022), Ball et al. (2023) — each introduced algorithmic machinery justified by the assumption that effectively using prior data requires specialized handling: prioritized sampling to upweight informative transitions, auxiliary losses to prevent catastrophic forgetting, staged training to manage distribution shift, or per-domain hyperparameters to balance offline and online objectives. RaE's results (Figures 3, 4, Table 1) show that on the domains tested, none of this machinery is necessary. The paper is careful not to claim these methods are worthless — they may prove essential in domains beyond those studied — but the empirical case that they are over-specified for the tested settings is strong.

What makes this a genuine innovation rather than an obvious baseline is the counterintuitive finding that the simplest approach had not been systematically evaluated as a competitive method in its own right. The paper's literature review (Section 4) reveals a field that jumped directly from pure offline RL (CRR, CQL) to algorithmically complex online-offline hybrids (AWAC, ensemble distillation) without first establishing how far uniform data mixing alone could go. RaE fills this gap, and in doing so, it provides the community with a simple, robust, and essentially free baseline that any future method claiming to improve data reuse should be required to beat. This is a contribution to experimental methodology as much as to RL algorithms — analogous to how the recognition that random search often matches sophisticated optimizers changed the hyperparameter optimization literature.

The paper's deliberate refusal to optimize the mixing ratio (sticking with 50/50 for all main results, stating explicitly in Section 3.2 that "RaE did not require any tuning") is itself a methodological statement. It argues that robustness across domains without tuning is a first-order design goal for practical RL workflows, and that the community's tolerance for per-domain hyperparameter optimization in prior methods obscured how much performance could be achieved without it. The fact that RaE matches AWAC — which requires sweeping over pre-training steps (25K, 50K, 100K) and a Lagrange multiplier λ (0.3, 1.0) — without any such tuning (Figures 3, 4) is the paper's strongest argument for simplicity-as-contribution.

Innovation 2: Broad State-Distribution Coverage Over Expert Trajectories — a Diagnostic Finding That Challenges Prevailing Assumptions

The paper's second major conceptual contribution emerges from the ablation study in Table 1: low-return data from early in training can be more beneficial for bootstrapping new runs than expert demonstrations, and expert data alone is often the least useful data regime. This finding runs counter to the dominant assumption in the demonstration-based RL literature (Vecerik et al., 2017; Nair et al., 2018; Bohez et al., 2022), which implicitly or explicitly treats high-return trajectories as the most valuable data source and designs mechanisms (prioritized replay, Q-filters, behavior-cloning losses) to focus learning on them.

The empirical evidence is concrete. With 10,000 episodes of offline data at a 90% online mixing ratio, Mixed Return data achieves 124% of from-scratch asymptotic performance, while Low Return data — which by definition contains zero successful trajectories — achieves 108%, and High Return data achieves 120%. With 100,000 episodes at 80% online, Low Return data reaches 126% — the single best-performing cell in the entire table — while High Return data achieves only 98%. The paper's interpretation (Section 3.4) is that "the advantage of mixing data may stem from the benefit of having a larger state distribution with mixed rewards."

This is a diagnostic finding, not a method. It tells the field what kind of data matters for cross-experiment transfer, and the answer — diverse state coverage, even from suboptimal policies — is different from what the demonstration-centric literature would predict. The mechanistic explanation, which the paper gestures at but does not fully develop, likely involves the critic's need to learn accurate Q-values across broad regions of state space to avoid over-optimistic extrapolation when the online policy visits states outside the offline data's support. Low-return data provides dense coverage of the state space that the agent actually encounters during exploration, while expert data may cover only a narrow corridor of high-performing states. When the online policy explores and deviates from that corridor, the critic trained primarily on expert data has no information about those off-corridor states and may produce wildly inaccurate Q-estimates, destabilizing policy improvement. Low-return data, by covering diverse (suboptimal) states, provides the critic with negative examples that anchor Q-values in unexplored regions.

This insight connects RaE to the broader finding by Yarats et al. (2022) that in offline RL, "don't change the algorithm, change the data" — data diversity matters more than algorithmic sophistication. RaE extends this principle to the online-offline hybrid setting and provides the first systematic evidence (to the authors' knowledge) that the value of prior data for online RL bootstrapping is primarily about state-space coverage, not about providing examples of successful behavior. If this finding generalizes, it has immediate practical implications: practitioners should save and reuse all experimental data, not just final-policy trajectories, and should invest compute in broad exploration during early experiments (specifically to populate the offline store with diverse transitions) rather than in polishing expert policies.

Innovation 3: Amortizing Exploration Cost Across a Project Lifecycle — a Workflow Philosophy, Not a One-Shot Performance Trick

The paper's third conceptual contribution is the framing of data reuse as a continuous, project-wide practice rather than a one-time transfer from a fixed offline dataset to a single online run. Section 5 articulates this vision explicitly: RaE can be applied "throughout the lifetime of a project, across multiple experiments, algorithms, and hyperparameter settings." This reframes the problem from "how do I use this specific offline dataset to improve this specific online run?" (the framing of AWAC, CRR fine-tuning, and most prior work) to "how should I organize my entire RL workflow so that exploration cost is amortized across every experiment I ever run?"

The experimental evidence for this vision comes from three demonstrations in Section 3.4 and Section 5:

• Iterative RaE (Figure 5a) shows that performance continues to improve (modestly) through a second iteration of data reuse before plateauing on a third. This suggests a workflow where a single long training run is replaced by a sequence of shorter runs, each building on the data of all previous ones — effectively a minimalist form of lifelong learning with no specialized continual-learning machinery.

• Cross-algorithm reuse (Figure 5b) demonstrates that data collected with DMPO can improve training with D4PG — a fundamentally different policy optimization mechanism. This means data collected during early project phases (when the algorithmic approach may still be undetermined) retains value even if the final algorithm choice is different.

• Cross-seed aggregation (Figure 5c) shows that combining data from high-variance random seeds — including seeds that individually performed poorly — produces a robust run that matches or exceeds the best individual seed. This means that the common practice of running 5-10 seeds and reporting only the best one (or the mean) discards data that could be aggregated to produce a strictly better policy.

This is not an incremental contribution to algorithm design. It is a proposal for a cultural change in RL experimentation: stop deleting replay buffers. The cost of storing transitions is negligible relative to the cost of generating them (especially in simulation, but even in real-world robotics where data collection involves physical time and wear), and the marginal benefit of having that data available for future experiments is substantial. The paper articulates this explicitly (Section 5): "With costs of data storage typically being far lower than compute, a different workflow where all experimental data (particularly in domains like robotics) is stored and reused to bootstrap learning could improve efficiency across project lifetimes."

Innovation 4: Decoupling Data Reuse from Algorithm Design — a Separation of Concerns With Practical Implications

The paper's fourth contribution is a software-architecture insight disguised as an empirical finding: data reuse can and should be implemented as an infrastructure concern (a persistent data store + a sampling ratio) rather than as an algorithmic concern (specialized loss functions, training phases, or optimization procedures). This separation of concerns is what enables RaE to work across DMPO, D4PG, CRR, and SAC-X — algorithms that share essentially no common optimization machinery — without any per-algorithm adaptation.

This insight is more subtle than it appears because the history of off-policy RL has tightly coupled data management and algorithm design. Experience replay itself (Lin, 1992; Mnih et al., 2015) was an algorithmic innovation — it required proving that off-policy TD learning converges under replay sampling, and it introduced hyperparameters (buffer size, sampling strategy) that interact with the learning dynamics. Subsequent methods added further coupling: prioritized replay weights transitions by TD error (which depends on the critic architecture), Q-filters for offline data weight transitions by advantage estimates (which depend on the policy and critic), and density-ratio methods for mixing offline and online data estimate importance weights that depend on the policy's current state distribution. Each of these ties the data pipeline to the specific algorithm being used.

RaE demonstrates that this coupling is, at least for the domains and algorithms tested, unnecessary. Uniform sampling from the offline store with a fixed mixing ratio works because the diversity of the data (coming from all stages of prior training across multiple policies) provides sufficient coverage that sophisticated sampling strategies add little marginal value. The paper does not prove this theoretically — it is a purely empirical claim — but the breadth of the evidence (four algorithms, three domains spanning locomotion, vision-based manipulation, and standard control benchmarks) makes it credible.

The practical implication is significant for RL infrastructure design. An organization building RL systems can implement a persistent data store and a mixing sampler once, at the framework level, and then use it with any off-policy algorithm without modifying the algorithm's internals. This is fundamentally different from implementing AWAC or CQL or CRR, which require algorithm-specific code changes, loss functions, and hyperparameter management for each new algorithm variant. In a research or engineering context where multiple algorithms are being developed and compared, the infrastructure approach is dramatically simpler to maintain and less error-prone. The paper's emphasis on "minimal changes to the RL workflow," "no additional algorithmic complexities," and "straightforward integration into existing infrastructure" (Section 1, Section 2.2, Section 5) signals that this architectural insight is as central to the contribution as the empirical results themselves.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on three distinct domain clusters. Locomotion Soccer (Haarnoja et al., 2023): a simulated Robotis OP3 humanoid robot in a 4m × 4m walled arena, rewarded for scoring goals against a random opponent while staying upright and in its own half. Two variants exist — State (proprioception + task-specific features including ball, goal, and opponent coordinates) and Vision (proprioception + a 40 × 30 egocentric camera render, making the environment partially observable). Training data consists of 400,000 episodes for State and 200,000 episodes for Vision. Manipulation RGB Stacking (Lee et al., 2021): a Rethink Sawyer robot arm with a Robotiq 2F-85 parallel gripper must stack parametrically extruded color-coded objects, given three static camera images plus proprioception but no object position tracking. Rewards are sparse — the agent receives a positive signal only upon successful stacking with the arm moved away. The offline dataset is 150,000 episodes from a prior SAC-Q experiment. RL Unplugged (Gulcehre et al., 2020): the three hardest DeepMind Control Suite domains in the benchmark — Humanoid run (3,000 episodes, dense forward-velocity reward, data collected with D4PG), Manipulator insert peg (1,500 episodes, sparse insertion reward, data collected with V-MPO), and Manipulator insert ball (1,500 episodes, sparse insertion reward, data collected with V-MPO). The dataset is subsampled and successful episodes are reduced by 2/3, and transitions are stored as single-step trajectories (not full episodes). Each episode consists of 1,000 timesteps.

• Base model(s). The paper uses four off-policy algorithms, not a single model family. For Locomotion Soccer: DMPO (Abdolmaleki et al., 2018) with a distributional critic and 5-step returns. For Manipulation RGB Stacking: SAC-Q (Riedmiller et al., 2018), which uses DMPO-style updates internally with a multi-headed architecture for subtask scheduling. For RL Unplugged: CRR (Wang et al., 2020), which learns from offline data via advantage-weighted regression and is evaluated both purely offline and with RaE. For an ablation: D4PG (Barth-Maron et al., 2018), a deterministic policy gradient algorithm with distributional critic. Network architectures are domain-specific: Locomotion uses the configurations from Haarnoja et al. (2023); Manipulation uses convolutional residual networks (2, 2, 2 blocks with 16, 32, 32 channels) feeding multi-headed policy and value outputs; RL Unplugged uses MLPs of sizes (256, 256, 128) for policy and (512, 512, 256) for value function, with the policy parameterizing a Mixture of Gaussians with 5 components as in Wang et al. (2020). The paper deliberately does not standardize architecture across domains — this tests whether RaE works with whatever architecture a practitioner would naturally use for each domain.

• Metrics. The primary metric is undiscounted episode return (accumulated reward), plotted either as asymptotic performance after convergence (Figures 3, bar charts with 95% confidence intervals after smoothing over 1,000 episodes) or as learning curves showing mean return against total online steps (Figures 4, 5, 7, 8, with standard deviation shown as shaded regions). For Manipulation, the reported metric is the "stack-leave" reward term — a sparse reward given only when the red object is precisely placed on the blue one and the arm has moved away by a predefined distance (the hardest subtask). No per-episode success-rate metric is reported despite the Manipulation domain having binary success conditions. For the ablation in Table 1, raw returns are normalized as a percentage of the asymptotic return achieved when learning entirely from scratch (100% online data, 400,000 online episodes).

• Baselines. The paper compares against four categories. (1) Fine-tuning: Pure offline pre-training with BC, CRR binary, or CRR exp on prior data, selecting the best-performing seed and algorithm variant, then fine-tuning both policy and value function online with standard off-policy learning. Offline hyperparameters are swept per domain. (2) AWAC (Nair et al., 2020): Offline pre-training for 25,000 steps (with additional sweeps at 50,000 and 100,000 steps), then switching to online learning with a shared offline-online buffer using the AWAC policy update with Lagrange multiplier λ swept over 0.3 and 1.0. (3) Random Weight Resetting (inspired by Nikishin et al., 2022): Periodic reinitialization of all network weights and optimizer state during a single online run that continues accumulating data in its replay buffer (no offline data added). Reset frequencies are swept at K, K/10, and K/100 updates, where K is the number of updates used to train the original data-generating policy. For RL Unplugged (where K is unknown), values of 10,000, 100,000, and 1,000,000 are used. (4) Pure offline (CRR): The best-performing CRR variant trained entirely offline — this establishes the performance achievable without any online data.

• Generation budget / compute accounting. The paper measures compute as total online episodes or total online steps consumed during training. RaE's additional cost is exclusively storage (disk for the persistent offline store) plus the negligible overhead of sampling from two buffers rather than one — there is no additional forward-pass or gradient computation cost relative to a standard off-policy run. All methods compared at equivalent online data budgets: when RaE uses 400,000 online episodes, the from-scratch baseline also uses 400,000 online episodes. The offline data is treated as a sunk cost (already collected during prior experiments) and is not counted against the current experiment's budget. This accounting is consistent with the paper's workflow-level framing: the whole point is that prior experiments' data was already paid for and should not be wasted. For the iterative RaE experiment (Figure 5a), each iteration uses an additional 10,000 online episodes, and data from all previous iterations accumulates in the offline store.

• Cross-validation / statistical protocol. All main results (Figures 3 and 4) are averaged across 5 seeds per method, with standard deviation shown as shaded regions in learning curves and 95% confidence intervals shown as dark lines in bar charts after smoothing over 1,000 episodes. For Section 3.4 ablations, experiments use 2 seeds with larger batch sizes (256 for Locomotion) to reduce variance. For the fine-tuning baseline, the best-performing seed from the offline sweep is selected before online fine-tuning — this gives fine-tuning an advantage relative to RaE, which always starts from random weights with no seed selection. For the RL Unplugged domains, all methods are evaluated with 5 seeds per method variant, and the best hyperparameter setting for each baseline is reported per domain. The paper does not report confidence intervals on the normalized percentages in Table 1, nor does it report statistical significance tests between RaE and baselines in the bar charts of Figure 3 — visual overlap of confidence intervals is the only criterion for assessing whether differences are meaningful. This is a limitation: without formal hypothesis tests, it is difficult to determine whether, for example, RaE's margin over fine-tuning on the Manipulation Place task (Figure 3c) is statistically reliable or within sampling noise.

Main Quantitative Results

Search-Type Baselines vs. RaE on Locomotion and Manipulation (Figure 3)

Headline result. RaE with a fixed 50/50 data mix achieves the highest asymptotic return across all four Locomotion and Manipulation tasks, with the most dramatic margin on the challenging vision-based Locomotion Soccer Vision domain, where it substantially outperforms all baselines including AWAC and fine-tuning.

Locomotion Soccer State (Figure 3a). RaE reaches approximately the highest bar among all methods. Fine-tuning and AWAC perform competitively — their 95% confidence intervals overlap with RaE's — suggesting that on this relatively well-behaved dense-reward state-based task, multiple approaches converge to similar asymptotic performance. Random weight resetting notably underperforms all data-reuse methods, confirming that weight reinitialization alone does not drive RaE's benefit. The pure offline CRR baseline (dotted blue line) sits substantially below all online methods, as expected given that it has no access to online data.

Locomotion Soccer Vision (Figure 3b). This is the paper's strongest result. RaE achieves a mean return visibly and substantially higher than every baseline. Fine-tuning and AWAC both improve over learning from scratch, but their bars are clearly shorter than RaE's. The 95% confidence interval for RaE does not overlap with any other method's confidence interval (based on visual inspection of the figure — the paper does not report exact numerical values), indicating a statistically meaningful gap. This result is particularly significant because the vision variant is a genuinely hard exploration problem: egocentric visual observations make the environment partially observable, the reward for goal-scoring is sparse (1,000 points on a single timestep when the ball enters the goal, then unavailable until the ball bounces back), and the agent must learn to coordinate looking, moving, and kicking from pixels. The large relative improvement from RaE suggests that prior data provides the exploration signal that vision-based learning struggles to generate on its own — the offline data likely contains examples of the ball, goal, and successful scoring trajectories that the agent would otherwise take many episodes to discover.

Manipulation Place (Figure 3c) and Manipulation Stack Leave (Figure 3d). RaE and fine-tuning perform similarly — their bars and confidence intervals overlap almost completely. AWAC underperforms both on these tasks (its bar is visibly shorter with non-overlapping confidence intervals). The pure offline CRR baseline substantially underperforms all data-reuse methods, consistent with the difficulty of learning precise stacking from sparse rewards without online interaction. The near-parity between RaE and fine-tuning on these tasks is informative: it suggests that when offline pre-training works well (as it does here, presumably because the 150,000 episodes of prior SAC-Q data provide sufficient coverage for CRR to learn a reasonable initialization), the benefit of weight transfer (fine-tuning) is roughly equivalent to the benefit of persistent data mixing (RaE). However, RaE achieves this without requiring the practitioner to choose the right offline algorithm and hyperparameters — a practical advantage that does not show up in the final bar heights but is central to the paper's simplicity argument.

Interpretation across tasks. The pattern across Figure 3 reveals an interaction between domain difficulty and the relative advantage of RaE over baselines. On the easiest task (Locomotion Soccer State, dense rewards, low-dimensional state), all methods converge to similar performance — data reuse helps but is not transformative. On the hardest task (Locomotion Soccer Vision, sparse rewards, high-dimensional partial observations), RaE's advantage is largest — the offline data provides exploration coverage that the online agent struggles to generate. The Manipulation tasks sit in between: sparse rewards but with a learned curriculum (SAC-Q scheduler) that structures exploration, making fine-tuning and RaE roughly equivalent. This interaction — RaE's benefit scaling with exploration difficulty — is not explicitly discussed by the paper but is a natural implication of the state-coverage hypothesis from the ablation results.

RaE vs. Baselines on RL Unplugged (Figure 4)

Headline result. On the three hardest RL Unplugged domains, RaE consistently matches or exceeds all comparable methods, with the largest margins on the sparsely rewarded Manipulator tasks where offline-only methods (CRR, fine-tuning) fail to make meaningful progress online.

Humanoid run (Figure 4a). This is the only densely rewarded domain in RL Unplugged (the agent receives continuous reward for forward velocity). Fine-tuning and CRR pure offline perform well — fine-tuning's learning curve (green) rises quickly and converges near RaE's final performance. AWAC performs slightly worse. Random weight resetting converges more slowly but eventually approaches similar performance. RaE's advantage here is modest: its learning curve (orange, likely — the paper uses consistent coloring across panels but does not specify which line is which in the caption; based on the text, "RaE consistently performs the best across all tasks") lies slightly above fine-tuning at convergence. The fact that multiple methods work well on this dense-reward task is consistent with the Locomotion Soccer State result — when rewards are informative and easy to discover, the marginal benefit of offline data is smaller.

Manipulator insert peg (Figure 4b) and Manipulator insert ball (Figure 4c). These sparse-reward tasks tell a fundamentally different story. CRR pure offline (dotted blue line) achieves very low return and does not improve — the offline dataset alone is insufficient to learn insertion. Fine-tuning (green) starts poorly and fails to improve substantially — the policy pre-trained offline on sparse-reward data provides a poor initialization that online fine-tuning cannot recover from. AWAC performs similarly to fine-tuning, with slow initial learning and low final performance. Random weight resetting (grey) performs worst of all — without offline data to guide exploration, the agent never discovers the sparse insertion reward. RaE is the only method that makes substantial progress on both insertion tasks, with learning curves that rise clearly above all other methods. This is the paper's most direct evidence for the claim that persistent data mixing (rather than offline pre-training) is the key mechanism: fine-tuning and RaE see the same offline data, but fine-tuning processes it in a preliminary offline phase while RaE keeps it present throughout online training. The divergent results on these sparse domains imply that the perpetual availability of diverse offline transitions — providing the critic with negative examples from failed insertion attempts and broad state-action coverage — prevents the catastrophic collapse that fine-tuning experiences when the online policy explores regions where the pre-trained critic has no reliable Q-estimates.

Learning speed vs. asymptotic performance. The learning curves in Figure 4 reveal that RaE's advantage is in both learning speed and final performance. On Manipulator insert ball (Figure 4c), RaE's curve separates from all other methods early (within the first ~100,000 online steps) and continues improving while other methods plateau. This suggests that the offline data provides an exploration curriculum — the agent encounters rewarding transitions in the offline data that it would not discover on its own, and bootstraps from these to discover even better behavior online.

Iterative RaE (Figure 5a)

Headline result. Applying RaE iteratively yields improving performance through two iterations before plateauing on a third. Each iteration uses only 10,000 online episodes — substantially less than a full training run — yet the final performance exceeds what a single run with the same total online data would likely achieve.

Specifics. The experiment starts with a dataset of 10,000 episodes collected from a standard online run (the "original experiment" in Figure 5a, shown as the first data point on the learning curve). RaE iteration 1 loads this data and trains with 50/50 mix for an additional 10,000 online episodes, producing a policy that outperforms the original. The data from iteration 1 is then stored. RaE iteration 2 loads the original data plus iteration 1's data (now 20,000+ episodes) and trains for another 10,000 online episodes, achieving further improvement. RaE iteration 3 adds another 10,000 online episodes but shows no improvement — the curve plateaus.

Why this matters. This result validates the paper's vision of RaE as a project-long workflow rather than a one-shot performance boost. It demonstrates that breaking a single long run into a sequence of shorter runs, each building on the data of all previous ones, can be more effective than one marathon session. The practical implication (discussed in Section 5) is that practitioners can run shorter, less expensive experiments and chain them together, with each iteration benefiting from the cumulative exploration budget. The plateau at iteration 3 suggests a saturation point — beyond some amount of prior data, additional offline transitions provide diminishing returns, likely because the state-action space is already well-covered and further data is redundant. The paper does not investigate whether the plateau is due to data saturation or to limitations of the base algorithm itself.

Algorithm-Agnosticism: RaE with D4PG (Figure 5b)

Headline result. RaE improves performance with D4PG — a fundamentally different policy optimization mechanism from DMPO — demonstrating that the benefit of data mixing does not depend on any specific algorithmic inductive bias.

Specifics. On the Locomotion Soccer State task, the D4PG from-scratch baseline achieves some level of return (the exact number is not reported; the figure shows a learning curve rising from near-zero to an asymptotic value). RaE with D4PG starts higher (because it benefits from offline data from step zero) and converges to a substantially higher asymptotic return — the gap between the two curves is large and persistent throughout training. The shape of the improvement mirrors what was observed with DMPO in Figure 3a, confirming that the effect is not specific to MPO's EM-style constrained optimization.

Why this matters. This ablation directly addresses a potential criticism: that RaE's success with DMPO might be due to some interaction between fixed-ratio data mixing and DMPO's trust-region policy updates or its distributional critic. By showing equivalent gains with D4PG — which uses a deterministic policy gradient with no trust region and a different exploration mechanism (action noise rather than stochastic policy sampling) — the paper strengthens the claim that RaE is truly algorithm-agnostic. This is important for the practical adoption argument: a practitioner should be able to drop RaE into any off-policy codebase and see benefits without worrying about algorithmic compatibility.

Cross-Seed and Variance Robustness (Figure 5c)

Headline result. RaE can aggregate data across random seeds of a high-variance experiment to produce a robust run that outperforms any individual seed, even when the offline data includes poorly performing seeds.

Specifics. The experiment uses data from four individual seeds of a single Locomotion Soccer State experiment (shown as dotted lines in Figure 5c). The seeds exhibit substantial variance: one seed (Seed 2, shown as the top dotted line) achieves high return, while the others achieve progressively lower returns. When RaE loads all four seeds' data into the offline store and trains with 50/50 mix (the solid line, labeled something like "Load All Data"), the resulting policy matches or exceeds the best individual seed. When RaE loads only the worst-performing seed ("Load Seed 4" or similar, the specific labeling is not clear from the paper text), performance is lower but still substantially above that seed's individual performance — the new run benefits from exploring beyond the narrow data distribution of the poor seed.

What this demonstrates. The paper's narrative describes this as evidence for "increased robustness" (Section 3.4). More precisely, it demonstrates that RaE can serve as a variance-reduction mechanism at the project level: rather than running 10 seeds and reporting the best one (or the mean), a practitioner can run 10 seeds, aggregate all their data with RaE, and produce a single policy that is at least as good as the best seed and substantially more reliable. This is practically valuable because it extracts value from seeds that would otherwise be discarded as "failed" — a concrete example of the amortized-exploration philosophy.

Fine-Tuning with RaE (Figure 7, Appendix C.2)

Headline result. Combining RaE's data mixing with weight transfer from offline pre-training ("fine-tuning with RaE") yields faster initial learning while matching or exceeding the asymptotic performance of RaE alone. This confirms that the two mechanisms — data mixing and weight initialization — are complementary, not competing.

Specifics. On Locomotion Soccer State (Figure 7a), the fine-tuning baseline (green) learns fastest initially — it starts from a pre-trained policy that already achieves moderate return — but converges to a slightly lower asymptotic value than RaE alone (orange). Fine-tuning with RaE (the combined method, color not specified but presumably a third line) starts as fast as fine-tuning and matches or slightly exceeds RaE's asymptotic performance. On Locomotion Soccer Vision (Figure 7b), fine-tuning with RaE (likely the topmost curve) clearly dominates both fine-tuning alone and RaE alone, combining fast initial progress with the highest final return.

Why this is notable. The paper's main narrative emphasizes that RaE works without weight transfer, but this experiment shows that if weight transfer is available (i.e., if the prior experiment used the same network architecture, which RaE does not require), combining it with data mixing is strictly better than either alone. This suggests a practical best practice: if you can transfer weights from a prior experiment, do so, but also keep the prior data mixing — the two mechanisms address different aspects of the learning problem (good initialization vs. broad data coverage). The paper does not include this combined method in the main comparisons of Figure 3 — it is tucked into Appendix C.2 — which is an interesting editorial choice that perhaps reflects the desire to keep the main narrative focused on simplicity. In practice, a practitioner implementing RaE would likely use both given the option.

Ablation Studies and Robustness Checks

Data quantity sensitivity (Table 1, top vs. bottom halves): RaE is effective with as little as 10,000 prior episodes — a finding robust across all three data quality regimes. With 10,000 episodes of Mixed Return data at 90% online, performance reaches 124% of from-scratch, the second-highest value in the table. With 10,000 episodes of Low Return data at 90% online, performance reaches 108% of from-scratch — data that contains no successful trajectories nonetheless yields improvement. However, with only 10,000 episodes, the 50% online default can hurt performance (51% for High Return at 50% online, 90% for Mixed Return at 50% online), indicating that when offline data is scarce, the default ratio should be adjusted upward toward more online data. At 100,000 episodes, the 50% default is consistently beneficial or neutral — the paper's universal 50% recommendation holds when the offline dataset is of moderate or larger size.

Data quality robustness (Table 1, rows within each data regime): The most striking and counterintuitive finding is that Low Return data (early-training, pre-success transitions) is not worthless. With 100,000 episodes at 80% online, Low Return data achieves the single best result in the table (126% of from-scratch), outperforming Mixed Return (106%) and High Return (98%) at the same ratio. The paper's explanation — that the benefit of mixing data "may stem from the benefit of having a larger state distribution with mixed rewards" — is plausible but not experimentally verified: the paper does not measure state-space coverage or critic prediction error as a function of data regime, so the mechanism remains a hypothesis. The practical takeaway is clear: save all data, not just good data. Expert data (High Return) is consistently the least beneficial or roughly tied — with 100,000 episodes at 50% online, High Return achieves 119% vs. Low Return's 97%, but this is one of the few cells where High Return leads, and across the full table it is never the unambiguous best. This contradicts the prevailing assumption in the demonstration-based RL literature that expert trajectories are the most valuable data source.

Mixing ratio sensitivity (Table 1, columns within each data regime × dataset size combination): The optimal ratio depends on both data quantity and quality, but the 50% default is robust when offline data is plentiful. With 10,000 episodes, higher online ratios (80–90%) consistently outperform lower ratios (50–70%) — the worst-case penalty for using 50% with scarce data is severe (51% of from-scratch for High Return). With 100,000 episodes, the sensitivity flattens: 50% online achieves 112% for Mixed Return vs. 110% at 90% online — a negligible difference. The paper's Figure 8a (Appendix C.3) confirms this with the full 400,000-episode dataset: 50–70% online performs similarly, while 90% online slightly degrades performance relative to learning from scratch (drops below the 100% reference line). This supports the paper's claim that the 50/50 default "did not require any tuning" (Section 3.2) — but only when the offline dataset is large relative to the domain's complexity. The paper does not provide guidance on what "large enough" means quantitatively, which is a gap for practitioners wanting to deploy RaE with unknown dataset sizes.

Robustness to changing dynamics (Figure 8b, Appendix C.4): When RaE data from the standard Locomotion Soccer State task is reused in an environment where random masses are attached to the robot's legs (perturbing its dynamics), RaE continues to outperform learning from scratch. The learning curves (Figure 8b) show RaE starting higher and maintaining a gap throughout training. The paper interprets this as evidence that RaE is "surprisingly robust" to dynamics changes, speculating that the perturbed environment can be viewed as a partially observed MDP where "under some conditions unknown to the agent, the dynamics of walking alter" and "reusing data can still guide learning." This is a practically important finding because real-world deployments commonly involve wear-and-tear, miscalibration, or environmental shifts that change dynamics. However, the nature of the perturbation is mild — added leg masses change the robot's inertial properties but do not alter the fundamental structure of the task (still a biped walking in a soccer arena with the same goal). The paper does not test more radical dynamics changes (e.g., changing the robot morphology, switching from soccer to a different task, or transferring to a different simulation engine), so the claim of "robustness to changing dynamics" should be interpreted as robustness to modest within-task distribution shift rather than to arbitrary domain transfer.

Algorithm-agnosticism with additional algorithms (Figure 5b and the multi-domain main results): Beyond the D4PG ablation (Figure 5b), the paper implicitly demonstrates algorithm-agnosticism by using different algorithms in different domains without adapting the RaE mechanism: DMPO for Locomotion Soccer, SAC-Q for Manipulation, CRR for RL Unplugged, and D4PG as an additional check. None of these algorithms share optimization machinery. The RaE procedure — 50/50 data mix, uniform sampling, no auxiliary losses — is identical across all of them. This is less a formal ablation than a design choice: by spreading evaluation across four algorithms, the paper preempts the criticism that RaE's success is algorithm-specific. However, the paper does not evaluate RaE with on-policy algorithms (PPO, A3C, etc.) — the method is explicitly scoped to off-policy learning where experience replay is already fundamental, and extending it to on-policy methods would require a fundamentally different training protocol (likely involving importance sampling).

Effect of offline data composition (Table 1, rows): The experiment varies data quality (High, Mixed, Low Return) but does not independently vary data diversity. For example, the Mixed Return data is sampled uniformly throughout a training run, meaning it includes transitions from policies across the full performance spectrum. The paper does not ablate whether the benefit comes from the quantity of distinct policies in the data (policy diversity) or the quantity of distinct states visited (state-space coverage). These are confounded: more policies typically imply more state coverage. A targeted ablation that controls for state coverage while varying policy diversity — e.g., collecting data from a single converged policy with added exploration noise vs. data from multiple policies at different performance levels — would strengthen the mechanistic claim, but is beyond the paper's scope.

Oracle difficulty estimation NOT used: Unlike the compute-optimal test-time scaling paper in the reference example, RaE does not condition on estimated prompt difficulty — it treats all transitions in the offline dataset uniformly regardless of the policy quality that generated them. The Table 1 results justify this choice post-hoc: since low-return data can be as or more useful than high-return data, there is no need to estimate "data quality" and filter accordingly. However, the paper does not test whether some form of intelligent subsampling (e.g., removing redundant transitions from the offline store to reduce its size while maintaining coverage) could match or exceed the performance of keeping everything. This is a practical consideration for domains where the offline store grows to terabytes — the paper's "store everything" philosophy is simple but may be storage-inefficient at extreme scales.

Critical Assessment

Claim 1: "A minimal change to the RL workflow can greatly improve the asymptotic performance of off-policy reinforcement learning algorithms."

This claim is well-supported for the domains tested but with an important qualifier: the improvement is large on exploration-hard tasks (vision-based locomotion, sparse-reward manipulation) and modest-to-negligible on dense-reward tasks where standard algorithms already perform well. The evidence:

• On Locomotion Soccer Vision (Figure 3b), RaE clearly outperforms all baselines, with a gap that appears statistically meaningful (non-overlapping 95% confidence intervals). The paper does not report the absolute return values, but the visual margin is larger than on any other task.

• On Manipulator insert peg and insert ball (Figures 4b, 4c), RaE is the only method that makes substantial progress — a qualitative difference, not just a quantitative one. All baselines including fine-tuning and AWAC essentially flatline while RaE's learning curve rises.

• On Locomotion Soccer State (Figure 3a) and Humanoid run (Figure 4a), RaE's margin over fine-tuning and AWAC is small — confidence intervals overlap — and the practical significance of the improvement is unclear.

The claim's qualifier should be: "greatly improves performance on tasks where exploration is a bottleneck, and provides modest or negligible improvements where exploration is easy." The paper does not explicitly make this distinction, but the data support it.

What would strengthen this claim: Reporting exact numerical values with confidence intervals in the text rather than relying on visual interpretation of bar charts. Running a formal statistical test (e.g., bootstrap confidence intervals on the difference in means) between RaE and the best baseline per domain. Testing on additional exploration-hard domains beyond the three studied (Locomotion Soccer Vision, Manipulator insert peg, Manipulator insert ball) to establish that the exploration-difficulty interaction generalizes.

Claim 2: "RaE performs as well as or better than alternative approaches with fewer hyperparameter choices to be made."

This claim is strongly supported by the empirical comparisons against AWAC and fine-tuning. The paper makes a clear case:

• AWAC requires sweeping two hyperparameters — number of offline pre-training steps (25K, 50K, 100K) and Lagrange multiplier λ (0.3, 1.0) — a total of 6 combinations per domain. The paper reports the best-performing combination for each domain. RaE has zero hyperparameters to sweep (the 50/50 ratio is fixed). Despite this asymmetry in tuning effort, RaE matches or exceeds AWAC on all six tasks (Figures 3 and 4).

• Fine-tuning requires sweeping three offline algorithms (BC, CRR binary, CRR exp) and their hyperparameters, then selecting the best seed for online transfer. RaE requires none of this. On Manipulation tasks (Figures 3c, 3d), the two are roughly equivalent; on sparse RL Unplugged tasks (Figures 4b, 4c), RaE substantially outperforms fine-tuning.

The "fewer hyperparameter choices" part of the claim is indisputable — RaE literally has no tuned hyperparameters while the baselines have 2+ each. The "performs as well or better" part holds across all tested comparisons: RaE is never the worst method and is often the best.

Caveat: The paper does not account for the hyperparameter tuning effort that was already done for the base algorithms (DMPO, SAC-Q, CRR) — those algorithms have their own hyperparameters (learning rates, batch sizes, KL constraints, etc.) that were presumably tuned in prior work. RaE inherits whatever tuning was done for the base algorithm but does not add to it. This is a reasonable framing (RaE should be drop-in compatible with already-tuned algorithms), but it means the total hyperparameter count of an RaE-based system is non-zero; it is the incremental tuning cost that is zero.

What would strengthen this claim: Testing RaE with a deliberately mis-tuned base algorithm — e.g., DMPO with a poorly chosen learning rate or KL constraint — to see whether data mixing can partially compensate for algorithmic hyperparameter errors. If RaE is robust to base-algorithm hyperparameter choices (which the paper does not test), the practical value would be even greater.

Claim 3: "Low-return data can be more beneficial than expert data, and expert data is the least beneficial in both [low-data and moderate-data] regimes."

This claim, based on Table 1, is partially supported but overstated in the paper's text. The evidence:

• With 10,000 episodes at 80% online: Low Return = 101%, Mixed Return = 110%, High Return = 80%. Here, expert data is clearly worst, and low-return data outperforms it.

• With 10,000 episodes at 90% online: Low Return = 108%, Mixed Return = 124%, High Return = 120%. Here, expert data outperforms low-return data (120% > 108%), contrary to the "least beneficial in both regimes" claim.

• With 100,000 episodes at 50% online: Low Return = 97%, Mixed Return = 112%, High Return = 119%. Expert data is the best in this cell.

• With 100,000 episodes at 80% online: Low Return = 126%, Mixed Return = 106%, High Return = 98%. Expert data is worst, and low-return data is best — the paper's strongest supporting cell.

The claim that "expert data is the least beneficial in both regimes" is too strong — there are cells where expert data performs best. A more accurate summary would be: no single data quality regime dominates across all conditions; Mixed Return data is consistently strong; Low Return data can be surprisingly effective despite containing no successful trajectories; and the value of expert data is lower than the demonstration-based RL literature would predict. The practical recommendation to "save all data" remains well-supported — since the optimal data regime depends on dataset size and mixing ratio in ways that are difficult to predict ex ante, saving everything and letting the algorithm figure it out is a robust strategy.

What would strengthen this claim: Reporting confidence intervals or standard errors on the Table 1 percentages, so that readers can assess whether the apparent differences (e.g., 126% vs. 98%) are statistically distinguishable given the small number of seeds (2 seeds per cell). Without error bars, it is possible that the differences are within sampling noise, particularly for a domain as high-variance as RL.

Claim 4: "RaE can be applied iteratively... a performance plateau is reached on a third iteration."

Supported by Figure 5a but based on a single domain (Locomotion Soccer State) with a specific dataset size (10,000 episodes per iteration). The paper does not test iterative RaE on any other domain, nor does it vary the per-iteration data budget. The plateau at iteration 3 may be specific to this task and budget — a harder task with a larger per-iteration budget might benefit from more iterations, or the plateau might occur earlier. The paper appropriately hedges this claim with "in some settings it may be preferable" (Section 5), but the generalizability of the iterative result is unproven.

Claim 5: "RaE is applicable across research life cycles and can increase resilience by reloading data across random seeds or hyperparameter variations."

Partially supported. The cross-seed experiment (Figure 5c) provides evidence for resilience across seeds. The "hyperparameter variations" claim is not directly tested — the paper does not show RaE combining data from runs with different hyperparameters (e.g., different learning rates, batch sizes, or KL constraints). The cross-algorithm experiment (Figure 5b, DMPO → D4PG) is suggestive that data transfers across substantially different configurations, but this tests algorithm transfer, not hyperparameter transfer within a single algorithm. The "life cycles" claim is a forward-looking vision statement (Section 5) rather than a fully validated empirical finding — the experiments demonstrate that RaE works across two iterations and across different seeds, but a full project lifecycle involving dozens of experiments, evolving algorithms, and shifting task specifications is not tested.

Genuine Weaknesses in the Experimental Design

No formal statistical comparisons. The paper relies entirely on visual interpretation of overlapping confidence intervals and bar charts. For several comparisons — particularly RaE vs. fine-tuning on the Manipulation tasks (Figures 3c, 3d), where bars are nearly identical — it is impossible to determine from the figures alone whether RaE truly "outperforms" or merely "matches." Reporting exact numbers with confidence bounds on the difference would strengthen every quantitative claim.

Small seed counts for ablations. The main results use 5 seeds (reasonable for RL). But the critical Table 1 ablation — which supports the paper's most counterintuitive claims about data quality — uses only 2 seeds per cell. With only 2 seeds, individual outlier seeds could dramatically shift the reported percentages. A 126% value based on 2 seeds is substantially less reliable than one based on 5 or 10.

The RL Unplugged dataset properties are opaque. The paper acknowledges (Section 3.1) that the RL Unplugged data "is notably different from the other domains" — it was collected by sub-sampling and reducing successful episodes by 2/3, and stored as single-step transitions. This means the offline data distribution is artificially constrained (under-representing success) in ways that are not transparently reported in the Gulcehre et al. (2020) paper. RaE's strong performance on these domains (particularly the sparse insertion tasks, where fine-tuning fails) could be partly an artifact of this subsampling — if the offline data were a natural, unfiltered collection of prior training runs (as in the Locomotion and Manipulation domains), fine-tuning might perform better. The paper uses RL Unplugged to test "flexibility," but the mismatch in data collection protocols makes it difficult to draw clean conclusions about why RaE outperforms baselines on these specific tasks.

No comparison to the simplest possible data-mixing baseline: load offline data into the replay buffer once at the start of training and then never reload it. RaE maintains a fixed mixing ratio throughout training. An even simpler approach — pre-fill the online buffer with offline data at initialization and then treat it as a standard FIFO buffer that gradually forgets the offline data as new online data arrives — is not tested. This "pre-fill" baseline would be even simpler than RaE (no persistent offline store, no separate sampling) and would establish whether perpetual access to offline data (RaE's key design choice) is actually necessary or whether a one-time injection suffices.

The cross-seed experiment (Figure 5c) conflates data quantity with data diversity. When RaE loads all four seeds, it sees 4× more offline data than when loading a single seed. The observed improvement could be due to having more data (quantity) rather than having data from more diverse policies (diversity). A control experiment that loads one seed's data replicated 4× (same data, quadruple quantity) would disentangle these effects but is not run.

No experiment tests whether RaE can handle bad data that actively misleads learning — e.g., data from an environment with a different reward function that penalizes what the current task rewards. The changing-dynamics experiment (Figure 8b) tests a mild physical perturbation but does not test reward mismatch. If a prior experiment optimized a different reward function (e.g., maximize forward velocity vs. maximize energy efficiency in the same locomotion environment), would mixing that data help or hurt? This is relevant to the "multiple source experiments" use case the paper envisions (Section 5), where data from "many related source experiments" with "different reward functions" would be mixed. The paper does not test this scenario.

Missing evaluation of RaE's sensitivity to offline data staleness. If a prior experiment was run months ago with a different version of the simulation environment (different physics parameters, different observation space, different action space), does RaE still work? The changing-dynamics experiment (Figure 8b) is a step in this direction, but it tests only a modest mass perturbation on the same robot in the same task. Real project lifetimes (the paper's "project-long learning" vision) will involve substantially larger distribution shifts between experiments. The paper does not test robustness to observation-space changes (e.g., adding or removing sensors) or action-space changes (e.g., changing control frequency or joint limits), which are common in iterative robotics development.

Summary of Conditional Validity

• RaE improves performance most on exploration-hard, sparse-reward tasks. On dense-reward tasks, it provides marginal or negligible gains over simpler baselines. The paper would be strengthened by explicitly characterizing this interaction rather than presenting RaE as uniformly beneficial.

• The 50/50 default mixing ratio is safe when the offline dataset is of moderate size or larger (roughly 100K+ episodes for the Locomotion Soccer State task). With small offline datasets (10K episodes), it can hurt performance, and higher online ratios (80–90%) should be used. The paper does not provide a principled way to determine the threshold at which 50/50 becomes safe.

• RaE's iterative application yields diminishing returns and plateaus quickly on the one domain tested. Whether this generalizes — and whether the plateau is due to data saturation, algorithm capacity limits, or something else — is unknown.

• RaE's robustness to dynamics changes is demonstrated only for a mild perturbation on a single domain. The claim of "increased resilience" across research life cycles would require testing across larger and more diverse distribution shifts.

• The claim that low-return data can outperform expert data is intriguing but fragile — it depends on the specific dataset size and mixing ratio, and is based on only 2 seeds per condition. It is best treated as a suggestive finding that challenges prior assumptions rather than as a firmly established result.

6. Limitations and Trade-offs

6.1 The Method Depends on Having a Substantial Corpus of Prior Data, and the 50/50 Default Ratio Can Actively Degrade Performance When Offline Data Is Scarce

The assumption or constraint. RaE assumes the practitioner already possesses a meaningful quantity of interaction data from prior experiments on the same (or a sufficiently related) domain. The paper's headline results use offline datasets of 150,000–400,000 episodes for the Locomotion and Manipulation domains — representing substantial prior experimental investment. The method provides no mechanism for generating this data if it does not already exist; it is purely a data-reuse strategy, not a data-generation or data-augmentation strategy.

The consequence. When offline data is scarce, RaE's default 50/50 mixing ratio can actively harm performance relative to learning from scratch. Table 1 shows that with only 10,000 episodes of High Return data at 50% online, performance drops to 51% of from-scratch — the agent performs substantially worse than if the offline data had never been used. Even with Mixed Return data at 50% online, performance reaches only 90% of from-scratch. The failure mode is clear: the agent overfits to a small offline dataset, and the equal-proportion mixing drowns out the online exploration signal. The paper explicitly notes this (Section 3.4): "We hypothesize that using more online data for learning prevents over-fitting to a small set of offline trajectories."

This has direct practical implications. A researcher beginning work on a new domain — the exact scenario where bootstrapping would be most valuable — has no prior data to load. A researcher who has run only a handful of preliminary experiments may have data that is too sparse to benefit from the 50/50 default and could even be harmed by it. The paper does not provide a quantitative threshold for how much data is "enough" to safely use RaE with the default ratio — the practitioner must guess whether their dataset crosses the threshold from harmful to helpful.

What evidence exists in the paper. Table 1 (Section 3.4) is the key evidence. The contrast between the 10,000-episode and 100,000-episode regimes is stark: at 10,000 episodes, 50% online often underperforms from-scratch learning (values below 100%); at 100,000 episodes, 50% online is consistently beneficial or neutral. However, the experiment uses only a single domain (Locomotion Soccer State), so the specific threshold of "around 100,000 episodes" should not be treated as a general rule — it is domain-dependent and the paper provides no way to estimate it for a new domain.

Mitigation status. The paper partially addresses this by recommending higher online ratios when data is limited: the Table 1 results show that with only 10,000 episodes, 80–90% online ratios recover and even exceed from-scratch performance (e.g., Mixed Return at 90% online achieves 124%). The discussion text in Section 3.4 acknowledges the pattern: "In the lower data regime (1e4 episodes) a mixture with more online data is beneficial." However, the paper provides no automated mechanism for detecting the "lower data regime" or for adaptively adjusting the mixing ratio — the practitioner must manually inspect their dataset size, recognize that the default may be unsafe, and tune the ratio accordingly, which undercuts RaE's central claim of being tuning-free. Section 5 lists this implicitly under "Potential Limitations and Strategies for Mitigation" but does not propose a concrete solution beyond the natural suggestion that "changes in dynamics or experimental settings might invalidate previously collected data" — which it follows with a speculative suggestion about collecting transitional data rather than addressing the scarcity problem directly.

---

6.2 RaE's Computational Cost Is Reported as Negligible, but the Storage Overhead and Infrastructure Burden Are Not Accounted For

The assumption or constraint. The paper frames RaE as a minimal workflow change that requires "only... the availability of a second replay mechanism" (Section 2.2) and asserts (Section 5) that "costs of data storage [are] typically being far lower than compute." This treats storage as free and abstracts away the engineering effort of building and maintaining a persistent cross-experiment data pipeline.

The consequence. The storage cost is not zero, and in some regimes it may be substantial. The Locomotion Soccer State domain collects 400,000 episodes. If each episode averages, say, 500 timesteps (a conservative estimate for a goal-scoring task where episodes may terminate early on failure), and each transition (s, a, r, s') is stored as raw float32 arrays with typical dimensionality, the offline store could reach hundreds of gigabytes for a single domain. Across multiple domains, multiple algorithm variants, and multiple seeds — the "project-long learning" vision the paper advocates — the storage requirement could reach terabytes. For academic labs or small teams without institutional cloud storage, this is a non-trivial cost.

More importantly, the paper does not account for the engineering overhead of building the persistent data pipeline. Standard off-policy RL codebases use in-memory replay buffers (Python lists or ring buffers) that are discarded at experiment termination. Implementing a persistent store requires: serialization and deserialization of transitions (potentially across different versions of environment or observation formats), a storage backend (disk, database, or distributed filesystem), a sampling mechanism that can efficiently draw uniform samples from a dataset that may be orders of magnitude larger than memory, and version management to handle evolving observation/action spaces across experiments. The paper acknowledges that "changes in dynamics or experimental settings might invalidate previously collected data" (Section 5) but does not discuss the software engineering challenge of gracefully handling such invalidation — does the practitioner manually curate which datasets to include, or does RaE blindly load everything and hope for the best?

What evidence exists in the paper. None. The paper does not report storage sizes for any of its offline datasets, does not measure the wall-clock time or I/O overhead of sampling from the persistent store vs. the in-memory online buffer, and does not discuss the infrastructure implementation beyond the conceptual description in Section 2.2. The "Potential Limitations" subsection (Section 5) mentions the possibility that "changes in dynamics" might invalidate data but frames this as a domain limitation rather than an infrastructure one.

Mitigation status. Not addressed. The paper treats RaE as self-evidently simple to implement, but the gap between "add a second replay buffer" (the conceptual description) and "build a reliable, versioned, cross-experiment persistent data store that integrates with distributed RL infrastructure" is substantial. A practitioner reading the paper would need to build this infrastructure from scratch — no reference implementation or design guidance is provided. This is not a fatal flaw (the contribution is the idea and its empirical validation, not a software release), but it means the "minimal change" framing understates the practical engineering cost, particularly for teams without existing data infrastructure.

---

6.3 The Method Is Only Validated for Off-Policy Algorithms; There Is No Path to Applying RaE to On-Policy Methods, Which Remain Dominant in Many Application Areas

The assumption or constraint. RaE is explicitly and exclusively designed for off-policy RL algorithms. The abstract states that "replaying data is a principal mechanism underlying the stability and data efficiency of off-policy reinforcement learning," and the entire method builds on the off-policy property that allows learning from data collected by any policy. On-policy algorithms (PPO, TRPO, A3C, and their variants), which require data to come from the current policy and cannot naïvely ingest data from prior policies, are outside RaE's scope.

The consequence. This is a significant scope limitation because on-policy methods remain widely used — particularly in domains where off-policy methods are known to be unstable (e.g., high-dimensional continuous control with complex contact dynamics) or where implementation simplicity is prioritized (PPO is often the default choice in robotics and game-playing research due to its robustness and ease of tuning). The paper's claim that RaE is "applicable across research life cycles" (Section 5) is implicitly restricted to projects that exclusively use off-policy algorithms, which excludes a large fraction of the RL practitioner community.

Adapting RaE to on-policy methods would require fundamentally different machinery — most likely importance sampling (IS) corrections to reweight offline transitions to be on-policy. IS introduces its own challenges: variance grows exponentially with the divergence between the offline policy and the current policy, and effective IS typically requires storing action probabilities alongside transitions (which the paper's protocol does not do, since off-policy methods do not need them). The paper does not discuss this extension or provide any indication that it would work.

What evidence exists in the paper. The paper is transparent about the off-policy scope: every algorithm evaluated (DMPO, D4PG, CRR, SAC-X) is fundamentally off-policy in its data consumption mechanism. Section 2's background explicitly frames RaE in terms of "off-policy algorithms" and "experience replay." However, the paper never explicitly states that on-policy methods are unsupported — this limitation is implicit in the framing rather than acknowledged as a restriction. The Discussion section's vision of "project-long learning" does not qualify that this vision only applies if the project exclusively uses off-policy algorithms, which is a notable omission.

Mitigation status. Not addressed. The paper makes no attempt to extend RaE to on-policy methods and does not discuss the possibility or challenges of doing so. This is a legitimate scope limitation rather than a failure — not every method needs to solve every problem — but the paper's forward-looking claims about "lifelong learning" and "applicability across research life cycles" would benefit from explicitly scoping them to off-policy workflows.

---

6.4 The Single-Domain, Fixed-Task Evaluation Does Not Test the Paper's Broader Vision of Transfer Across Tasks, Reward Functions, or Substantially Different Environments

The assumption or constraint. All of RaE's main experiments reuse data from prior runs on exactly the same task — same environment, same reward function, same observation and action spaces, same dynamics. The only exception is the changing-dynamics experiment (Figure 8b, Appendix C.4), which perturbs the Locomotion Soccer State domain by adding random masses to the robot's legs — a modest within-task distribution shift. The paper does not evaluate RaE in settings where the prior data comes from a different task (e.g., locomotion data reused for a navigation task), a different reward function (e.g., data from a run maximizing forward velocity reused for a run maximizing energy efficiency), or a substantially different environment (e.g., data from one robot morphology reused for a different morphology).

The consequence. The paper's Discussion (Section 5) explicitly envisions RaE being applied in these more ambitious transfer settings. Under "Multiple Source Experiments," it states:

"Consider many tasks defined via different reward functions but with the same underlying dynamics (e.g. the family of all manipulation tasks with a specific morphology). High-return data on some tasks may result in lower returns in another. However this data is still informative and may be useful in improving exploration when transferred."

This is a hypothesis, not a validated claim. The paper provides zero evidence that RaE works when the offline data comes from tasks with different reward functions. In fact, there is reason to be skeptical: if a prior experiment optimized a reward function that conflicts with the current task's reward function (e.g., prior data rewards the agent for moving quickly while the current task rewards energy efficiency), mixing that data could actively mislead the critic. The agent would observe high-Q actions in the offline data that are actually low-Q under the current reward function, potentially destabilizing Q-learning. The changing-dynamics experiment (Figure 8b) is encouraging — it shows RaE is robust to modest physics perturbations — but it does not test reward mismatch, which is the core challenge of the multi-task transfer scenario the paper envisions.

Similarly, the paper does not test whether RaE works when the observation or action spaces change across experiments — a common occurrence in iterative robotics development where sensors are added, removed, or repositioned, or where control frequencies or joint limits are adjusted. The paper's protocol of storing raw (s, a, r, s') tuples assumes these spaces are stable across experiments. If they change, the practitioner must decide whether to pad/mask incompatible dimensions, train separate encoders, or discard the stale data — none of these strategies are discussed or evaluated.

What evidence exists in the paper. Only the changing-dynamics experiment (Figure 8b). The paper interprets this result optimistically: "This somewhat surprising finding may be explained by modeling the environment as a partially observed MDP where under some conditions unknown to the agent, the dynamics of walking alter. Reusing data can still guide learning in such a setting showing the robustness of our approach." However, the perturbation is mild (added leg masses change inertial properties but do not alter the fundamental task structure), and the result demonstrates robustness to a dynamics shift, not to a task or reward shift. The paper acknowledges the broader limitation obliquely in Section 5: "changes in dynamics or experimental settings might invalidate previously collected data" and suggests "an intermediate step to collect transitional data between old and new settings might be useful" — but this is speculation with no experimental backing.

Mitigation status. Not addressed empirically. The paper flags the issue (Section 5) but provides no experimental validation of RaE across tasks, reward functions, or observation/action space changes. The "Multiple Source Experiments" use case remains a forward-looking vision unsupported by data.

---

6.5 The Paper Does Not Test Whether RaE Can Degrade Performance When the Offline Data Contains Actively Misleading or Contradictory Information Relative to the Current Task

The assumption or constraint. RaE treats all prior data as beneficial by default, with uniform sampling and no filtering mechanism. The only discrimination applied is the implicit Q-filtering that occurs during learning (since off-policy algorithms will, in principle, learn to distinguish good from bad actions through TD learning). The paper does not test scenarios where some of the prior data is actively counterproductive for the current task — e.g., data collected under a different reward function that penalizes what the current task rewards, data from an earlier version of the environment with different physics that would teach the critic incorrect transition dynamics, or data from an adversarial or corrupted source.

The consequence. This is not a hypothetical concern — it directly follows from the "Multiple Source Experiments" use case the paper promotes. If a project has accumulated data from many related manipulation tasks with different reward functions (some rewarding speed, some rewarding precision, some rewarding energy efficiency), and a new experiment optimizes one specific reward function, the offline data will contain transitions that are high-reward under other reward functions but potentially low-reward under the current one. During training, the critic will observe these transitions — the state, action, and next-state are all valid (the dynamics are shared), but the stored reward r is wrong for the current task. The distributional TD update will incorporate these misleading rewards, potentially pulling the Q-function toward incorrect values.

The paper's finding that Low Return data can outperform High Return data (Table 1) provides indirect evidence that RaE is robust to low-quality data, but low-quality data (suboptimal trajectories that receive low reward under the current task's reward function) is different from misleading data (trajectories that receive high reward under a different reward function but low reward under the current one). Low-quality data still provides correct reward labels — they are just small rewards. Misleading data provides incorrect reward labels, and the TD learning machinery has no way to distinguish a correctly labeled high-reward transition from an incorrectly labeled one (since the reward is treated as ground truth from the environment).

What evidence exists in the paper. None. The paper does not evaluate RaE with reward-mismatched offline data, adversarial data, or corrupted data. The changing-dynamics experiment (Figure 8b) is the closest analogue, but it tests dynamics mismatch (different transition probabilities), not reward mismatch (different reward functions). The cross-seed experiment (Figure 5c) combines data from seeds with different performance levels, but all seeds share the same reward function — they differ in the quality of the trajectories they produce, not in the reward labels attached to those trajectories.

Mitigation status. Not addressed. The paper does not discuss this failure mode, does not propose any filtering or re-weighting mechanism to handle reward mismatch, and does not acknowledge it as a potential limitation of the "store everything" philosophy when applied across tasks with different objectives. This is a notable gap given how prominently the multi-task transfer vision features in the Discussion.

---

6.6 All Experiments Use a Single Simulation Framework (MuJoCo/DM Control Suite) With a Narrow Range of Domains; Generalization to Real-World Robotics, Discrete Action Spaces, or Fundamentally Different RL Problems Is Untested

The assumption or constraint. Every experiment in the paper uses continuous control tasks implemented in the MuJoCo physics simulator (Todorov et al., 2012) with the DeepMind Control Suite framework (Tassa et al., 2018, 2020). The domains span locomotion and manipulation, but all share common properties: continuous state and action spaces, dense or sparse scalar reward functions, deterministic or near-deterministic transition dynamics, and simulated physics with well-behaved contact dynamics. The paper does not evaluate RaE on: discrete action spaces (e.g., Atari games, combinatorial optimization), real-world robotic systems with sensor noise and actuation delays, multi-agent settings, partially observable domains beyond the single vision-based locomotion task, or domains where the reward is learned, human-provided, or non-stationary.

The consequence. The paper's claims of generality — "works across multiple algorithms," "applicable across research life cycles," "a simple and practical method that can be generally applied" (Section 5) — are empirically supported only for the narrow class of continuous-control MuJoCo domains tested. Several properties of these domains may make them particularly favorable for RaE:

• Deterministic or low-variance dynamics mean that a transition observed in prior data remains valid (the same (s, a) will produce roughly the same s') even after the environment has been reset or the simulation version has changed. In stochastic environments or real-world systems with time-varying dynamics (wear and tear, lighting changes, sensor drift), old transitions become stale and may mislead the critic.

• Well-shaped state spaces (joint angles, velocities, object positions) mean that uniform sampling from a diverse offline dataset provides meaningful coverage. In pixel-based domains (beyond the single egocentric-vision locomotion task), the state space is vastly larger and less structured — random frames from prior training may provide negligible useful signal because the probability of sampling a relevant frame is vanishingly small.

• Stationary environments mean that the MDP definition is fixed across experiments. In real-world deployments, the environment often changes between experiments (different lab lighting, different robot calibration, different object positions), and the assumption that p(s_{t+1} | s_t, a_t) is identical across all prior experiments becomes violated.

The paper's one experiment with partial observability (Locomotion Soccer Vision, Figure 3b) shows RaE working well, which is encouraging, but it is a single domain and does not test the limits of how severe partial observability can be before offline data becomes more confusing than helpful.

What evidence exists in the paper. The paper limits its evaluation to the three domain clusters described in Section 3.1 and Appendix A. There is no experiment outside the MuJoCo/DM Control ecosystem, no discrete-action task, and no real-world validation. The paper acknowledges the limitation implicitly by scoping its claims to "off-policy RL" and the tested domains, but it does not explicitly discuss the continuous-control bias in its evaluation or the potential failure modes in other settings.

Mitigation status. Not addressed beyond the natural caveat that any empirical paper's conclusions are bounded by its experimental scope. The Discussion section's forward-looking claims ("a different workflow where all experimental data is stored and reused to bootstrap learning could improve efficiency across project lifetimes") do not qualify that this vision has only been tested on a specific class of simulated continuous-control problems. This is a standard limitation of empirical RL research — few papers test across fundamentally different domain types — but it is particularly relevant here because RaE's value proposition depends on the generality of the "store everything, mix at 50/50" recipe. If that recipe fails for discrete actions or real-world systems, the claim of broad applicability weakens substantially.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper causes a methodological reframing rather than a paradigm shift. It does not introduce a new algorithm, architecture, or theoretical result. Instead, it establishes that the simplest possible approach to cross-experiment data reuse — a fixed 50/50 mix of prior and online data with no algorithmic modifications — matches or exceeds a decade's worth of increasingly complex methods that added prioritized replay, Q-filters, behavior-cloning losses, ensemble distillation, multi-stage training, and learned density ratios. The magnitude of the contribution is in shifting the burden of proof: any future method claiming to improve data reuse in off-policy RL must now demonstrate that its added complexity beats a naïve uniform mixture, not just that it beats learning from scratch.

This is best understood as an experimental methodology contribution analogous to the recognition that random search often matches sophisticated Bayesian optimization for hyperparameter tuning, or that carefully tuned baselines can match state-of-the-art methods in meta-learning. RaE is the "null hypothesis" for data reuse. The paper's literature review (Section 4) reveals a field that jumped directly from pure offline RL (CRR, CQL) to algorithmically complex online-offline hybrids without first establishing how far uniform data mixing alone could go. By filling this gap, the paper provides the community with a simple, essentially free baseline that should be included in every future comparison — and in doing so, it may redirect research effort away from incremental algorithmic complexity and toward understanding why simple data mixing works and when it stops working.

The paper also reconciles a tension in the literature between the demonstration-based RL community (which emphasizes the value of expert trajectories and designs mechanisms to focus learning on them) and the offline RL community (which has increasingly recognized that data diversity matters more than data quality — Yarats et al., 2022; Lambert et al., 2022). RaE's finding that low-return data from early in training can outperform expert data (Table 1: Low Return at 100K episodes and 80% online achieves 126% of from-scratch performance, the single best cell in the table, while High Return at the same settings achieves only 98%) provides direct evidence for the diversity-over-quality view and challenges the foundational assumption of demonstration-based methods. This is a diagnostic finding, not a method — it tells the field what kind of data matters for bootstrapping, and the answer (broad state-space coverage from suboptimal policies) is different from what the expert-trajectory literature would predict. Future work on data collection for RL should prioritize diverse exploration over expert performance when the goal is to generate reusable data for downstream experiments.

The paper redirects research attention in two specific ways. First, it makes improving the data pipeline (storage, sampling, version management, cross-experiment compatibility) a first-class research concern rather than an afterthought. If a 50/50 uniform mix works this well, further gains may come from smarter data management (e.g., automatic detection of stale or harmful data, adaptive mixing ratios) rather than from smarter policy updates. Second, it makes the exploration difficulty of a domain a key variable for predicting when data reuse will help. The paper's results show that RaE's benefit is largest on exploration-hard tasks (Locomotion Soccer Vision, sparse Manipulator insertion) and modest-to-negligible on dense-reward tasks (Locomotion Soccer State, Humanoid run). This interaction — RaE's value scaling with exploration difficulty — suggests that test-time compute allocation, curriculum design, and exploration bonus mechanisms should all be re-evaluated with the understanding that off-policy data mixing already provides substantial exploration signal for free on hard problems. Research directions that become less attractive: developing ever-more-complex auxiliary losses for offline-to-online transfer (since uniform mixing already achieves most of the gain), and methods that require per-domain hyperparameter tuning for data reuse (since the untuned 50/50 default is competitive).

Follow-Up Research This Work Enables

Automated detection of the safe mixing ratio from offline dataset statistics. Table 1 shows that the optimal online ratio depends on both dataset size and data quality: with 10,000 episodes, 80–90% online is necessary to avoid performance degradation (50% online drops to 51% of from-scratch for High Return data), while with 100,000 episodes, 50% works well universally. The paper provides no mechanism for a practitioner to determine, without running expensive sweeps, whether their specific offline dataset is "large enough" for the 50/50 default to be safe. A concrete follow-up would train a lightweight classifier — using features like total transition count, state-action coverage metrics (e.g., mean nearest-neighbor distance in the offline dataset), return distribution statistics, and an estimate of the environment's intrinsic exploration difficulty — to predict the optimal mixing ratio. The training data would come from running RaE at multiple ratios (50/70/80/90/100% online) across dozens of domains and recording which ratio maximized asymptotic performance. If successful, this would close the paper's most significant practical gap: the need to guess whether the default ratio is safe. A strong result would demonstrate that the predicted ratio achieves ≥95% of the oracle-optimal ratio's performance on held-out domains.

Testing RaE with deliberately mis-matched reward functions across experiments. The paper's Discussion (Section 5) envisions RaE applied across "many tasks defined via different reward functions but with the same underlying dynamics," but provides zero experimental evidence for this scenario. A critical stress-test would evaluate RaE in a multi-task manipulation suite (e.g., Meta-World or RLBench) where prior experiments optimized different reward functions. The experimental design: for each target task, construct the offline dataset from experiments on other tasks that share dynamics but have different (and potentially conflicting) reward functions — e.g., a prior experiment rewarded the robot for pushing an object to the left, while the current task rewards pushing it to the right. Measure whether RaE's performance exceeds from-scratch learning, degrades relative to from-scratch (because the critic learns from misleading reward labels), or requires a mechanism to re-label or filter offline transitions. A negative result (RaE degrades) would establish a crucial boundary condition: the "store everything" philosophy is safe only when reward functions are consistent across experiments, and multi-task transfer requires reward-aware filtering. A positive result (RaE still helps) would validate the paper's most ambitious vision and would suggest that the critic can learn to ignore reward labels from the offline data when they conflict with online experience — a finding with implications for continual and lifelong RL.

Combining RaE with targeted exploration bonuses for the online phase. The paper's results show that RaE's benefit is largest on exploration-hard tasks (Locomotion Soccer Vision, Figure 3b; Manipulator insertion, Figure 4b-c), where the offline data provides exploration coverage the agent would otherwise struggle to generate. This suggests a complementary mechanism: if the offline data covers broad regions of state space, the online agent can use an exploration bonus (e.g., Random Network Distillation, disagreement-based intrinsic motivation, or count-based exploration) that specifically rewards visiting states not represented in the offline store. This would focus online exploration on genuinely novel regions rather than rediscovering states already covered by prior experiments, potentially yielding faster learning than either RaE or exploration bonuses alone. A concrete experiment would compare RaE + exploration bonus vs. RaE alone vs. exploration bonus alone on the hardest domains from the paper (Locomotion Soccer Vision, Manipulator insertion) and on procedurally generated environments where the offline data's state coverage is deliberately incomplete. The prediction: the combination should outperform both individually, with the gap largest when the offline data covers most-but-not-all of the relevant state space.

Evaluating RaE with on-policy algorithms via importance sampling. The paper explicitly scopes RaE to off-policy methods. Extending the core idea — persistent data reuse across experiments — to on-policy algorithms (PPO, TRPO, A3C) would require importance sampling (IS) corrections to reweight offline transitions to be on-policy, since on-policy methods assume data comes from the current policy. A concrete follow-up would measure (1) the variance of IS weights as a function of how far the current policy has diverged from the policies that generated the offline data, (2) whether clipping or capping IS weights (as in PPO's objective) is sufficient to make offline data useful without destabilizing training, and (3) at what point in training offline data becomes counterproductive (when IS weights have unacceptably high variance) and should be phased out. A strong experimental design would use a domain where both off-policy and on-policy methods are commonly applied (e.g., continuous control from pixels with PPO vs. SAC) and compare RaE-on-policy (with IS) against RaE-off-policy (without IS) and from-scratch on-policy learning. This would establish whether the benefit of data reuse extends to the on-policy paradigm or whether it is fundamentally tied to the off-policy property of learning from arbitrary data distributions.

Characterizing the failure modes when offline data becomes actively harmful. The paper demonstrates that RaE is robust to low-quality data (Table 1: Low Return data at 10K episodes and 90% online achieves 108% of from-scratch) and to mild dynamics changes (Figure 8b: added leg masses). It does not test scenarios where offline data is actively misleading — e.g., transitions from an environment with different reward labeling, corrupted sensor readings, or adversarial perturbations designed to degrade learning. A systematic stress-test would construct offline datasets with controlled proportions of misleading transitions (e.g., X% of transitions have their reward sign flipped, or their next-state corrupted by Gaussian noise of increasing variance) and measure the fraction at which RaE's performance drops below from-scratch learning. This would establish a "safe operating envelope" for the "store everything" philosophy and would inform practical guidelines for when practitioners should audit or filter their offline data before mixing. A particularly informative variant would test whether the Q-function's TD errors on offline transitions can serve as an automatic detector of misleading data — if a transition's TD error is consistently and significantly larger than the mean, it may be an outlier that should be removed or downweighted.

Iterative RaE with growing offline datasets and explicit policy improvement targets. The paper's iterative experiment (Figure 5a) shows that RaE plateaus after two iterations on Locomotion Soccer State with 10,000 online episodes per iteration. This could be because the offline data saturates (no new states are being discovered) or because the base algorithm (DMPO) reaches its capacity limit. A diagnostic follow-up would run iterative RaE with (1) varying per-iteration data budgets to test whether larger budgets enable more iterations before plateau, (2) progressively harder task variants (e.g., increasing goal distance or reducing reward density) at each iteration to test whether the offline data from easier variants transfers to harder ones, and (3) explicit measurement of state-space coverage metrics (e.g., average distance from each online-visited state to its nearest neighbor in the cumulative offline dataset) to determine whether the plateau coincides with coverage saturation. This would transform the "iterative RaE" concept from a suggestive single-domain result into a principled framework for deciding how many iterations to run and how to allocate data collection budgets across them.

Practical Applications and Downstream Use Cases

Robotics laboratories running iterative hardware experiments. In real-world robotics, data collection is physically expensive — each episode requires robot time, human supervision, and wear on hardware. A typical development cycle involves running dozens of experiments to tune reward functions, adjust controller parameters, or test algorithm variants, with each experiment collecting hours of interaction data that is typically discarded. RaE provides a concrete recipe: store all transitions from every experiment, and pre-load them into the replay buffer for every subsequent experiment at a 50/50 mix ratio. The paper's results on the Manipulation RGB Stacking domain (Figure 3c-d, using 150,000 episodes of prior SAC-Q data) directly demonstrate that this works for vision-based sparse-reward robotic manipulation — precisely the regime where real-world robotics operates. The practical benefit is two-fold: improved asymptotic performance (RaE matched or exceeded fine-tuning and AWAC on stacking tasks) and robustness to algorithmic choices (RaE works with DMPO, SAC-Q, D4PG, or CRR, so the lab can switch algorithms without losing the value of accumulated data). A robotics lab adopting RaE would need to implement a persistent data store with versioning (to handle sensor additions or changes in observation space across hardware iterations) and would need to establish that 50/50 is a safe default for their specific data volumes — a concern partially mitigated by the paper's finding that higher online ratios (80–90%) are safe even with small datasets (Table 1).

Large-scale RL training pipelines that generate multiple seeds and hyperparameter sweeps. The paper's cross-seed experiment (Figure 5c) demonstrates that RaE can aggregate data from a high-variance experiment across random seeds to produce a policy that matches or exceeds the best individual seed. In a production training pipeline — e.g., training a locomotion controller for a quadruped robot, or a dexterous manipulation policy for a factory automation task — it is standard practice to run 10–50 seeds and select the best one. The remaining 9–49 seeds' data is discarded. RaE provides a mechanism to extract value from that discarded data: run an additional RaE experiment that loads all seeds' data and trains with the 50/50 mix. The paper's Figure 5c shows this matches the best individual seed's performance on Locomotion Soccer State. If this generalizes, the marginal cost of an RaE aggregation run (one additional training run's compute) is negligible relative to the cost of the initial sweep (10–50 training runs' compute), making it essentially free performance. The practical implementation is trivial: point the offline store at the sweep's output directory and launch one more training job.

Sim-to-real transfer where simulation data is abundant but real-world data is scarce. A common workflow in robotics involves training initially in simulation (where data is cheap and unlimited), then fine-tuning on the real robot (where data is expensive). RaE's finding that even low-quality simulation data — including early-random-exploration transitions with zero task reward — is beneficial for bootstrapping (Table 1: Low Return at 10K episodes and 90% online achieves 108% of from-scratch) directly supports this workflow. A practitioner would: (1) run extensive simulation experiments with broad exploration (diverse initial conditions, randomized dynamics, varied reward scales), storing all data; (2) deploy the real robot and run RaE with the simulation data as the offline store at a ratio tuned to the real-world data budget (higher online ratio if real-world data is extremely scarce — the paper's Table 1 suggests 80–90% online for small datasets); (3) benefit from the simulation data's state-space coverage even though the reward labels and transition dynamics may differ from reality. The changing-dynamics experiment (Figure 8b) provides preliminary evidence that RaE is robust to simulation-to-reality dynamics gaps (leg mass perturbations serve as a proxy for unmodeled dynamics), though this would need validation on a physical system.

Benchmark development and fair comparison protocols in the RL community. The paper's finding that RaE's performance matches or exceeds AWAC and fine-tuning — methods that require per-domain hyperparameter tuning — has implications for how the field evaluates new data-reuse algorithms. A concrete application: RL benchmark suites (RL Unplugged, D4RL, Meta-World) should include an RaE baseline as a standard comparison point. The baseline is trivial to implement: a 50/50 mix of the provided offline dataset and online data, sampled uniformly, with no algorithmic modifications to the base off-policy agent. Any new method claiming to improve data reuse efficiency should be required to demonstrate that it beats this baseline — not just that it beats learning from scratch. The paper's results on RL Unplugged (Figure 4) already demonstrate that RaE is a competitive baseline on the hardest domains in that benchmark, suggesting its immediate adoption is feasible without additional data collection. This would raise the bar for methodological contributions in offline-to-online RL and would help the community distinguish genuine algorithmic advances from effects that can be achieved with simpler infrastructure changes.

When to Prefer This Method

The paper positions RaE not against a single named alternative but against the broader class of data-reuse methods that introduce algorithmic complexity (AWAC, fine-tuning with CRR/CQL, demonstration-based methods with prioritized replay and auxiliary losses). The decision rule that emerges from the experimental results is:

• Prefer RaE (uniform 50/50 mix, no algorithmic changes) when:

  • You have access to prior experimental data from the same or a related domain (at least ~100K episodes for Locomotion-scale tasks; the safe threshold is domain-dependent and not precisely characterized in the paper).
  • You want a drop-in solution that works without per-domain hyperparameter tuning — RaE requires zero additional tuning beyond what your base off-policy algorithm already needs.
  • You are working with an off-policy algorithm (DMPO, D4PG, SAC, CRR, or similar) and do not want to modify its internal optimization procedure.
  • Your domain involves exploration challenges (sparse rewards, high-dimensional observations, partial observability) where the empirical results show RaE's largest gains (Figures 3b, 4b-c).
  • You are running multiple experiments (seeds, hyperparameter sweeps, algorithm comparisons) and want to amortize exploration cost across them rather than discarding data between runs.

• Consider fine-tuning with offline pre-training instead when:

  • You need the fastest possible initial learning — weight transfer from a pre-trained policy provides faster early progress than RaE's from-scratch weight initialization (Figure 7a-b), though RaE catches up asymptotically.
  • Your offline dataset is small (roughly <10K episodes on a locomotion-scale domain) and you do not want to manually adjust the mixing ratio — Table 1 shows that RaE's 50/50 default can hurt performance in this regime, while offline pre-training followed by pure online fine-tuning avoids the risk of overfitting to scarce offline data.

• Consider combining RaE with weight transfer (fine-tuning with RaE, Appendix C.2) when:

  • You can transfer weights from a prior experiment (same architecture) AND you have the prior data — this combination yields both fast initial learning and the highest asymptotic performance (Figure 7a-b), making it the practical best-of-both-worlds option when both conditions are satisfied.