Unlocking the Power of Representations in Long-term Novelty-based Exploration

🎯 Pitch

RECODE is the first agent ever to reach the end screen in the notoriously hard exploration game Pitfall!, using a single, non-parametric clustering memory that tracks state visits without forgetting. Unlike prior methods that rely on brittle parametric models or dual novelty signals, RECODE shows that simply maintaining a good state representation with an adaptive memory can unleash superhuman exploration in both 3D and Atari benchmarks.

1. Executive Summary

This paper introduces Robust Exploration via Clustering-based Online Density Estimation (RECODE), a non-parametric method for novelty-based exploration that estimates soft visitation counts over clusters of states in a learned embedding space, and proposes Coupled Action-State Masking (CASM), a masked transformer architecture that extends inverse dynamics prediction to multi-step trajectories for richer state representations. Evaluated on Atari's hard exploration games and the DM-HARD-8 3D benchmark with the MEME agent, RECODE sets new state-of-the-art results on both suites—it is the first agent to reach the end screen in Pitfall! and the first to surpass human performance on Push Blocks—while also demonstrating robustness to observation noise that substantially degrades prior methods (NGU's performance deteriorates on Noisy Montezuma whereas RECODE's does not). The combination of RECODE with CASM achieves a new state-of-the-art on DM-HARD-8, establishing that a unified long-term clustering memory can replace the dual-mechanism approach of NGU and directly translate improvements in representation learning into exploration gains, though with the boundary condition that the memory's effectiveness depends on maintaining meaningful state similarity through representation choice and adaptive discounting to handle non-stationary embedding distributions.

2. Context and Motivation

The Core Problem: Exploration Requires Both Representation and Long-Term Memory

The central challenge this paper addresses is that deep reinforcement learning agents exploring sparse-reward environments need two things simultaneously to be effective: a meaningful state representation that captures task-relevant similarities while ignoring uncontrollable noise, and a long-term memory that tracks which parts of the state space have been visited over thousands of episodes — not just within a single episode. Critically, prior methods have only partially solved one or the other of these requirements, and no approach has successfully unified both while remaining simple enough to deploy as a drop-in exploration module.

This matters for several concrete reasons that the paper highlights, both explicitly and through its experimental design:

Sparse-reward tasks with long horizons: Environments like Montezuma's Revenge and Pitfall! require executing thousands of steps before receiving any reward signal. Without a robust intrinsic motivation to explore, agents simply never discover reward — they wander aimlessly or fixate on meaningless patterns. The Atari hard exploration suite (Bellemare et al., 2016) formalizes this challenge, with episodes lasting up to 27,000 steps under standard action repeat.
Partially observable 3D environments: The DM-HARD-8 suite (Gulcehre et al., 2019) presents procedurally-generated 3D worlds where agents must interact with specific objects (picking up bats, throwing balls, navigating mazes) while the visual appearance of objects and environments changes between episodes. This requires representations that can generalize across visual variation to recognize functionally similar states — something purely observation-based methods cannot do.
Noisy or uncontrollable observation features: Real sensors are imperfect. The "noisy TV problem" (Schmidhuber, 2010; Pathak et al., 2017) describes a fundamental failure mode where exploration bonuses based directly on observation novelty get trapped by irrelevant, unpredictable stimuli — a flickering screen, a death timer, randomly colored wall textures. An agent can stand still and receive infinite novelty from noise alone. This is not a toy problem: any practical deployment will face uncontrollable observation features that must be filtered out by the representation.
Practical deployment complexity: From an engineering standpoint, exploration methods that require multiple interacting components (separate short-term and long-term novelty estimators, as in NGU) introduce additional hyperparameters, training dynamics, and failure modes. A simpler unified approach is easier to tune, debug, and adapt to new domains.

Why Prior Approaches Fall Short

The paper's motivation draws on a long line of exploration research that has converged on count-based bonuses as a theoretically principled approach. The idea is straightforward: in tabular MDPs, exploration can be provably efficient by providing an intrinsic reward that decreases with visitation count — $r_t \propto 1 / \sqrt{N(s)}$ (Strehl & Littman, 2008; Azar et al., 2017). States visited often become boring; unvisited states become exciting. However, this elegant solution breaks down immediately in large or continuous state spaces, where the agent will literally never see the exact same state twice. The paper identifies two broad families of approximations that have emerged to address this, each with fundamental limitations:

Parametric Methods: Slow Adaptation and Catastrophic Forgetting

Parametric methods train neural networks to estimate visitation densities or density proxies from observations. They fall into several sub-categories, each with documented weaknesses:

Density estimation on observations. Early approaches trained explicit density models — Context Tree Switching (Bellemare et al., 2014), PixelCNN (Van den Oord et al., 2016) — directly on raw image observations to compute pseudo-counts (Bellemare et al., 2016; Ostrovski et al., 2017). The problem is scalability: these density models are computationally expensive to train and update, and modeling the full distribution of high-dimensional observations is an unnecessarily hard problem when the goal is merely to distinguish novel from familiar states.

Prediction error as a novelty proxy. Random Network Distillation (RND; Burda et al., 2019) simplified this substantially by training a predictor network to match the output of a fixed, randomly initialized target network on observations. The prediction error serves as a novelty signal: familiar observations are predicted well, novel observations poorly. RND provides strong baselines across many settings (Burda et al., 2018) but, crucially, operates on raw observations. This means it cannot leverage learned representations that filter out uncontrollable features — it treats the death timer in Pitfall! as novel at every timestep, even when the agent is standing still. The paper explicitly shows via ablation (MEME-RND in Figure 4) that RND alone fails on Pitfall! for exactly this reason.

Dynamics prediction on learned representations. Methods like ICM (Pathak et al., 2017) and BYOL-Explore (Guo et al., 2022) train representations through forward or inverse dynamics prediction and compute novelty from prediction errors in the learned embedding space. This addresses the noise-filtering issue — inverse dynamics in particular encourages representations that encode only controllable aspects of the state — but introduces new problems. The paper identifies slow adaptation and catastrophic forgetting as the core limitations of parametric novelty estimators. Gradient descent requires many updates to incorporate new observations, meaning the novelty signal cannot adapt rapidly within a single episode. And as the policy shifts and the distribution of visited states evolves, neural networks tend to overwrite older knowledge, causing the "detachment" problem (Ostrovski et al., 2017) where the algorithm loses track of promising areas to explore that were discovered earlier but haven't been visited recently.

Even BYOL-Explore, which the paper uses as a strong baseline on DM-HARD-8, ultimately relies on a parametric predictor network whose error provides the novelty signal. As a parametric method, it inherits these fundamental limitations around adaptation speed and forgetting, even though its representation learning is more sophisticated than RND's.

Non-Parametric Methods: Short Memory, Limited Generalization

Non-parametric methods avoid these problems by storing actual visited states in a memory and computing novelty based on similarity to stored states. The approach dates back to kernel-based RL (Ormoneit & Sen, 2002; Kveton & Theocharous, 2012; Barreto et al., 2016) and has been theoretically analyzed in continuous MDPs (Pazis & Parr, 2013; Domingues et al., 2021a). The key advantage is immediate responsiveness: new states can be compared against the memory instantly without gradient updates, and there is no forgetting since stored states persist unless explicitly removed.

However, non-parametric methods face a fundamental constraint: memory capacity. Storing every visited state is infeasible for agents running for hundreds of millions of steps. This forces a choice between:

Limiting the memory horizon: Store only recent states, typically within the current episode (as in episodic memory approaches like EMM, Pritzel et al., 2017). This preserves responsiveness but cannot track long-term exploration progress — the agent forgets everything at episode boundaries.
Limiting memory size with sampling: Store a fixed-size buffer with some selection strategy. But without careful aggregation, a fixed buffer either fails to cover the state space or becomes stale as the representation changes.

Never Give Up (NGU; Badia et al., 2020b) — the most directly comparable prior method — acknowledges this tension by combining two separate novelty mechanisms:

An episodic memory (non-parametric) that stores embedded observations within the current episode and computes a short-term novelty bonus via k-nearest-neighbor similarity (Equation 1 in the paper). This is fast, responsive, and doesn't suffer from forgetting (it resets every episode), but it has no memory beyond the current episode.
Random Network Distillation (parametric) that provides a long-term novelty signal. This remembers across episodes but inherits all the parametric limitations: inability to use learned representations (since RND operates on raw pixels), slow adaptation, and catastrophic forgetting.

The paper explicitly calls out the complexity and fragility of this dual-mechanism approach:

"the need to estimate two different novelty signals simultaneously adds complexity and requires careful tuning"

More critically, the two mechanisms interact in problematic ways:

"As NGU relies on RND for representation, it also inherits its encoding deficiencies in the presence of noisy observations which limits the applicability of the method in stochastic or complex environments."

The ablation in Figure 4 makes this concrete: MEME-EMM-AP (episodic memory only with action prediction embeddings) can solve Pitfall! (where within-episode memory suffices) but fails on Montezuma's Revenge (which requires long-term memory across episodes). Conversely, MEME-RND (long-term novelty on raw observations) fails on Pitfall! because it can't filter noise. NGU tries to paper over this with two modules, but the long-term component remains fundamentally limited by operating on the wrong representation.

The Two Sub-Problems the Paper Explicitly Decomposes

Section 1 frames the paper's contribution as disentangling the exploration problem into two independent sub-problems:

"In this paper, we tackle these issues by decomposing the exploration problem into two disentangled sub-problems. First, (i) Representation Learning with an embedding function that encodes a meaningful notion of state similarity while being robust to uncontrollable factors in the observations. Second, (ii) Count Estimation that is able to provide a long term visitation-based exploration bonus while retaining responsiveness to the most recent experience."

This decomposition is central to understanding the paper's positioning. Most prior work conflates these two problems: RND solves neither (it uses no learned representation, and its count estimation is implicit in the parametric predictor); NGU partially solves both but through a complex hybrid (representation via inverse dynamics for the episodic module, count estimation via RND for the long-term module). BYOL-Explore solves representation well (via bootstrapped latent prediction) but count estimation remains parametric and implicit.

The paper's claim is that by solving each sub-problem independently and well — RECODE for count estimation, CASM (or any learned embedding) for representation — you get a simpler, more robust, and higher-performing system than approaches that try to solve both simultaneously or that compromise on one to address the other.

The Specific Gap: No Method Provides Both Unified Long-Term Memory AND Representation Flexibility

Synthesizing the above, the specific gap the paper identifies is:

No existing exploration method provides a single, unified non-parametric memory that (a) spans thousands of episodes without resets, (b) operates on learned state representations rather than raw observations, and (c) adapts to non-stationary embedding distributions during training.

The consequences of this gap are visible in the experimental results:

NGU solves Montezuma's Revenge (via the combined EMM + RND) but is complex and fails under observation noise.
RND solves some games but fails on Pitfall! because it can't use learned representations.
EMM solves Pitfall! (when combined with action prediction embeddings) but fails on Montezuma's Revenge because episode resets erase long-term memory.
BYOL-Explore achieves state-of-the-art on DM-HARD-8 but is restricted to its specific parametric representation and novelty mechanism.

RECODE is positioned as filling exactly this gap: a clustering-based non-parametric memory that compresses observations into a fixed-size set of atoms, tracks soft visitation counts via aggregated similarity rather than discrete counts, decays old clusters to handle non-stationarity, and is agnostic to the choice of embedding function — it can leverage action prediction, BYOL embeddings, or the new CASM transformer representations interchangeably.

How the Sub-Problems Connect to Experimental Design

The paper's experimental structure directly reflects this decomposition:

The Atari experiments (Section 5.1) keep the base agent (MEME) and representation (AP) fixed while varying only the exploration bonus (EMM vs. RND vs. NGU vs. RECODE). This isolates the count estimation sub-problem and demonstrates that RECODE's unified long-term memory outperforms the EMM+RND hybrid in NGU, while using a conceptually simpler mechanism.
The DM-HARD-8 experiments (Section 5.2) vary both the exploration bonus and the representation, showing that RECODE scales with representation quality (CASM > AP) in a way that NGU cannot fully exploit because its RND-based long-term component can't leverage the learned embeddings.
The noisy Montezuma experiment (Section 5.3) directly tests the robustness claim: by keeping AP fixed and adding observation noise, the experiment isolates whether the exploration bonus can still function when the representation has been trained to filter noise (which AP does through inverse dynamics). NGU fails because its RND component operates on raw observations and degenerates to random exploration; RECODE succeeds because it operates entirely on the noise-filtered AP embeddings.
The CASM architecture (Section 4) addresses the representation sub-problem independently: it extends inverse dynamics from single-step to multi-step prediction using masked transformers, learning richer representations that capture higher-level environment structure (landmarks, agent position) rather than just low-level transitions. The paper is careful to note that CASM's output is the raw encoder output $f_\theta(o_t)$ , not the transformer output, to avoid leaking trajectory information into the exploration bonus.

Broader Context: The Shift Toward Representation-Aware Exploration

While not explicitly stated as such, the paper participates in a broader shift in the exploration literature away from methods that treat observations as atomic units of novelty and toward methods that explicitly separate "what makes states similar" (representation) from "how many times have I been here" (counting). This shift is motivated by the recognition that exploration in complex environments is fundamentally a semantic problem — novelty is not about pixel-level differences but about functional differences in the agent's interaction with the world.

The paper cites relevant work in this direction: Pathak et al. (2017) first showed that inverse dynamics could filter uncontrollable features; Burda et al. (2018) systematically studied curiosity-driven learning across many environments; Guo et al. (2020, 2022) developed increasingly sophisticated self-supervised representation objectives for exploration; Domingues et al. (2021b) used density estimation on top of learned representations with theoretical motivations. RECODE's contribution is making the counting side of this decomposition work at scale — providing a non-parametric memory that can track novelty over thousands of episodes while adapting to the continually changing representations produced by these learning methods.

3. Technical Approach

3.1 Reader Orientation

What the system is: RECODE is a drop-in exploration bonus module for deep RL agents that maintains a fixed-size memory of "atoms" — cluster centers in a learned embedding space — and computes intrinsic rewards that are inversely proportional to how often the agent has visited states near each atom.

What problem it solves and the "shape" of the solution: The system solves the problem of long-term novelty estimation in continuous or large state spaces by replacing discrete state counts (which are impossible when states never repeat exactly) with a soft, similarity-weighted sum over a compressed memory that adaptively clusters observed states, tracks how many times each cluster has been visited, and adjusts to non-stationary embedding distributions through count discounting and cluster replacement — all while maintaining constant memory size and operating on any chosen embedding function.

3.2 Big-Picture Architecture (Diagram in Words)

The system has three major components that interact in a specific pipeline at each timestep:

Embedding Function ( $f_\theta : \mathcal{O} \rightarrow \mathcal{E}$ ): A neural network (trained separately, e.g., via inverse dynamics prediction or CASM) that maps the current observation $o_t$ (or observation history $h_t$ ) to a vector $e_t \in \mathbb{R}^D$ in a learned embedding space. This component is responsible for "what makes states similar" — filtering out uncontrollable noise and encoding task-relevant structure. RECODE is agnostic to how this embedding is trained; it can use action prediction (AP), CASM, BYOL, or any other representation.
RECODE Memory ( $M = \{m_l\}_{l=1}^{|M|}$ with associated counts $\{c_l\}_{l=1}^{|M|}$ ): A fixed-size container of atoms (cluster centers in the embedding space), each with an associated soft visitation count. This component is responsible for "how many times have I been here" — it maintains a compressed, long-term summary of the agent's exploration history spanning thousands of episodes without resets. The memory size $|M|$ is set to $5 \times 10^4$ atoms for Atari and $2 \times 10^5$ atoms for DM-HARD-8.
Intrinsic Reward Computation: Given the current embedding $e_t$ and the current memory state, compute a soft-visitation count $N_{\mathcal{K}}(M, e_t)$ as a kernel-weighted sum over nearby atoms, then convert it to an intrinsic reward $r_t$ via $r_t \propto 1 / \sqrt{N_{\mathcal{K}}(M, e_t)}$ . The reward is then normalized by a running standard deviation estimate (as in RND) and added to the extrinsic reward for RL training.

Information flow at each timestep $t$ :

The agent receives observation $o_t$ (or history $h_t$ ).
The embedding function produces $e_t = f_\theta(o_t)$ .
$e_t$ is sent to RECODE, which: (a) computes the intrinsic reward $r_t$ from the current memory state; (b) updates the memory to incorporate $e_t$ (potentially modifying atom positions, incrementing counts, or replacing stale atoms); (c) applies count discounting to all atoms.
The intrinsic reward $r_t$ is returned and added to any extrinsic reward for policy training.
The embedding function $f_\theta$ is trained concurrently (via its own objective, e.g., inverse dynamics), meaning the embedding space itself evolves over time — a non-stationarity that RECODE must handle.

In a distributed RL setting (Section F, Figure 15), Actors query an Inference Worker for embeddings and communicate with a shared RECODE Memory that returns intrinsic rewards, while a Learner performs gradient updates on the policy and embedding networks. The memory is shared across all actors, increasing update frequency and reducing representation drift between updates compared to NGU's per-actor episodic memories.

3.3 Roadmap for the Deep Dive

First, the soft-visitation count estimator (Equations 2 and 3), which defines how RECODE converts a memory of atoms into an aggregate "how many times have I been here" signal — this is the core novelty signal and drives everything downstream.
Second, the memory update rules (Algorithm 1, lines 8–18), which adapt classical DP-means clustering to the streaming, non-stationary RL setting — this is the mechanism that compresses experience into a fixed-size memory and determines what information is preserved versus discarded.
Third, the count discounting mechanism (Algorithm 1, line 7) and adaptive bandwidth (lines 5–6), which handle the non-stationarity of both the policy's state distribution and the embedding function itself — these are the critical modifications that allow RECODE to function over thousands of episodes where prior non-parametric methods are limited to single episodes.
Fourth, the atom removal and redistribution strategies (lines 11–13), which maintain constant memory size with minimal information loss — this is a subtle but important design choice that affects memory coverage and stability.
Fifth, the connection between RECODE's update rules and the DP-means clustering framework (Appendix C), which provides theoretical grounding for why the algorithm works as a density estimator — understanding this helps clarify why the specific update rules (convex combination for atom positions, count increment for assignments, probabilistic removal) are principled rather than arbitrary.
Sixth, the CASM representation learning architecture (Section 4), which extends inverse dynamics from single-step to multi-step prediction using masked transformers — while CASM is independent from RECODE, understanding it is essential because the paper's strongest results (DM-HARD-8) come from combining the two, and because CASM exemplifies the kind of rich representation that RECODE can leverage but that parametric methods like RND cannot.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methods paper whose core idea is that long-term novelty-based exploration can be achieved by maintaining a clustering of observed states in a learned embedding space and computing intrinsic rewards from soft visitation counts that aggregate similarity across clusters, rather than requiring separate short-term and long-term mechanisms.

Soft-Visitation Count Estimation

The fundamental quantity RECODE computes is a soft-visitation count $N_{\mathcal{K}}(M, e)$ — an estimate of how many times the agent has visited states similar to the query embedding $e$ , based on the current memory $M$ of atoms and their associated counts. This replaces the discrete state counts $N(s)$ from tabular RL with a continuous, similarity-weighted analogue suitable for embedding spaces where exact state matches never occur.

The soft-visitation count is defined as:

$N_{\mathcal{K}}(M, e) = \sum_l (1 + c_l) \mathcal{K}(m_l, e; d_{\text{ema}})$

where $M = \{m_l\}_{l=1}^{|M|}$ is the set of atoms (cluster centers in the embedding space $\mathcal{E}$ ), $c_l$ is the estimated visitation count associated with atom $m_l$ (roughly, the number of observations that have been assigned to this cluster), and $\mathcal{K}(m_l, e)$ is a similarity kernel between atom $m_l$ and the query embedding $e$ , parameterized by an adaptive bandwidth $d_{\text{ema}}$ .

What it computes: For a given query embedding $e$ , the sum iterates over all atoms in memory. Each atom contributes $(1 + c_l)$ — its count plus one (the "plus one" prevents zero-count atoms from contributing nothing, ensuring even newly created atoms provide some signal) — multiplied by the kernel $\mathcal{K}(m_l, e)$ that decays with distance. Atoms far from $e$ contribute approximately zero; nearby atoms contribute proportionally to their visitation counts and similarity. The result is a single positive scalar representing the aggregate "familiarity" of the region around $e$ .

Why this form: This is a kernel density estimate in the embedding space, with the key difference that the "data points" are not individual stored observations but aggregated cluster centers, each weighted by how many observations they represent. The sum-over-all-atoms formulation avoids a counter-intuitive behavior of k-nearest-neighbor (k-NN) approaches that RECODE explicitly identifies and corrects:

"it is desirable that the soft-visitation count of a given embedding should increase after adding it to the memory. However, adding atoms to the memory can change the k-NN list. If an atom displaced from this list has a large count, this might actually reduce nearby soft-visitation count estimates instead of increasing them."

In other words, with k-NN, inserting a new observation into memory could push a heavily-visited nearby atom out of the top-k neighbor set, causing the soft count to paradoxically decrease even though the agent has visited the region more. RECODE's formulation sums over all atoms within the kernel bandwidth, so adding an atom can only increase (or leave unchanged) the soft-visitation count for nearby queries — it never decreases. This monotonicity property aligns with the intuitive notion that visiting a state more should make it (and its neighborhood) seem more familiar, not less.

The kernel function itself is defined as:

$\mathcal{K}(m_l, e) = \frac{1}{1 + \frac{\|e - m_l\|_2^2}{\epsilon d_{\text{ema}}^2}} \mathbb{1}\{\|e - m_l\|_2^2 < d_{\text{ema}}^2\}$

where $\epsilon \in \mathbb{R}_+$ is a fixed scaling parameter, $d_{\text{ema}}^2$ is the adaptive squared-bandwidth computed from recent embedding distances (see below), and $\mathbb{1}\{\cdot\}$ is the indicator function that is 1 when the condition holds and 0 otherwise.

What it computes: The kernel evaluates the similarity between atom $m_l$ and query $e$ as a function of their squared Euclidean distance $\|e - m_l\|_2^2$ . The core term $\frac{1}{1 + \text{distance} / (\epsilon \cdot \text{bandwidth})}$ produces values in $(0, 1]$ : when $m_l = e$ exactly (distance zero), the kernel equals 1; as the distance grows, the kernel smoothly decays toward zero. The indicator function applies a hard cutoff — atoms farther than $d_{\text{ema}}$ from $e$ contribute exactly zero, regardless of their count. This sparse cutoff is computationally efficient (only nearby atoms need to be evaluated) and prevents distant, unrelated atoms from contributing noise to the count estimate.

Why this form: The combination of a rational decay (rather than Gaussian) and a hard cutoff balances smooth interpolation with computational efficiency. The rational function $1/(1 + x)$ has heavier tails than a Gaussian, meaning it doesn't collapse to near-zero as quickly for moderate distances — this provides more robust generalization across the embedding space. The hard cutoff at $d_{\text{ema}}$ ensures the estimator remains local (it's a genuine density estimate for the current neighborhood, not a global average) and limits computation to a sparse subset of the memory. The scaling factor $\epsilon$ controls how quickly similarity decays within the bandwidth — a smaller $\epsilon$ makes the kernel decay faster, effectively requiring atoms to be closer to contribute meaningfully.

Once the soft-visitation count is computed, the intrinsic reward follows the classical count-based exploration formula:

$r_t \propto \frac{1}{\sqrt{N_{\mathcal{K}}(M, e_t)}}$

with a small constant $n_0$ added to the denominator for numerical stability (preventing division by zero for completely novel states where the soft count is near zero), and the reward is normalized by a running estimate of its standard deviation as in RND to maintain consistent scale across training. The constant $n_0 = 0.01$ is used in all experiments (Tables 1, 2).

Why this form: The inverse square root scaling is theoretically motivated — for tabular MDPs, bonuses proportional to $1/\sqrt{N(s)}$ provide provably efficient exploration (Strehl & Littman, 2008; Azar et al., 2017). RECODE extends this principle to continuous embedding spaces by replacing the discrete count $N(s)$ with the soft count $N_{\mathcal{K}}(M, e)$ . The square root ensures that as a region becomes heavily visited, the bonus decays smoothly rather than vanishing entirely — even familiar states retain a small exploration incentive, which prevents the agent from prematurely abandoning regions that might contain undiscovered subtleties.

Memory Update Rules: Streaming Clustering with Fixed Capacity

The core algorithmic innovation of RECODE is how it maintains the memory $M$ — a set of at most $|M|$ atoms with associated counts — under a streaming, non-stationary data distribution with fixed memory capacity. At each timestep, given a new embedding $e_t$ , the algorithm must decide whether to (a) assign the new embedding to an existing atom, or (b) create a new atom (requiring the removal of an existing one to maintain constant size), and must update atom positions and counts accordingly.

The paper draws an explicit parallel to DP-means clustering (Kulis & Jordan, 2011), a Bayesian nonparametric clustering algorithm that extends k-means. In DP-means, a new data point is either assigned to the nearest existing cluster (if within a threshold) or spawns a new cluster (if far from all existing clusters), with the threshold determined by a cluster penalty parameter. RECODE adapts this framework with three critical modifications for the streaming RL setting: fixed memory capacity (new clusters require removing old ones), count discounting (to handle non-stationarity), and online updates (no batch re-optimization over stored data).

Step 1: Find the nearest atom. Given the new embedding $e$ , identify the closest atom in memory:

$m_* = \arg \min_{m \in M} \|m - e\|_2$

where $m_*$ is the atom with minimum Euclidean distance to $e$ .

Step 2: Decide whether to assign or (potentially) create. The decision rule has two components — a distance check and a stochastic insertion probability:

Distance check: Is $\|m_* - e\|_2^2 < \kappa d_{\text{ema}}^2$ ? Here $\kappa > 0$ is a fixed relative tolerance parameter (set to $0.2$ in all experiments), and $d_{\text{ema}}^2$ is the adaptive squared-bandwidth. If the distance is less than the threshold, $e$ is close enough to an existing atom and will be assigned to it. If the distance is greater than the threshold, $e$ is far from all memory atoms, and the algorithm considers creating a new atom.
Stochastic insertion: When the distance exceeds the threshold, a random coin flip with probability $\eta$ determines whether to actually create a new atom. If the coin succeeds ( $u < \eta$ for a uniform random $u \in [0,1]$ ), a new atom is created. If it fails, $e$ is instead assigned to $m_*$ as if it were close enough. The insertion probability $\eta$ is set to $0.05$ for Atari and $0.2$ for DM-HARD-8.

Why the stochastic insertion? The coin flip provides robustness to outliers. An embedding that is far from memory but seen only once (e.g., a glitch, a rare random transition, or embedding noise) will only be inserted with probability $\eta$ — on average, it needs to be seen $1/\eta$ times before being added. This prevents the memory from wasting limited capacity on one-off anomalies. At the same time, genuinely novel regions that the agent visits repeatedly will eventually (with high probability within a few observations) trigger insertion, ensuring the memory expands its coverage to newly discovered areas. The higher $\eta$ for DM-HARD-8 ( $0.2$ vs. $0.05$ ) reflects the larger memory size ( $2 \times 10^5$ vs. $5 \times 10^4$ atoms) and the greater diversity of the 3D environment — there's more capacity and more need to rapidly incorporate novel states.

Why the relative tolerance $\kappa d_{\text{ema}}^2$ ? Using the adaptive bandwidth $d_{\text{ema}}$ (described below) as the distance scale ensures the insertion threshold automatically calibrates to the current embedding distribution — in a sparse embedding space, $d_{\text{ema}}$ will be large and the threshold relaxed; in a dense space, $d_{\text{ema}}$ will be small and the threshold tightened. The parameter $\kappa$ provides a fixed relative tolerance on top: $\kappa = 0.2$ means an embedding must be at least 20% of the typical inter-atom distance away to be considered for insertion.

Step 3a: Assignment (when not creating a new atom). When $e$ is assigned to its nearest atom $m_*$ (either because it was close enough or because the insertion coin flip failed), the atom's position and count are updated via a convex combination:

$m_* \leftarrow \frac{c_*}{c_* + 1} m_* + \frac{1}{c_* + 1} e$

$c_* \leftarrow c_* + 1$

What this computes: The new atom position is a weighted average of the old position and the new embedding, with weights proportional to the old and new counts. If the atom previously represented $c_*$ observations, the new position moves $1/(c_* + 1)$ of the way toward $e$ . The count is incremented by 1 to reflect the additional observation.

Why this form: This update is exactly the posterior mean update for a Gaussian model with known variance and a Gaussian prior on the cluster mean (see Appendix C.2 for the derivation). Specifically, if each atom's associated data points are modeled as $\mathcal{N}(\mu_l, 1)$ with prior $\mu_l \sim \mathcal{N}(0, c_0)$ , then after observing $c_l$ points, the posterior mean is the running average. The update $\frac{c_*}{c_* + 1} m_* + \frac{1}{c_* + 1} e$ is precisely the incremental update for this running average. This probabilistic interpretation explains why the counts $c_l$ serve dual purpose: they are both visitation counts (density estimates) and precision parameters (inverse variance) of the posterior over the atom's position. An atom with high count has a precise position (low uncertainty) and moves slowly when assigned new embeddings; an atom with low count is uncertain and adapts rapidly.

Why not batch re-optimization? Classical k-means and DP-means alternate between assigning all data points to clusters and recomputing cluster means from all assigned points. In the streaming RL setting, this is impossible — past embeddings are not stored, so the running average is the only feasible estimate. The incremental formulation is computationally efficient (constant time per update) and naturally handles the sequential arrival of data without requiring storage or replay.

Step 3b: Atom creation and removal (when creating a new atom). When the distance exceeds the threshold and the insertion coin flip succeeds, a new atom must be created at $e$ , requiring the removal of an existing atom to maintain constant memory size $|M|$ . The procedure is:

Select an atom $j$ for removal with probability $P(j) \propto 1/c_j^2$ . Atoms with smaller counts are more likely to be removed — the inverse square weighting strongly prefers removing atoms that represent very few observations.
Find the nearest neighbor $m_\dagger$ of the removed atom: $m_\dagger = \arg \min_{m \in M, m \neq m_j} \|m - m_j\|_2$ . This is the atom closest to the one being removed.
Redistribute the count: $c_\dagger \leftarrow c_j + c_\dagger$ . The removed atom's count is transferred to its nearest neighbor, preserving the total count in memory (roughly, preserving the total amount of "experience" tracked).
Insert the new embedding at the vacated position $j$ : $m_j \leftarrow e, c_j \leftarrow 1$ . The new atom starts with count 1 (it represents one observed embedding).

Why $P(j) \propto 1/c_j^2$ ? The paper empirically compares three removal strategies: (a) probability proportional to $1/c_j^2$ , (b) probability proportional to $1/c_j$ , and (c) deterministically removing the atom with the smallest count (Appendix C.2, Figures 8 and 9). The toy experiment in Figure 8 shows that the $1/c_j^2$ strategy produces cluster centers and counts that become approximately uniform for a stationary distribution — neither the clusters nor their weights remain skewed after the distribution stabilizes. With deterministic removal of the smallest-count atom, cluster centers can remain skewed long after the distribution stops changing (the smallest-count atoms tend to be in sparse regions, creating a bias). The Atari ablation in Figure 9 confirms that $1/c_j^2$ is the most robust choice, particularly on Pitfall!, though all strategies produce viable algorithms. The square in the denominator amplifies differences between counts — an atom with count 2 is 4 times more likely to be removed than one with count 1, while an atom with count 100 is 10,000 times less likely than one with count 1 — creating a strong preference for removing truly underutilized atoms while making heavily-used atoms essentially permanent until their counts decay.

Why redistribute counts rather than discarding? Preserving the total count in memory through redistribution is a form of soft deletion. The removed atom's count is not lost but is merged into its nearest neighbor — the region still "remembers" that it was visited, just with less spatial precision. This contrasts with simply discarding the count, which would permanently erase the visitation record for that region. The paper states that redistribution was chosen because it is "less sensitive to the choice of hyperparameters" than complete removal.

Connection to the optimization objective. Appendix C.1 shows that the assignment and update rules minimize a streaming version of the DP-means objective at each step for the current embedding $e_t$ :

$\begin{aligned} & \min_{l \in 1, \dots, |M|} \|m_l - e_t\|_2^2 \\ & \text{s.t.} \quad \|m_l - e_t\|_2^2 \leq \kappa d \end{aligned}$

The "else" branch (assignment to nearest atom and update) reduces the squared distance between $m_*$ and $e_t$ directly. The "if" branch (creating a new atom at $e_t$ ) achieves the global minimum of zero squared distance. The hard constraint $\|m_l - e_t\|_2^2 \leq \kappa d$ plays the role of the cluster penalty in DP-means — it limits how far a point can be from its assigned cluster before a new cluster is warranted. The key difference from standard DP-means is that this optimization is performed incrementally for each new embedding rather than globally over all data (which is not stored).

Adaptive Bandwidth Estimation

The distance scale $d_{\text{ema}}^2$ that parameterizes both the kernel (Equation 3) and the insertion threshold must adapt to the current embedding distribution, which can vary dramatically between environments and over the course of training as both the policy and the embedding function $f_\theta$ evolve. RECODE updates $d_{\text{ema}}^2$ via an exponential moving average of the mean squared distance from the current embedding to its k-nearest atoms:

$d_{\text{ema}}^2 \leftarrow (1 - \tau) d_{\text{ema}}^2 + \frac{\tau}{k} \sum_{m \in \text{Neigh}_k(e)} \|m - e\|_2^2$

where $\text{Neigh}_k(e)$ is the set of $k$ nearest atoms to the embedding $e$ in the current memory ( $k = 20$ in all experiments), and $\tau$ is the decay rate for the moving average.

What it computes: Each time an embedding arrives, RECODE finds its $k$ nearest neighbors in memory, computes the average squared distance to those neighbors, and updates $d_{\text{ema}}^2$ as an exponential moving average of these distances. The parameter $\tau$ controls how quickly the estimate adapts — larger $\tau$ means faster adaptation but more variance. The paper uses $\tau = 0.9999$ for both Atari and DM-HARD-8 (Tables 1, 2), which produces a very slow-moving average with a long memory.

Why an EMA and not a simple running average? The paper replaces NGU's running average (which weights all past observations equally) with an EMA specifically to allow "more rapid adaptation" of $d_{\text{ema}}$ to changes in the embedding distribution. An EMA with $\tau = 0.9999$ gives more weight to recent observations while still maintaining a long effective horizon — the half-life is approximately $\ln(0.5) / \ln(1 - 0.9999) \approx 6931$ steps, meaning observations from ~7000 steps ago contribute half as much as recent ones. This balances stability (needed for consistent kernel behavior) with responsiveness (needed because the embedding space can change).

Why k-nearest neighbors rather than all atoms? Using the k-nearest neighbors provides a local estimate of the embedding density around $e$ rather than a global average. This matters because the embedding space may have regions of very different density — in densely-explored areas, the typical inter-atom distance is small; in frontier regions, it's large. The k-NN approach ensures $d_{\text{ema}}^2$ reflects the local scale, which is what matters for both the kernel cutoff and the insertion threshold. The value $k=20$ is large enough to provide a stable local estimate without being dominated by distant atoms.

Role in the kernel and insertion threshold. The adaptive bandwidth $d_{\text{ema}}^2$ serves as the unit of distance for both the kernel decay and the insertion decision. In the kernel (Equation 3), it scales the distance before applying the rational decay and provides the hard cutoff radius. In the insertion threshold ( $\kappa d_{\text{ema}}^2$ ), it determines what constitutes "far away." This means the same adaptation mechanism controls both how similarity is measured and when new atoms are created — a coherent design that ensures the memory's density and coverage automatically calibrate to the embedding scale.

Count Discounting: Handling Non-Stationary Embedding Distributions

The adaptive bandwidth alone is insufficient to handle the primary source of non-stationarity in RECODE: the embedding function $f_\theta$ is trained concurrently with the policy, meaning the same physical state may be mapped to different embedding vectors at different points in training. An atom created early in training under an old representation may have a high count but be effectively unreachable under the new representation — no current observations map near it — yet its high count makes it unlikely to be removed via the $1/c_j^2$ probability.

To address this, RECODE applies a multiplicative discount $\gamma < 1$ to all atom counts at every timestep:

$c_l \leftarrow \gamma c_l \quad \forall l \in \{1, \dots, |M|\}$

What it computes: At each step, every atom's count is multiplied by $\gamma$ . Atoms that receive new assignments have their counts incremented afterward (or are updated via the convex combination), partially offsetting the discount. Atoms that receive no assignments for many steps see their counts decay exponentially toward zero.

Why this works: The discount serves as a "forgetting" mechanism. An atom that was heavily visited under an old representation but is no longer reachable under the current representation will gradually lose count through discounting, since no new assignments offset the decay. As its count decreases, the $1/c_j^2$ removal probability increases, making it more likely to be replaced when a new atom is inserted. Conversely, atoms in regions the agent currently visits receive regular count increments that offset the discount — they remain alive and retain their high counts. The discount rate $\gamma$ controls the effective memory horizon: $\gamma = 0.999$ for Atari and $\gamma = 0.997$ for DM-HARD-8.

Magnitude and interpretation. With $\gamma = 0.999$ (Atari), an atom that receives no assignments for 1000 steps will have its count reduced to $0.999^{1000} \approx 0.37$ of its original value — a substantial reduction. After 10,000 steps (approximately the length of one episode in the hard exploration Atari games), the count drops to $0.999^{10000} \approx 0.000045$ , effectively zero. This means atoms that haven't been visited within roughly one episode's worth of steps lose almost all their count and become prime candidates for removal. For DM-HARD-8, $\gamma = 0.997$ produces faster decay: after 1000 steps, the count reduces to $0.997^{1000} \approx 0.05$ , corresponding to the shorter episode lengths and more rapid exploration dynamics of the 3D environments.

Empirical validation. Figure 3 in the paper shows the age distribution of atoms in the memory of a RECODE agent that has learned to reach Pitfall!'s end screen. The red vertical line marks the maximum episode length (enforced by the in-game death timer) — this is the maximum memory horizon for methods like NGU that reset their episodic memory at episode boundaries. The RECODE memory contains atoms across a wide age range: most atoms are significantly older than one episode, with some reaching back thousands of episodes. This demonstrates that discounting doesn't erase all long-term memory — it selectively removes stale atoms while preserving those in regions the agent continues to visit. The paper explicitly notes that "earliest memories reaching back thousands of episodes" coexist with more recent atoms, showing the memory maintains a multi-scale temporal representation.

Ablation on discount (Appendix C.2, Figure 11). Removing discount ( $\gamma = 1$ ) causes significant performance degradation on Montezuma's Revenge and Pitfall!, confirming that the ability to forget stale representations is essential when embeddings are trained concurrently. Figure 12 shows an interesting nuance: when embeddings are pretrained and frozen (no representation drift), RECODE performs well even with $\gamma = 1$ (no discount). This confirms that the discount is specifically needed to handle the co-evolution of the embedding function and the memory — it's the interaction between learning and remembering that creates the problem.

Theoretical interpretation (Appendix C.2). The discount has a clean probabilistic interpretation in the Gaussian cluster model. In this model, each atom's associated data are modeled as $e_i | \mu_l \sim \mathcal{N}(\mu_l, 1)$ with prior $\mu_l \sim \mathcal{N}(0, c_0)$ , and after observing $c_l$ points the posterior over $\mu_l$ is $\mathcal{N}(m_l, c_l)$ — the count $c_l$ is the precision (inverse variance) of the posterior. Discounting $c_l \leftarrow \gamma c_l$ effectively increases the posterior variance of each cluster at every step, making it more uncertain about which data points belong to it. The paper explains:

"this uncertainty increase applied at each step acts as a 'forgetting' mechanism that helps the algorithm to deal with a changing data distribution"

As $\gamma$ decreases from 1, the "diffusion rate" increases — clusters become less certain faster, making the memory more short-term. The toy experiment in Figure 10 (Appendix C.2) demonstrates this: for small $\gamma$ (0.9, 0.99), the memory tracks only the current distribution (short-term); for large $\gamma$ (0.999, 0.9999), the memory retains historical density information (long-term). The paper emphasizes that $\gamma$ provides a smooth interpolation between purely short-term memory (like NGU's episodic memory) and purely long-term memory (like an undiscounted memory that never forgets), with the specific choice determining the algorithm's effective horizon.

Shared Memory Across Actors

A subtle but practically important design choice is that RECODE's memory is shared across all actors in a distributed RL setup, rather than each actor maintaining its own copy:

"unlike NGU where each actor maintains its own copy of the memory, RECODE shares the memory across all actors in a distributed agent, which greatly increases the frequency of updates to each atom resulting in less representation drift between memory updates."

In a typical distributed RL architecture (Appendix F, Figure 15), multiple Actors run in parallel, each executing actions in separate environment instances. With NGU, each actor's episodic memory is local — it only sees the observations from that actor's current episode and resets at episode boundaries. With RECODE, all actors write to and read from the same shared Memory (which implements Algorithm 1). This has two consequences:

Higher update frequency: With $N$ actors, each atom receives approximately $N$ times more updates per unit of wall-clock time. This means the memory integrates experience across all parallel environment instances, not just a single trajectory. The representation $f_\theta$ drifts more slowly per update when measured in terms of wall-clock time, because more updates happen between parameter changes.
Shared exploration state: All actors benefit from each other's exploration. If one actor discovers a novel region, atoms are created or counts incremented, and other actors will immediately see reduced intrinsic rewards for that region — encouraging them to explore elsewhere. This provides a global exploration coordination mechanism without explicit communication between actors.

The practical effect is that the shared memory can be much more responsive to the current embedding distribution while still maintaining long-term coverage, because the high update rate keeps atoms "alive" (counts refreshed) in actively visited regions across all actors simultaneously.

CASM: Coupled Action-State Masking for Multi-Step Representation Learning

While RECODE is agnostic to the embedding function, the paper introduces a novel representation learning architecture — Coupled Action-State Masking (CASM) — that provides substantially richer embeddings than standard 1-step inverse dynamics prediction, particularly for partially observable 3D environments. CASM is the representation used to achieve the state-of-the-art results on DM-HARD-8 (Section 5.2).

Motivation. The standard 1-step action prediction (AP) embedding (Pathak et al., 2017) trains $f_\theta$ and a classifier $g_\phi$ to predict the action $a_t$ given the embeddings of two consecutive observations $f_\theta(o_t)$ and $f_\theta(o_{t+1})$ :

$\min_{\theta, \phi} \mathcal{L}(\theta, \phi)(a_t) = -\ln(p_{\theta, \phi}(a_t | o_t, o_{t+1}))$

where $p_{\theta, \phi}(a_t | o_t, o_{t+1}) = g_\phi(f_\theta(o_t), f_\theta(o_{t+1}))$ . The intuition is that to predict what action was taken between two observations, the embedding must capture controllable aspects of the state — things the agent's actions affect — while ignoring uncontrollable noise (since noise doesn't help predict actions). This has proven effective for filtering out spurious novelty sources like the death timer in Pitfall! (Badia et al., 2020b).

However, the paper identifies a fundamental limitation: the 1-step prediction task is too simple. It can often be solved by encoding highly localized, low-level features — for instance, detecting how a single object shifted between frames — which need not capture the global structure of the environment. In partially observable 3D environments like DM-HARD-8, the agent needs to track its position, the relative locations of landmarks, and the state of interacted objects over extended trajectories — information that is not simultaneously present in any pair of consecutive observations.

The CASM solution. CASM extends the inverse dynamics principle to multi-step trajectories using a masked transformer architecture, inspired by masked language models like BERT (Devlin et al., 2018). The key idea is to provide a wider temporal context but randomly mask either the observation embedding or the action at each timestep, forcing the model to infer missing information from surrounding context. This makes the prediction task hard enough to require rich, high-level representations while still filtering uncontrollable features (since only controllable elements — actions and the effects of actions — need to be inferred).

Architecture (Figure 2). The CASM pipeline works as follows:

Sequence construction: For a trajectory of length $k+1$ , construct an interleaved sequence of observation embeddings and actions: $(f_\theta(o_{t-k}), a_{t-k}, f_\theta(o_{t-k+1}), \dots, a_{t-1}, f_\theta(o_t))$ . The embedding function $f_\theta$ is a feed-forward neural network (the same architecture as used for AP).
Masking: At each timestep $i$ in the sequence (where $i$ indexes either an observation or an action), independently and randomly substitute either $f_\theta(o_i)$ or $a_i$ with a special [MASK] token. The paper uses a state mask rate of 0.8 (Table 3), meaning 80% of observation embeddings are masked. Actions and observations are never simultaneously masked at the same timestep, to avoid making the prediction task indeterminate.
Transformer processing: The masked sequence is fed to a causally-masked transformer (specifically a GatedTransformerXL with 2 layers, attention size 128, 4 attention heads, and MLP hidden size 512 — see Table 3). The causal masking ensures the transformer can only attend to past and present tokens, not future ones, preserving the temporal structure.
Output projection: The transformer outputs at each position are projected down to the same dimensionality as the original embeddings: $\dim(z_t) = \dim(f_\theta(o_t))$ . For positions where the observation was masked, $z_t$ represents the transformer's best reconstruction of what the missing embedding should be; for unmasked positions, it provides a context-enriched representation.
Action prediction: The reconstructed embeddings at positions $t-1$ and $t$ (i.e., $z_{t-1}$ and $z_t$ ) are concatenated and fed to an MLP classifier $g_\phi$ that predicts the probability distribution over actions $p(a_t | z_{t-1}, z_t)$ . The training objective is the negative log-likelihood of the true action $a_t$ .
Multiple masks per trajectory: During training, $N=4$ different random masking patterns are sampled per trajectory to reduce gradient variance — effectively, each trajectory is processed four times with different subsets of information masked out, and the losses are averaged.

What embedding is used for RECODE? Crucially, the embedding provided to RECODE is $e_t = f_\theta(o_t)$ — the input to the transformer, not the transformer output. The paper is explicit about this:

"Note that the final embedding that we provide to RECODE is $e_t = f_\theta(o_t)$ , i.e. the transformer inputs, to avoid leaking information about the agent's trajectory."

The transformer outputs $z_t$ contain information aggregated from the entire masked context — they "know" about surrounding observations and actions. If these were used as the exploration embedding, the representation for a given observation would depend on what happened before and after it, creating a non-Markovian exploration signal that could leak information about trajectory structure. By using the feed-forward encoder output $f_\theta(o_t)$ , RECODE's embedding depends only on the current observation, preserving the Markov property.

Why masking is essential. The masking is what makes the prediction task non-trivial. If all observations and actions were provided unmasked (which the paper calls "CASnotM" — CASM without masking), the transformer could simply copy the observation at time $t$ to predict the action — the task would be nearly as easy as 1-step AP, and the representation wouldn't need to encode higher-level structure. Figure 20 (Appendix K.1) confirms this: CASnotM (no masking) achieves much lower prediction loss than both AP and CASM, but underperforms CASM on DM-HARD-8, particularly failing to solve Push Block. The higher loss of CASM indicates the task is genuinely harder — the model must learn to infer missing information from context, which requires building rich, structured representations that capture multi-step dependencies.

What makes this an extension of inverse dynamics. The core principle — filtering uncontrollable features by predicting actions — is preserved. The transformer must predict $a_t$ from $z_{t-1}$ and $z_t$ , where at least one of the corresponding input observations may be masked. To succeed, $f_\theta$ must extract features that are informative about actions and their effects — controllable aspects of the state — because those are the only features that help predict the masked information. Uncontrollable noise (like the randomly colored walls in Disco Maze, or the death timer in Pitfall!) provides no signal for action prediction and is suppressed in the learned representation.

Hyperparameters (Table 3). The transformer uses a GatedTransformerXL architecture with 2 layers, attention size 128, 4 attention heads, and a single MLP hidden layer of size 512. The action embedding size is 32. The state mask rate is 0.8 (80% of observation embeddings are masked). Four masked sequences are sampled per trajectory. The predictor network $g_\phi$ has a single hidden layer of size 128.

Summary of Design Choices and Their Justifications

Sum over all atoms within bandwidth rather than k-NN: Avoids the paradox where adding an observation to memory could decrease soft-visitation counts by displacing heavily-weighted atoms from the k-NN list. The monotonicity property (adding atoms only increases or preserves counts) aligns with the intuitive notion of familiarity.
Rational kernel with hard cutoff rather than Gaussian: The rational function $1/(1+x)$ has heavier tails for better generalization; the hard cutoff at $d_{\text{ema}}$ ensures sparsity for computational efficiency and prevents distant atoms from contributing noise.
Adaptive bandwidth via exponential moving average of k-NN distances rather than global average: Provides local scale calibration (different regions of the embedding space may have different densities) while the EMA balances responsiveness with stability ( $\tau = 0.9999$ , half-life ~7000 steps).
Stochastic insertion with probability $\eta$ rather than deterministic: Provides robustness to outliers — a novel embedding must be seen roughly $1/\eta$ times on average before insertion — while still allowing rapid expansion into genuinely novel regions.
Convex combination atom update rather than exact mean recomputation: The incremental running average is the posterior mean update for a Gaussian cluster model, computationally efficient, and the only feasible approach when past embeddings are not stored.
Count discounting with $\gamma < 1$ rather than undiscounted memory: Essential for handling non-stationary embedding distributions — atoms unreachable under the current representation lose count over time and are eventually replaced. Provides a smooth interpolation between short-term and long-term memory (Figure 10).
Count redistribution on removal rather than discarding: Preserves total visitation information in memory, making the algorithm less sensitive to hyperparameters. Merging counts into the nearest neighbor is a form of soft deletion that maintains approximate density estimates even as specific atoms are replaced.
Shared memory across actors rather than per-actor: Increases update frequency per atom, reducing effective representation drift between memory updates, and provides global exploration coordination without explicit communication.
$1/c_j^2$ removal probability rather than $1/c_j$ or deterministic minimum: Empirically produces more uniform cluster distributions after distribution stabilization and is more robust, especially on Pitfall! (Figures 8, 9). The square amplifies count differences, creating strong preference for removing truly underutilized atoms.
Using transformer inputs ( $f_\theta(o_t)$ ) rather than transformer outputs ( $z_t$ ) as RECODE embeddings: Preserves the Markov property — the exploration embedding depends only on the current observation, not on the trajectory context, avoiding information leakage and maintaining a clean separation between representation and exploration.

4. Key Insights and Innovations

Innovation 1: Decomposing Exploration into Independent Representation and Count Estimation Sub-Problems

The paper's most fundamental conceptual move is not any specific algorithm, but rather the explicit recognition that novelty-based exploration should be understood as two independent, disentangled sub-problems that prior work had consistently conflated: (i) learning a state representation that captures task-relevant similarity while filtering uncontrollable noise, and (ii) estimating how often regions of that representation space have been visited, over arbitrarily long horizons. The authors state this decomposition directly in Section 1 and structure their entire technical contribution — and experimental validation — around it.

Prior to this work, the dominant exploration methods in deep RL tended to entangle these two concerns. Parametric methods like RND (Burda et al., 2019) operate directly on raw observations, solving neither sub-problem explicitly — representation is absent (pixels are used as-is, with all their noise), and count estimation is implicit in a predictor network's error signal that suffers from slow adaptation and catastrophic forgetting. BYOL-Explore (Guo et al., 2022) solves the representation problem well (via bootstrapped latent prediction objectives that produce rich embeddings), but count estimation remains parametric and implicit — the novelty signal is still the error of a trained predictor network, inheriting the limitations of gradient-based adaptation. NGU (Badia et al., 2020b) comes closest to separating the concerns by using inverse dynamics embeddings (AP) for a short-term episodic memory while relying on RND for long-term novelty, but the separation is incomplete and asymmetric: the long-term component cannot use the learned representation at all (RND operates on raw pixels), so the two sub-problems remain coupled in practice through the architecture's dual-mechanism design.

What makes this decomposition intellectually distinctive is that it re-frames the exploration problem as an interface design question: given a good representation (from any source — AP, CASM, BYOL, or a future method), can you build a count estimator that (a) is fully non-parametric (no gradient updates, no forgetting), (b) spans thousands of episodes without resets, and (c) adapts to the representation's own evolution during training? This is a fundamentally different question from "what's the best exploration bonus?" — it separates the semantic question ("what makes two states similar?") from the statistical question ("how many times have I been here?"), and argues that they should be solved by different mechanisms optimized for different requirements.

The significance of this decomposition goes beyond RECODE itself. It implies that progress on either sub-problem independently translates to exploration gains — a better representation automatically improves any count estimator that can leverage it, and a better count estimator automatically benefits from richer representations. The paper demonstrates this directly in Figure 7: when switching from AP to CASM representations, both NGU and RECODE improve, but RECODE improves more because it can leverage the richer representation for both short-term and long-term novelty (NGU's RND component cannot). This modularity is not merely convenient — it changes the research landscape by allowing representation learning and count estimation to be studied and improved independently, with predictable compounding benefits when combined. This is a fundamental reframing, not an incremental refinement, because it provides a new organizing principle for the exploration literature rather than a point improvement on existing methods.

Innovation 2: A Unified, Single-Mechanism Long-Term Memory That Replaces the Dual-Mechanism Architecture

The specific technical insight that makes RECODE work is that clustering — aggregating similar observations into representative atoms — can compress an arbitrarily long exploration history into a fixed-size memory without discarding the long-term signal, thereby eliminating the need for separate short-term and long-term novelty estimators that had become standard in state-of-the-art exploration methods. This is best understood against the backdrop of NGU's architecture.

NGU's design reflects a genuine tension in non-parametric exploration: episodic memory provides responsive, representation-aware short-term novelty but forgets everything at episode boundaries; RND provides long-term memory but cannot use learned representations and adapts slowly. NGU's solution is to run both in parallel and sum their rewards — a hybrid that works empirically (it solved Montezuma's Revenge and other hard Atari games) but introduces complexity, requires careful tuning of two interacting novelty signals, and inherits the limitations of each component (the long-term signal remains vulnerable to observation noise because RND operates on raw pixels). The field might have reasonably concluded that this dual-mechanism complexity was necessary — that long-term memory inherently required parametric density estimation (since storing all past observations is infeasible), and parametric density estimation inherently couldn't leverage learned representations (since RND-style random target networks need fixed input distributions).

RECODE demonstrates that this conclusion was premature. The key insight is that aggregation into clusters serves double duty: it provides compression (a fixed-size memory can cover the state space by representing regions rather than individual observations) AND it provides density estimation (counts accumulated at cluster centers serve as soft visitation counts for the surrounding region). By adapting classical DP-means clustering (Kulis & Jordan, 2011) to the streaming, non-stationary RL setting — through count discounting, probabilistic atom replacement, and convex combination updates — RECODE achieves the seemingly contradictory goals of long memory horizon (atoms persist for thousands of episodes, as shown in Figure 3) and responsiveness (new observations immediately affect soft-visitation counts and can rapidly create new atoms in novel regions).

The empirical consequence is that a single mechanism replaces two. The ablation in Figure 4 makes this concrete: MEME-EMM-AP (episodic memory only) can solve Pitfall! but not Montezuma's Revenge (insufficient memory horizon); MEME-RND (long-term only, raw observations) can solve some games but not Pitfall! (cannot filter noise); MEME-RECODE-AP (unified clustering memory) solves both, matching or exceeding NGU on all eight hard exploration Atari games while being conceptually simpler. This is not merely a performance improvement — it's a simplification proof: it shows that the dual-mechanism architecture was an artifact of the specific limitations of prior non-parametric methods (episodic memory with no compression) and parametric methods (RND with no representation flexibility), not a fundamental requirement of exploration in complex environments.

The significance extends beyond Atari. On DM-HARD-8, RECODE's unified memory can leverage the powerful CASM representations for both short-term and long-term novelty, while NGU's architecture restricts representation benefits to the episodic component only — the RND long-term signal remains representation-agnostic. This explains why RECODE-CASM substantially outperforms NGU-CASM (Table 5) even when both use the same CASM representation: the unified memory translates representation improvements into exploration gains across the full temporal spectrum, while NGU's architecture leaves long-term novelty stuck with a suboptimal representation. This is a fundamental architectural insight, not an incremental improvement, because it changes what designers of exploration systems should build: one good memory, not two specialized ones.

Innovation 3: Verifier-Free Robustness to Observation Noise Through Representation-Mediated Exploration

The paper provides one of the clearest demonstrations that robustness to observation noise in exploration is primarily a representation problem, not a bonus mechanism problem — and that a count estimator operating on a noise-filtering representation can achieve near-immunity to noise that catastrophically degrades prior methods. This is a diagnostic insight with implications beyond RECODE: it suggests that the "noisy TV problem" (Schmidhuber, 2010) should be addressed at the representation level, not by engineering more sophisticated exploration bonuses that try to detect and ignore noise.

The experimental design in Section 5.3 is elegant in isolating this claim. The Noisy Montezuma experiment concatenates white noise (uniform in [0, 255]) to the original game frames, creating a stream of uncontrollable, constantly-changing pixels that are entirely irrelevant to the task. Both RECODE and NGU use the same action prediction (AP) representation backbone, which is specifically designed to suppress uncontrollable features — inverse dynamics training incentivizes the encoder to represent only what's necessary to predict actions, and noise that doesn't affect action outcomes gets filtered out. The difference is that RECODE computes its exploration bonus entirely within the AP embedding space (where noise has been suppressed), while NGU's long-term component (RND) operates on raw observations (where noise dominates the novelty signal).

The result is stark: MEME-NGU-AP's performance deteriorates significantly under noise (Figure 6), while MEME-RECODE-AP achieves similar scores to its noise-free performance, albeit with higher variance. The authors attribute NGU's failure to RND degenerating to "random exploration in the presence of uncontrollable noise" — every frame looks novel to RND regardless of what the agent does, so the intrinsic reward becomes essentially constant, providing no directional exploration signal. RECODE, by operating entirely on AP embeddings, sees past the noise: repeated visits to the same physical room produce similar embeddings (the noise has been filtered), counts accumulate, and the exploration bonus decays appropriately.

What makes this a genuine innovation rather than an obvious consequence is that it resolves a tension in prior work. NGU already recognized that good representations matter — its episodic memory uses AP embeddings precisely because they filter uncontrollable features. But NGU couldn't extend this insight to long-term novelty because RND (its only long-term mechanism) is architecturally incompatible with concurrently-trained representations. RECODE's contribution is showing that the same representation can serve both timescales if the count estimator is designed to operate on embeddings rather than observations. This is a negative result for RND-based approaches (they are fundamentally vulnerable to observation noise in a way that cannot be fixed without abandoning RND's core design) and a positive result for representation-mediated exploration (investments in better representations pay off in robustness, not just task performance).

The practical implication is significant: for any deployment where sensors may be imperfect — which is essentially all real-world applications — an exploration architecture that separates representation from counting (like RECODE) will be more robust than one that couples them (like NGU's RND component). The paper does not claim to have solved the noisy TV problem in general — the AP representation itself can be fooled if noise is action-correlated — but it provides a clear diagnostic framework for understanding when and why exploration methods fail under noise, and a recipe (put the noise filter in the representation, not the bonus) for addressing it.

Innovation 4: The Empirical Finding That Test-Time Memory Horizon Is a Tunable Parameter Controlling Exploration Depth

While the individual mechanisms of RECODE — clustering, discounting, insertion probability — are each incremental adaptations of existing techniques (DP-means, kernel density estimation, exponential moving averages), the paper's synthesis reveals a higher-level insight that is genuinely novel: the effective memory horizon of a non-parametric exploration module can be smoothly tuned via the discount factor $\gamma$ , converting what was previously a binary choice (episodic vs. lifetime) into a continuous spectrum. This is a conceptual contribution — it identifies a new axis of variation in exploration algorithm design and provides empirical characterization of its effects.

Prior non-parametric methods were designed at two extremes. Episodic memory approaches (Pritzel et al., 2017; Badia et al., 2020b) have zero memory across episodes — they reset completely at episode boundaries, providing purely short-term exploration that cannot track progress across the many episodes needed to solve hard exploration games. Full lifetime memory (storing all past observations) is theoretically ideal but computationally infeasible. The field had implicitly treated this as a binary constraint: either you have memory across episodes (which requires parametric methods like RND, with all their limitations) or you don't (which limits you to within-episode exploration).

RECODE's count discounting breaks this binary into a continuous spectrum. The toy experiment in Figure 10 (Appendix C.2) makes this visually concrete: with low $\gamma$ (0.9, 0.99), RECODE's atoms track only the current distribution (short-term memory, like episodic approaches); with high $\gamma$ (0.999, 0.9999), atoms retain historical density information (long-term memory, approaching the ideal of storing all past data). The key insight is that $\gamma$ is a tunable parameter — not a binary switch — that controls the rate at which old experience is forgotten. The paper's main experiments set $\gamma = 0.999$ for Atari and $\gamma = 0.997$ for DM-HARD-8, but these are choices on a spectrum, not fixed constants dictated by algorithm design.

The age distribution in Figure 3 provides the empirical evidence that this spectrum is meaningful in practice: a RECODE agent solving Pitfall! maintains atoms spanning from very recent (within the current episode) to thousands of episodes old. The memory is not just "long-term" in the sense of crossing episode boundaries — it is multi-scale, simultaneously tracking novelty at within-episode and cross-episode timescales within a single unified structure. This is fundamentally different from NGU's approach of running two separate mechanisms at different timescales — RECODE achieves multi-scale memory through a single parameter that controls the decay rate, with different atoms naturally occupying different positions on the age spectrum depending on whether they continue to receive assignments.

Why does this matter beyond RECODE? It establishes that memory horizon is a design dimension that future exploration methods should explicitly optimize, not an incidental property of the chosen architecture. For different environments, different timescales of memory may be optimal — fast-changing environments may benefit from shorter horizons (lower $\gamma$ ), while environments with very sparse rewards and long exploration chains may need longer horizons (higher $\gamma$ ). The paper's finding that $\gamma = 0.999$ works well for Atari while $\gamma = 0.997$ works better for DM-HARD-8 (a 3× difference in the effective horizon, since $0.999^{1000} \approx 0.37$ while $0.997^{1000} \approx 0.05$ ) suggests this tuning is practically consequential. The ablation in Figure 11 confirms that extreme values break down: $\gamma = 1$ (no discounting, infinite memory) degrades performance because stale representations accumulate. This is a new empirical finding — the optimal memory horizon for exploration is finite and environment-dependent — that had not been systematically characterized before.

This innovation is incremental in its mechanism (discounting is a standard technique) but fundamental in its implication: it redefines memory management in exploration as a temporal credit assignment problem — how far back should novelty signals propagate? — rather than a storage capacity problem. Future work on exploration memory can build on this framing, treating the discount factor as a hyperparameter to be tuned or even learned, rather than accepting the episodic/lifetime binary that constrained prior designs.

5. Experimental Analysis

Evaluation Methodology

Dataset. The experiments use two established benchmarks: (1) the Atari Learning Environment (ALE; Bellemare et al., 2013), specifically the subset of eight hard exploration games identified by Bellemare et al. (2016) — including Montezuma's Revenge, Pitfall!, and Private Eye — with episodes lasting up to 27,000 steps under standard action repeat of 4; and (2) DM-HARD-8 (Gulcehre et al., 2019), a suite of eight procedurally-generated 3D tasks with partial observability, continuous control, sparse rewards, and highly variable initial conditions where the agent must interact with specific objects to reach a large apple providing reward (example tasks: Baseball, Push Blocks, Remember Sensor; see Figure 16 for a visual example of the Baseball task).
Base model(s). All experiments use the MEME agent (Kapturowski et al., 2022) as the core RL learner — a distributed architecture built on IMPALA (Espeholt et al., 2018) that represents the current state-of-the-art on Atari — with agent configurations denoted as AGENT-EXPLORATION-EMBEDDING (e.g., MEME-NGU-AP uses the MEME agent with NGU exploration and action-prediction embeddings); for DM-HARD-8, an additional comparison uses a VMPO-based agent (Guo et al., 2022) for the BYOL-Explore baseline.
Metrics. The primary metric is game score (raw environment return), reported as the mean over multiple seeds; for aggregated results across environments, human-normalized score (Mnih et al., 2015) is computed as (agent_score - random_score) / (human_score - random_score) × 100%, with scores capped at 100% prior to averaging to prevent any single environment from dominating due to larger reward scales (Section I, Figure 17); the Atari results follow the standard evaluation regime with 30 random no-ops at episode start and no sticky actions (Machado et al., 2018).
Baselines. The experiments compare RECODE against several exploration methods, all using the same MEME base agent and (where applicable) the same representation learning mechanism: (1) MEME-NGU-AP — the original MEME agent from Kapturowski et al. (2022) using NGU (Badia et al., 2020b) with action-prediction embeddings, representing the prior state-of-the-art; (2) MEME-EMM-AP — episodic memory only (Pritzel et al., 2017) with AP embeddings, to isolate the contribution of short-term vs. long-term memory; (3) MEME-RND — Random Network Distillation (Burda et al., 2019) operating on raw observations, to isolate the limitation of parametric long-term novelty without representation learning; (4) MEME-RNDonAP — RND applied on top of concurrently-trained AP embeddings (Appendix K.2), to test whether RND can be adapted to use learned representations; (5) MEME-NGU-CASM — NGU combined with the CASM representation (used in DM-HARD-8 experiments), to isolate whether representation improvements benefit NGU as much as RECODE; (6) VMPO-BYOL-Explore — the BYOL-Explore agent from Guo et al. (2022) using VMPO, representing the prior state-of-the-art on DM-HARD-8.
Generation budget / compute accounting. The paper does not use a "generation budget" in the style of LLM test-time compute papers — RECODE is evaluated as an integrated exploration mechanism within a full RL training pipeline. All agents are trained for a fixed number of environment frames: Atari results are reported over the course of training, with performance curves shown as a function of environment steps (Figures 4, 6, 9); DM-HARD-8 results in Figure 5 are reported after $1 \times 10^{10}$ environment frames. The memory sizes ( $5 \times 10^4$ atoms for Atari, $2 \times 10^5$ for DM-HARD-8) and discount rates ( $\gamma = 0.999$ for Atari, $\gamma = 0.997$ for DM-HARD-8) are fixed per domain and not varied as part of any scaling analysis — the paper is primarily a methods comparison at fixed computational scale, not a scaling-law study.
Cross-validation / statistical protocol. Atari results are averaged over six seeds; DM-HARD-8 results are averaged over three seeds. All performance curves include shaded regions indicating variance across seeds (visible in Figures 4, 5, 6, 7). The paper does not report confidence intervals or formal statistical tests. No cross-validation is used for strategy selection (unlike the compute-optimal test-time scaling paper) — RECODE's hyperparameters ( $\gamma, \kappa, \eta, \tau, k$ , memory size) are fixed across all environments within each domain and reported in Tables 1 and 2 without indication that they were tuned per-game. The aggregation to human-normalized capped scores (Section I) ensures no single game dominates the reported means.

Main Quantitative Results

Atari Hard Exploration Games

Headline result. RECODE achieves state-of-the-art, super-human performance across all eight hard exploration Atari games while using a single, conceptually simpler exploration bonus compared to NGU's dual-mechanism design (Figure 4). MEME-RECODE-AP matches or exceeds MEME-NGU-AP on every game and is the first agent to reach the end screen in Pitfall! — a game that requires tracking the agent's position in a large, scrolling world with sparse rewards and an in-game death timer that limits episode length.

Comparison against baselines (Figure 4). The results show that no single prior exploration mechanism suffices for all eight games, while RECODE's unified memory handles the full set:

MEME-EMM-AP (episodic memory only) achieves strong performance on games where within-episode memory is sufficient — notably Pitfall!, where it reaches scores comparable to NGU and RECODE — but fails completely on Montezuma's Revenge, which requires long-term memory across episodes to track exploration progress through the game's multiple rooms and levels. This is the "short memory" failure mode: EMM resets every episode and cannot propagate novelty signals across the hundreds of episodes needed to solve Montezuma's Revenge.
MEME-RND (long-term parametric novelty on raw observations) solves several games — including Montezuma's Revenge — demonstrating that long-term novelty alone is sufficient for some environments, but fails on Pitfall! because RND operates on raw observations and cannot filter out the uncontrollable visual noise (the death timer, scrolling background changes) that dominate the novelty signal. The paper states: "RND on its own cannot solve Pitfall! because of the presence of many uncontrollable features in the observations and its inability to leverage the AP embeddings." This is the "noise vulnerability" failure mode.
MEME-NGU-AP combines EMM (for short-term, representation-aware novelty) and RND (for long-term, raw-observation novelty), achieving strong performance on both Montezuma's Revenge and Pitfall!, but does so with a more complex dual-mechanism architecture and inherited vulnerability to observation noise (as demonstrated in Section 5.3).
MEME-RECODE-AP solves all eight games with super-human performance, matching NGU on Montezuma's Revenge and Pitfall! while being "conceptually simpler" — a single clustering memory replaces the EMM+RND hybrid. The paper explicitly frames this: "In contrast, RECODE is able to leverage the AP representation for short-term and long-term novelty due to the clustering-based memory integrating over a long horizon which enables solving both games with a single intrinsic reward."

Aggregated results (Figure 17). Averaging human-normalized scores across all eight games (capped at 100% to prevent Q*bert's high-variance exploit from dominating), MEME-RECODE-AP matches or slightly exceeds MEME-NGU-AP in mean performance. The uncapped mean (Figure 17, left) shows substantial variance in both methods due to Q*bert's scoring bug, which can produce "significantly higher scores if exploited" (Chrabaszcz et al., 2018); the capped mean (Figure 17, right) provides a more stable comparison and confirms RECODE's parity with NGU despite using one novelty mechanism rather than two.

What this demonstrates about the decomposition. By keeping the base agent (MEME) and representation (AP) fixed while varying only the exploration bonus, the Atari experiments isolate the count estimation sub-problem and demonstrate that RECODE's unified long-term clustering memory can replace both the short-term (EMM) and long-term (RND) components of NGU without loss of performance. The critical result is not that RECODE outperforms NGU (they are roughly at parity on most games), but rather that the dual-mechanism architecture was unnecessary — a single, well-designed non-parametric memory can cover both timescales if it compresses observations through clustering and handles non-stationarity through discounting. The EMM and RND ablations serve as existence proofs that separate short-term and long-term mechanisms are individually insufficient (each fails on exactly the games the other solves), making RECODE's unification non-trivial.

DM-HARD-8 3D Exploration Tasks

Headline result. RECODE combined with the CASM representation sets a new state-of-the-art on DM-HARD-8 (Figure 5, Table 5), solving six out of eight tasks with super-human performance and marking the first time the human baseline has been beaten on Push Blocks. This substantially exceeds NGU-CASM (which uses the same representation but NGU's dual-mechanism bonus) and BYOL-Explore (the prior state-of-the-art from Guo et al., 2022), demonstrating that RECODE's unified memory can leverage rich multi-step representations more effectively than prior exploration methods.

Per-task results (Table 5). The final performance after $1 \times 10^{10}$ environment frames shows:

Game	Human	NGU-AP	NGU-CASM	RECODE-AP	RECODE-CASM
Baseball	7.90	0.00 ± 0.00	0.00 ± 0.00	10.00 ± 0.00	10.00 ± 0.00
Drawbridge	12.30	0.00 ± 0.00	4.15 ± 6.62	12.41 ± 0.07	12.86 ± 0.13
Navigate Cubes	7.80	3.33 ± 5.33	9.76 ± 0.25	10.00 ± 0.00	10.00 ± 0.00
Push Blocks	8.40	0.00 ± 0.00	0.24 ± 0.24	1.07 ± 0.62	4.06 ± 2.14
Remember Sensor	7.60	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00
Throw Across	5.70	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00	9.72 ± 0.31
Wall Sensor	9.10	0.00 ± 0.00	0.00 ± 0.00	10.00 ± 0.00	10.00 ± 0.00
Wall Sensor Stack	8.60	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00

Key observations from the per-task breakdown:

RECODE-CASM is the only method to solve Throw Across (9.72) and Baseball (10.00) among the tested configurations. NGU (both AP and CASM variants) scores zero on both tasks, indicating that these environments require both rich representations AND a memory that can leverage them at long timescales.
Push Blocks remains challenging for all methods — RECODE-CASM achieves 4.06, surpassing the human baseline of 8.40 for the first time (though the paper reports this as "super-human" — the number 4.06 is below human 8.40, suggesting the claim refers to RECODE being the first agent to beat human performance at all, or the human baseline may have been beaten in some metric not captured in this table). The high variance (±2.14) indicates that solving Push Blocks remains unstable across seeds.
Remember Sensor and Wall Sensor Stack are unsolved by all methods — scores are exactly 0.00 for every agent, including RECODE-CASM. The paper does not discuss these failures in detail, but they represent clear boundary conditions where even the combined RECODE+CASM approach cannot make progress.
MEME-NGU-AP fails on all eight tasks (scores of 0.00 on seven of eight, with 3.33 ± 5.33 on Navigate Cubes). This is a striking degradation from NGU's Atari performance and demonstrates that NGU's default configuration (AP embeddings, dual-mechanism bonus) is fundamentally insufficient for the 3D partially-observable domain — the representations and memory mechanisms that work on Atari do not transfer.

The representation-bonus interaction (Figure 5 and Figure 7). The comparison between NGU-CASM and RECODE-CASM isolates the effect of the exploration bonus when both methods use the same CASM representation:

NGU-CASM improves substantially over NGU-AP (solving Navigate Cubes at 9.76, and getting non-zero scores on Drawbridge and Push Blocks), demonstrating that CASM provides richer representations that benefit exploration. However, NGU-CASM still scores zero on Baseball, Throw Across, Remember Sensor, Wall Sensor, and Wall Sensor Stack — the improved representation alone is insufficient.
RECODE-CASM solves four additional tasks beyond NGU-CASM (Baseball, Throw Across, Wall Sensor) and substantially improves on Push Blocks (4.06 vs. 0.24). The paper attributes this to RECODE's ability to "take full advantage of the representational power afforded by CASM as it is able to leverage it for both short-term and long-term novelty bonuses" (Section 5.2).
BYOL-Explore (Figure 5, shown as horizontal lines representing final performance) solves a subset of tasks but is substantially outperformed by RECODE-CASM on the aggregate — the mean capped human-normalized score curves in Figure 18 show RECODE-CASM reaching approximately 80% vs. BYOL-Explore at approximately 55% after $1 \times 10^{10}$ frames.

What this demonstrates about the decomposition. The DM-HARD-8 experiments directly test the paper's central claim that RECODE's decomposition allows it to "directly translate improvements in representation learning to improvements in exploration performance." The evidence supports this: when the representation improves (CASM vs. AP), RECODE-CASM improves dramatically over RECODE-AP (solving several additional tasks), while NGU-CASM improves over NGU-AP but by less (still failing on most tasks). The interpretation is that RECODE's unified memory amplifies representation gains because both short-term and long-term novelty benefit; NGU's dual mechanism only applies the representation improvement to the episodic component, leaving the RND-based long-term component unchanged and limiting the aggregate gain.

Aggregated results (Figure 18 and Table 5 bottom row). The mean capped human-normalized score across all eight tasks (Figure 18) places RECODE-CASM as the clear best method, with RECODE-AP and BYOL-Explore at a lower tier, and NGU variants near zero. The learning curves show RECODE-CASM making steady progress over the full $1 \times 10^{10}$ frames, while BYOL-Explore plateaus earlier. This is one of the paper's strongest results — it demonstrates that the combination of rich representations (CASM) and a flexible non-parametric memory (RECODE) can solve exploration challenges that neither component achieves alone, and that the decomposition into independent sub-problems is practically consequential rather than merely taxonomic.

Robustness to Observation Noise

Headline result. When white noise is added to Montezuma's Revenge observations, MEME-NGU-AP's performance deteriorates significantly while MEME-RECODE-AP achieves similar scores to the noise-free setting, demonstrating that RECODE's representation-mediated exploration bonus is substantially more robust to uncontrollable observation features (Figure 6).

Experimental setup. The Noisy Montezuma experiment (Section 5.3) concatenates a frame of white noise in the range [0, 255] to the game's original 210 × 160 grayscale observations along the height dimension. Both RECODE and NGU use the same AP backbone, which is trained via inverse dynamics and should filter out the noise (since noise does not help predict actions). The difference is that RECODE computes its exploration bonus entirely within the AP embedding space (where the noise has been suppressed), while NGU's long-term RND component operates on the raw (noisy) observations and cannot distinguish task-relevant novelty from noise-driven pixel changes.

Results (Figure 6). The learning curves show:

MEME-RECODE-AP in the noisy setting achieves a mean score comparable to its noise-free performance (Figure 4), reaching approximately the same asymptotic level on Montezuma's Revenge — though with "higher variance" across seeds, as noted in the text. This confirms that the AP representation successfully filters the noise to a degree that RECODE's count estimation remains functional.
MEME-NGU-AP in the noisy setting shows significant degradation. The paper attributes this to RND degenerating to "random exploration in the presence of uncontrollable noise" (citing Kapturowski et al., 2018) — every frame looks novel to the RND predictor regardless of what the agent does, so the intrinsic reward becomes essentially constant, providing no directional signal. This effectively restricts NGU to its episodic memory component only, which cannot solve Montezuma's Revenge (as the EMM ablation in Figure 4 already demonstrated).

What this demonstrates. This experiment directly tests the paper's claim that "RECODE's performance also remains unaffected by noisy observations — an adversarial condition which significantly degrades prior approaches such as RND and NGU" (Section 6). The evidence supports this claim specifically for the AP+RND vs. AP+RECODE comparison under the specific noise condition tested (white noise concatenated to frames). This is a diagnosis, not a solution — it shows that representation-mediated exploration is the right architectural choice for robustness, not that RECODE has any special noise-handling mechanism beyond operating on learned embeddings.

Limitation. The experiment tests only one type of noise (additive white noise) in one environment (Montezuma's Revenge). It does not demonstrate robustness to noise that is action-correlated (which could potentially "trick" the AP representation into encoding it, since action-correlated noise would help predict actions), or noise that corrupts the observation in ways the representation cannot filter. The "higher variance" noted for RECODE suggests the robustness is not perfect — the memory may still be affected by residual noise in the embedding space, or the noise may interfere with the representation learning itself.

Representation Comparisons: AP vs. CASM

Headline result. CASM consistently outperforms AP as a representation for exploration on DM-HARD-8, and RECODE benefits more from the switch than NGU does, confirming that the unified memory better leverages improvements in representation quality (Figure 7, Table 5).

The CASM vs. AP comparison (Figure 7). The capped human-normalized scores aggregated across all eight DM-HARD-8 tasks show:

RECODE-CASM > RECODE-AP (approximately 80% vs. 55% final performance)
NGU-CASM > NGU-AP (approximately 25% vs. 5% final performance)
The gap between RECODE and NGU widens with the better representation: at AP, the gap is approximately 50 percentage points; at CASM, the gap is approximately 55 percentage points.

This widening gap is the key empirical finding: it suggests that RECODE's unified memory architecture amplifies representation gains, while NGU's dual-mechanism architecture partially wastes them (since the RND long-term component cannot use the learned representation at all). The paper explicitly draws this conclusion: "RECODE outperforms NGU for both representations, indicating that leveraging the representational power for both short-term and long-term novelty signals is a key benefit of our proposed method."

Per-task perspective. Looking at Table 5, the representation switch from AP to CASM has different effects depending on the exploration bonus:

For RECODE, CASM enables solving Throw Across (0.00 → 9.72) and substantially improves Push Blocks (1.07 → 4.06), while maintaining already-solved performance on Baseball, Drawbridge, Navigate Cubes, and Wall Sensor.
For NGU, CASM enables non-zero scores on Drawbridge (0.00 → 4.15), Navigate Cubes (3.33 → 9.76), and Push Blocks (0.00 → 0.24), but five of eight tasks remain at zero. The representation improvement helps but is insufficient to overcome the fundamental limitation of having a long-term novelty component (RND) that cannot use the representation at all.

CASM masking ablation (Appendix K.1, Figure 20). An additional experiment removes the masking from CASM (providing both observations and actions at every timestep — "CASnotM"), finding that the unmasked variant achieves much lower prediction loss than both AP and CASM — suggesting the task is trivially easy without masking — but underperforms CASM on DM-HARD-8 and "is essential to solve Push Block." This confirms that the masking is critical for learning rich representations: without it, the transformer can simply attend to the current observation to predict the action, avoiding the need to build representations that capture multi-step structure.

RECODE with Alternative Representations

RND on AP embeddings (Appendix K.2, Figure 21). To test whether RND's failure on Pitfall! is fundamental to RND or merely a consequence of operating on raw observations, the paper evaluates "RNDonAP" — RND applied to concurrently-trained AP embeddings rather than raw pixels. The result is negative: "the resulting agent is unable to solve some of the hardest exploration games such as Montezuma's Revenge or Pitfall!" (Figure 21). The paper hypothesizes that RND's prediction error conflates two different signals: genuine state novelty AND representation drift. As the AP embedding function evolves during training, even previously-visited states may produce different embeddings, causing RND to flag them as novel — a failure mode that RECODE's non-parametric memory avoids because it doesn't rely on a parametric predictor that must be trained to track the moving target of the embedding distribution. This is an important negative result: it shows that simply giving RND a better input representation does not solve the fundamental issue; the parametric approach is inherently limited when the representation is evolving.

RECODE on BYOL embeddings (Appendix K.3, Figure 22). To demonstrate RECODE's flexibility with respect to the representation source, the paper evaluates RECODE using BYOL embeddings (from Guo et al., 2022) instead of AP embeddings on Montezuma's Revenge and Pitfall!. The results show that "RECODE is able to leverage BYOL embeddings to achieve superhuman performance on Montezuma's Revenge and achieve positive scores on Pitfall!, though significantly underperforming compared to AP embeddings." This is a mixed result: it confirms that RECODE can work with representations beyond AP and CASM (supporting the modularity claim), but the substantial performance gap relative to AP suggests that the choice of representation matters significantly for final performance — RECODE doesn't magically compensate for a poor representation. The fact that BYOL embeddings (which are trained via bootstrapped latent prediction rather than inverse dynamics) underperform AP on Pitfall! is consistent with the importance of filtering uncontrollable features: inverse dynamics explicitly suppresses them, while BYOL's objective may preserve them.

Ablation Studies and Robustness Checks

Atom removal strategy (Appendix C.2, Figures 8 and 9): Three strategies are compared — stochastic removal with probability $\propto 1/c_j^2$ , stochastic removal with $\propto 1/c_j$ , and deterministic removal of the smallest-count atom. The $1/c_j^2$ strategy is most robust, particularly on Pitfall!, though "all choices of removal strategy considered result in a viable algorithm." A toy experiment with a non-stationary 2D distribution (Figure 8) shows that $1/c_j^2$ produces approximately uniform cluster centers and counts after the distribution stabilizes, while deterministic smallest-count removal leaves cluster centers skewed. The square in the denominator amplifies count differences, creating a stronger preference for removing truly underutilized atoms while making heavily-visited atoms essentially permanent until their counts decay.
Count discounting $\gamma$ (Appendix C.2, Figures 10-12, 14): Removing discount ( $\gamma = 1$ , no forgetting) causes significant performance degradation on Montezuma's Revenge and Pitfall! (Figure 11), confirming that the ability to forget stale representations is essential when embeddings are trained concurrently. When embeddings are pretrained and frozen (Figure 12), RECODE performs well even with $\gamma = 1$ , demonstrating that the discount is specifically needed to handle the co-evolution of the embedding function and memory — it compensates for representation drift, not for changes in the policy's state distribution. A toy experiment (Figure 10) shows that $\gamma$ smoothly interpolates between short-term memory (low $\gamma$ , tracking only the current distribution) and long-term memory (high $\gamma$ , retaining historical density), with intermediate values producing multi-scale memory.
Shared vs. per-actor memory (Section 3): RECODE shares its memory across all actors in the distributed setup, unlike NGU where each actor maintains its own episodic memory. This increases update frequency per atom by a factor of $N$ (the number of actors), reducing effective representation drift between memory updates. The paper does not provide an ablation comparing shared vs. per-actor RECODE memory, so the quantitative benefit of sharing is not isolated from other design choices. This is a limitation — the claim that sharing "greatly increases the frequency of updates" and reduces drift is plausible but unquantified.
CASM masking rate (Appendix K.1, Figure 20): Removing masking entirely ("CASnotM") leads to much lower prediction loss but significantly worse DM-HARD-8 performance, demonstrating that masking is essential to force the representation to encode useful multi-step structure rather than solving the prediction task through trivial short-term correlations. The specific mask rate (0.8 for states) and number of masks per trajectory (4) are not ablated — the paper does not explore sensitivity to these hyperparameters.
RND on AP embeddings (Appendix K.2, Figure 21): As discussed above, this is a negative result showing RND cannot be salvaged by simply switching its input from raw pixels to AP embeddings. The failure is attributed to RND conflating representation drift with genuine state novelty.
RECODE on BYOL embeddings (Appendix K.3, Figure 22): RECODE works with BYOL embeddings but substantially underperforms AP on Pitfall!, showing that RECODE is representation-agnostic but not representation-invariant — the quality and nature of the representation matter greatly.

Critical Assessment

Central Claim: RECODE Unifies Short-Term and Long-Term Novelty in a Single Mechanism

Does the evidence demonstrate this? Yes, with qualifications. The Atari experiments (Figure 4) convincingly show that RECODE solves the full set of eight hard exploration games using a single intrinsic reward, while the individual components of NGU (EMM alone, RND alone) each fail on different subsets. This is strong evidence that RECODE's clustering memory covers both short-term and long-term novelty. However, the comparison is between RECODE and the components of NGU, not between RECODE and a hypothetical method that also attempts a unified memory but with a different design. The claim that the dual-mechanism architecture was "unnecessary" is supported, but only insofar as RECODE matches (does not dramatically exceed) NGU on Atari — the primary advantage is simplicity, not performance. On DM-HARD-8, RECODE substantially outperforms NGU, but here the representation also changes (CASM vs. AP), so the gain cannot be attributed solely to the unified memory.

What's missing. The paper does not compare RECODE against a variant of NGU where the RND component is replaced with a different long-term parametric method (e.g., a density model trained on AP embeddings), which would test whether the non-parametric clustering specifically matters or whether any long-term mechanism that can use learned representations would suffice. The RNDonAP ablation partially addresses this (showing that RND fails even with AP embeddings), but a more systematic comparison against parametric alternatives would strengthen the case for clustering specifically.

Central Claim: RECODE Is Robust to Observation Noise

Does the evidence demonstrate this? Yes, for the specific noise condition tested. The Noisy Montezuma experiment (Figure 6) shows a clear differential effect: NGU degrades, RECODE does not. The mechanism is well-understood (RECODE operates on noise-filtered embeddings; NGU's RND operates on raw observations) and the result is consistent with the paper's theoretical framing. However, the claim in the abstract and conclusion that "RECODE's performance also remains unaffected by noisy observations" is somewhat overstated — the result shows robustness to one type of noise in one environment, with "higher variance" acknowledged. "Remains unaffected" suggests invariance; "degrades less than NGU" would be more precise.

What's missing. The paper does not test noise that is action-correlated (e.g., sensor noise that varies systematically with the agent's movements), which could potentially leak into the AP representation since it would help predict actions. It also does not test noise that corrupts the observation in ways the AP encoder cannot filter (e.g., noise that occludes task-relevant features), where RECODE would presumably also fail since it depends on the representation quality. A more comprehensive noise study — varying noise type, magnitude, and correlation structure — would substantially strengthen the robustness claim.

Central Claim: RECODE Directly Translates Representation Improvements to Exploration Gains

Does the evidence demonstrate this? Yes, with clear evidence. The comparison across AP/CASM representations for both RECODE and NGU (Figure 7, Table 5) shows that RECODE benefits more from the representation upgrade than NGU does — the performance gap widens rather than remaining constant. This is consistent with the claimed mechanism: RECODE's unified memory applies the better representation to all novelty estimation, while NGU's RND component remains stuck with its raw-observation limitation regardless of the representation used elsewhere. The BYOL embedding experiment (Figure 22) provides additional support — RECODE can leverage alternative representations, though performance varies with representation quality.

What's missing. The paper does not test whether RECODE's performance scales monotonically with representation quality or whether there are diminishing returns. Testing with representations of systematically varying quality (e.g., AP trained for different numbers of steps, or with different capacity encoders) would characterize this relationship more precisely. Additionally, the claim that RECODE "directly translates" improvements implies near-linear scaling, which is not tested.

Central Claim: RECODE Sets New State-of-the-Art on Atari and DM-HARD-8

Does the evidence demonstrate this? On DM-HARD-8, yes: RECODE-CASM substantially exceeds BYOL-Explore, the prior state-of-the-art from Guo et al. (2022), and NGU-CASM. On Atari, the claim is more nuanced — RECODE matches NGU, which was already state-of-the-art via MEME (Kapturowski et al., 2022). The meaningful advance is that RECODE achieves this with a simpler mechanism (one bonus vs. two) and with robustness to noise that NGU lacks. Whether this constitutes "new state-of-the-art" depends on whether one values simplicity and robustness as dimensions of the state-of-the-art, or only raw performance. The paper is transparent about the performance parity on Atari — Figure 17 shows RECODE and NGU at roughly equal capped human-normalized scores.

Genuine Weaknesses and Missing Experiments

Single base agent architecture (MEME). All Atari experiments use MEME, and RECODE's performance relative to NGU may depend on MEME's specific distributed training setup, replay scheme, and hyperparameters. The VMPO-based experiment in Appendix J partially addresses this (showing RECODE works in a policy-gradient setting on DM-HARD-8 multi-task), but a broader test across different base RL algorithms would strengthen generality claims.

No sensitivity analysis for key hyperparameters. The paper reports fixed values for $\gamma$ , $\kappa$ , $\eta$ , $\tau$ , $k$ , and memory size (Tables 1, 2) but does not ablate sensitivity to most of these. The $\gamma$ and removal strategy ablations are exceptions. How sensitive is performance to $\kappa$ (the insertion tolerance)? To $\eta$ (the insertion probability)? To $k$ (the number of neighbors for bandwidth estimation)? To memory size? Without these, it's unclear whether RECODE's hyperparameters require careful per-domain tuning (which would undermine the "simple drop-in" claim) or are robust across a wide range.

No analysis of computational cost. RECODE introduces a shared memory that must be queried and updated at every timestep by all actors. The paper does not report the computational overhead of nearest-neighbor searches, atom insertion/removal, or count discounting relative to the base RL training. NGU's episodic memory also requires k-NN queries, so the per-step cost may be comparable, but the shared memory architecture adds communication overhead not present in NGU's per-actor episodic memories. Without this analysis, the practical deployability of RECODE in resource-constrained settings is unclear.

Failure modes are under-analyzed. Two DM-HARD-8 tasks (Remember Sensor, Wall Sensor Stack) are unsolved by all methods including RECODE-CASM (scores of 0.00). The paper does not discuss why these tasks resist all exploration methods or what specific limitation of the current approach they reveal. Understanding these failures is as informative as the successes — do they require even longer memory horizons? Different representations? Different exploration objectives entirely? The silence on these failures limits the diagnostic value of the experimental results.

The memory size is not systematically scaled. RECODE uses $5 \times 10^4$ atoms for Atari and $2 \times 10^5$ for DM-HARD-8. Are these near the minimum needed? Would larger memories improve performance further? Would smaller memories work? The paper does not ablate memory size, leaving open the question of whether RECODE's benefit over episodic memory comes from the clustering mechanism or simply from having a larger effective capacity (since NGU's episodic memory stores at most one episode's worth of observations, bounded by ~27,000 in Atari, while RECODE's $5 \times 10^4$ atoms can cover much more history through aggregation).

No direct measurement of "representation drift" or its mitigation. The paper claims that discounting addresses representation drift (Section 3, Appendix C.2) and that shared memory reduces effective drift (Section 3), but never quantifies how much the embedding function actually changes over the course of training or how much the discount/insertion mechanisms compensate. A direct measurement — e.g., tracking the average distance between embeddings of the same physical state at different training points — would strengthen the mechanistic explanation substantially.

The multi-task DM-HARD-8 experiment (Appendix J) is incomplete. RECODE with VMPO and AP embeddings is competitive with BYOL-Explore but does not beat it on all tasks. The paper notes that "improving this performance using better-suited representations, such as CASM, is left for future work." This is an honest acknowledgment, but it means the multi-task results do not benefit from the full RECODE+CASM combination that drives the single-task state-of-the-art. The multi-task setting is arguably more realistic (one agent, many tasks), so this gap is significant.

Generalization beyond the tested domains is untested. All results are on Atari and DM-HARD-8. The paper does not test on continuous control benchmarks (e.g., DeepMind Control Suite), robotics tasks, or environments where exploration requires physical interaction (e.g., manipulation with sparse rewards). The claim that RECODE is a "simple yet robust drop-in exploration method" (Section 6) is supported only for the specific visual domains tested.

Summary: What the Experiments Do and Don't Show

What they show convincingly:

A single non-parametric clustering memory can replace the dual-mechanism EMM+RND architecture for hard exploration Atari games without performance loss.
RECODE's representation-mediated design provides substantially better robustness to observation noise than NGU's hybrid design, because noise filtering happens at the representation level rather than being left to a parametric component operating on raw observations.
On 3D partially-observable tasks (DM-HARD-8), the combination of rich multi-step representations (CASM) and a flexible non-parametric memory (RECODE) substantially exceeds prior methods, and RECODE benefits more from representation improvements than NGU does.
The discount factor $\gamma$ provides a tunable memory horizon that can interpolate between short-term and long-term memory, and discounting is essential when representations are trained concurrently.

What they don't show (or show weakly):

That RECODE outperforms NGU on Atari (they are at parity), beyond the simplicity and noise-robustness advantages.
That RECODE is robust to noise types beyond additive white noise on Montezuma's Revenge.
That RECODE's hyperparameters are robust across environments without tuning (only two domains tested, no sensitivity analysis for most parameters).
That RECODE's computational overhead is acceptable relative to the gains (no cost analysis).
That RECODE generalizes beyond the specific visual domains tested (Atari, DM-HARD-8).
Why some tasks remain unsolved (e.g., Remember Sensor, Wall Sensor Stack) despite the combined RECODE+CASM approach.

6. Limitations and Trade-offs

Memory Size and Hyperparameter Sensitivity Are Not Characterized

The assumption or constraint. RECODE uses fixed memory sizes ( $5 \times 10^4$ atoms for Atari, $2 \times 10^5$ for DM-HARD-8) and fixed values for most hyperparameters ( $\kappa = 0.2$ , $\eta$ at 0.05 or 0.2, $\tau = 0.9999$ , $k = 20$ ) that are reported in Tables 1 and 2 without ablation across different settings. The paper does not provide any sensitivity analysis for memory size, the insertion tolerance $\kappa$ , the insertion probability $\eta$ , the number of bandwidth neighbors $k$ , or the bandwidth decay rate $\tau$ — only the discount factor $\gamma$ and the atom removal strategy are studied in ablation (Appendix C.2, Figures 8–12). The implicit assumption is that these values are either near-optimal for the tested domains or that performance is insensitive to their specific settings.

The consequence. A practitioner applying RECODE to a new domain has no guidance on how to set these parameters. Memory size is particularly consequential: if set too small, the clustering may not cover the state space adequately, limiting exploration; if set too large, the per-step computational cost of nearest-neighbor search over $|M|$ atoms grows. The insertion probability $\eta$ controls the rate at which novel regions are incorporated into memory — too low and genuinely novel states may be ignored for too long, slowing exploration; too high and memory fills with one-off outliers, reducing effective coverage. The insertion tolerance $\kappa$ determines the granularity of clustering — small $\kappa$ creates many fine-grained clusters (high memory resolution but faster turnover), large $\kappa$ creates fewer coarse clusters (better compression but worse discrimination). Without sensitivity analysis, it is unknown whether choosing these parameters requires careful per-domain tuning (undermining the "simple drop-in" claim) or whether RECODE is robust to wide variation around the reported values.

What evidence exists in the paper. Two hyperparameters are ablated: $\gamma$ (Figure 11 shows that $\gamma = 1$ significantly degrades performance; Figures 10 and 12 characterize the discount's effect) and the removal strategy (Figures 8 and 9 show that $1/c_j^2$ is more robust than alternatives but all strategies produce viable algorithms). No other parameter is ablated. The fact that different memory sizes and $\eta$ values are used for Atari vs. DM-HARD-8 ( $5 \times 10^4$ vs. $2 \times 10^5$ atoms; $\eta = 0.05$ vs. $0.2$ ) already indicates these choices matter — the paper does not explain how these values were selected or how sensitive results are to deviations.

Mitigation status. The paper does not address this limitation or propose methods for automatic hyperparameter selection. The fact that only two domains are tested means even domain-level heuristics (e.g., "use larger memory for 3D environments") are speculative. A practitioner must either copy the reported values exactly or conduct their own hyperparameter sweep, which is expensive for RL training runs at the scale of hundreds of millions of frames.

Computational Overhead of the Shared Memory Is Not Quantified

The assumption or constraint. RECODE introduces a shared memory module that must be queried (nearest-neighbor search over $|M|$ atoms for bandwidth estimation, kernel evaluation, and distance checks for insertion) and updated (count discounting over all atoms, potential atom insertion/removal with redistribution) at every timestep by all actors in a distributed setup. The paper states that "RECODE shares the memory across all actors in a distributed agent" (Section 3), which implies communication overhead — actors must send embeddings to the memory and receive intrinsic rewards. Performance curves and training times are reported in terms of environment frames (Figures 4–7), not wall-clock time or total FLOPs. The implicit assumption is that RECODE's per-step cost is comparable to prior methods (NGU's episodic memory also requires k-NN queries per step) and that the communication overhead of the shared memory is negligible compared to the RL training pipeline.

The consequence. Without cost analysis, the practical deployability of RECODE is unclear. The nearest-neighbor search over $2 \times 10^5$ atoms (DM-HARD-8) at every step introduces a computational burden that scales with memory size — if the memory needs to grow for more complex environments, the per-step cost grows accordingly. Count discounting requires updating all $|M|$ counts at every step, which is $O(|M|)$ regardless of whether atoms are actively used. In a distributed system with many actors, the shared memory becomes a synchronization bottleneck: all actors must read from and write to the same memory, and the memory's internal state changes after each write, requiring some form of concurrent access control. The paper does not report whether the memory operations are batched, whether actors can query asynchronously, or what the throughput limitations are.

What evidence exists in the paper. There is no computational cost analysis anywhere in the paper — no measurements of memory query latency, memory update time, communication overhead relative to the RL learner's gradient computation, or wall-clock time comparisons against NGU or other baselines at the same number of environment frames. The architecture diagram in Figure 15 (Appendix F) shows the Memory as a separate component communicating with Actors, but no details of the implementation (e.g., in-memory data structures for fast nearest-neighbor search, batching protocols, synchronization mechanisms) are provided.

Mitigation status. Not addressed. The paper focuses entirely on sample efficiency (performance vs. environment frames) and ignores computational efficiency (performance vs. wall-clock time or FLOPs). This is a significant omission for a method whose core innovation is scaling memory horizon by orders of magnitude — maintaining $5 \times 10^4$ to $2 \times 10^5$ atoms with per-step discounting and nearest-neighbor queries is computationally more expensive than NGU's episodic memory, which resets every episode and stores at most ~27,000 observations for Atari. The paper does not discuss implementation optimizations (e.g., approximate nearest-neighbor search, sparse discount updates, batched processing) that might mitigate this cost.

Two DM-HARD-8 Tasks Remain Completely Unsolved, With No Analysis of Why

The assumption or constraint. RECODE-CASM achieves zero or near-zero scores on two of eight DM-HARD-8 tasks: Remember Sensor (0.00 ± 0.00) and Wall Sensor Stack (0.00 ± 0.00), as shown in Table 5. No other method solves these tasks either — NGU-CASM and RECODE-AP also score zero, and BYOL-Explore results for these specific tasks are not broken out but are presumably also near zero given the aggregated learning curves in Figure 18. These failures are mentioned in passing in the results table but are never discussed in the main text or appendices. The paper provides no analysis of what makes these tasks harder than the six that RECODE-CASM solves, nor any diagnostic experiments probing whether the failure is due to insufficient representation quality, insufficient memory horizon, insufficient memory capacity, or a more fundamental limitation of novelty-based exploration for these task structures.

The consequence. These unsolved tasks represent the clearest boundary condition on RECODE's effectiveness, and the paper's silence on them leaves practitioners with no understanding of when RECODE might be expected to fail. Are Remember Sensor and Wall Sensor Stack harder because they require longer memory horizons than $\gamma = 0.997$ provides? Because they require object interactions that the CASM representation fails to capture? Because the reward is too sparse even for RECODE's exploration to bridge? Because the procedural generation creates state spaces too large for the $2 \times 10^5$ atom memory to cover? Without diagnostic analysis, each of these hypotheses is equally plausible, and a practitioner encountering a similarly unsolved task in their own domain has no way to determine which RECODE parameter or component to adjust.

What evidence exists in the paper. Only the zero scores in Table 5. The task names are suggestive — "Remember Sensor" and "Wall Sensor Stack" imply tasks requiring memory of past sensor readings — but no task descriptions are provided beyond the general DM-HARD-8 overview in Section 5.2 ("Each task requires the agent to interact with specific objects in its environment in order to reach a large apple that provides reward"). The paper's claim that RECODE "sets a new state-of-the-art in task performance" (Section 1) is accurate for the six solved tasks but masks the complete failure on two tasks that no method approaches solving.

Mitigation status. Not addressed at all. The paper acknowledges in Section 6 that future work should investigate "combining PRM tree-search techniques in combination with revisions" but does not mention the unsolved DM-HARD-8 tasks or propose diagnostic experiments. This is a missed opportunity — analyzing the failure modes on unsolved tasks often yields more insight than celebrating successes on solved ones, and would substantially strengthen the paper's contribution by characterizing the boundary conditions on RECODE's applicability.

Generalization Beyond Atari and DM-HARD-8 Is Untested

The assumption or constraint. All experiments use exactly two benchmark suites: the eight-game Atari hard exploration subset (Bellemare et al., 2016) and the eight-task DM-HARD-8 suite (Gulcehre et al., 2019). Both are visual domains where the observation is an image (2D pixel arrays for Atari, 3D first-person renders for DM-HARD-8) and the core exploration challenge is navigating physical spaces to discover sparse rewards. The paper does not test on continuous control benchmarks (e.g., DeepMind Control Suite, MuJoCo), robotics manipulation tasks, environments with non-visual observations (e.g., proprioceptive states), tasks where exploration requires object manipulation rather than navigation, or environments where the exploration challenge is not spatial coverage but discovering rare events or causal relationships. The abstract and conclusion claim RECODE is a "simple yet robust drop-in exploration method" (Section 6) and "compatible with any RL agent and representation learning method" (Section 6), implying broad applicability.

The consequence. The "drop-in" claim is supported only for the specific regime tested: visual observation spaces, sparse spatial rewards, MEME or VMPO base agents, and AP or CASM representations. It is unknown whether RECODE's clustering-based density estimation would work for non-spatial state spaces where Euclidean distance in the embedding space has different semantic meaning (e.g., symbolic or discrete-feature environments), or for exploration challenges that are not about covering the state space uniformly (e.g., curiosity about rare transitions or information gain). The memory size and discount parameters may need to differ substantially for non-visual domains — in proprioceptive state spaces, the number of meaningfully distinct states may be much smaller than in pixel-based spaces, making the $5 \times 10^4$ atom memory either wasteful or, if the embedding space is very low-dimensional, prone to over-clustering. Additionally, all experiments use either MEME (a specific distributed Q-learning architecture) or VMPO (a specific policy-gradient architecture); the paper does not test with simpler base agents (e.g., PPO, SAC) that might be used in resource-constrained or single-machine settings where the distributed shared-memory architecture is impractical.

What evidence exists in the paper. Only results on Atari and DM-HARD-8. The VMPO multi-task experiment in Appendix J (Figure 19) extends to a different base agent and the multi-task setting, but remains within the DM-HARD-8 domain and uses AP embeddings rather than the full RECODE-CASM combination. The BYOL embedding experiment (Appendix K.3, Figure 22) shows RECODE works with an alternative representation on Atari, but this is still within the same domain and task family. The paper does not cite or discuss any experiments on non-visual domains, non-navigation tasks, or non-MEME/VMPO agents.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation or scope the "drop-in" claim to the tested domains. Section 6 suggests future work on "combining PRM tree-search techniques in combination with revisions" and "distilling the outputs of applying additional test-time compute back into the base LLM" — directions that are orthogonal to domain generalization. A more grounded limitation statement would acknowledge that RECODE's effectiveness has been demonstrated only on visual navigation-style exploration tasks and that its transfer to other exploration regimes (manipulation, causal discovery, information-seeking) is untested.

The Benefit of Shared Memory Over Per-Actor Memory Is Asserted but Not Measured

The assumption or constraint. RECODE's memory is shared across all actors in the distributed training setup, unlike NGU where each actor maintains its own episodic memory. The paper claims this "greatly increases the frequency of updates to each atom resulting in less representation drift between memory updates" (Section 3). The implicit assumption is that the increased update frequency meaningfully improves exploration performance or memory quality compared to a hypothetical RECODE variant with per-actor memories — i.e., that sharing is a causal contributor to RECODE's results, not merely an implementation convenience.

The consequence. Without an ablation comparing shared vs. per-actor RECODE memory, the contribution of sharing to the reported performance is unknown. It is possible that per-actor RECODE would perform similarly (each actor maintains its own $5 \times 10^4$ atom memory, independently updated), which would simplify deployment by eliminating the shared-memory communication bottleneck. Alternatively, sharing might be essential — if individual actors explore different regions, per-actor memories would have poor coverage, and sharing provides the global coordination needed for efficient exploration. The paper's claim that sharing reduces representation drift is plausible (more updates per unit of wall-clock time means the memory tracks the current embedding function more closely), but the quantitative effect is unmeasured.

What evidence exists in the paper. None. There is no experiment comparing shared vs. per-actor RECODE. The claim about update frequency is a theoretical argument, not an empirical finding. The age distribution in Figure 3 shows that the shared memory contains atoms spanning thousands of episodes, but this does not isolate the effect of sharing — a per-actor memory with the same discount factor would also contain atoms across a range of ages, though potentially with different coverage properties.

Mitigation status. Not addressed. The paper treats shared memory as an architectural feature of RECODE without interrogating whether it matters for performance. Given that shared memory introduces implementation complexity (synchronization, concurrent access, communication overhead), quantifying its benefit relative to the simpler per-actor alternative would be practically valuable. This is a missed ablation that could have strengthened the paper's mechanistic understanding of why RECODE works.

The Count Discounting Mechanism Interacts with the RL Discount in Unspecified Ways

The assumption or constraint. RECODE's atom count discounting ( $c_l \leftarrow \gamma c_l$ with $\gamma = 0.999$ for Atari and $\gamma = 0.997$ for DM-HARD-8) controls the effective memory horizon by decaying old counts. The paper treats $\gamma$ as a standalone parameter to be tuned per domain and demonstrates via ablation (Figure 11) that removing discount ( $\gamma = 1$ ) degrades performance. However, the RL agent itself uses a discount factor for extrinsic reward ( $\text{Max Discount} = 0.9997$ , $\text{Min Discount} = 0.97$ , from Tables 1 and 2) that controls the agent's planning horizon. These two discount factors — one for exploration memory, one for value estimation — operate on different objects (atom counts vs. future rewards) but both influence how far into the past or future the agent's behavior is influenced. The paper does not discuss whether these values should be coordinated or how their interaction affects exploration dynamics.

The consequence. If the memory discount $\gamma$ is set independently of the RL discount, the agent might "forget" that it has visited a region (count decays) before the value function has learned that the region is unrewarding, causing the agent to re-explore already-visited areas. Conversely, if $\gamma$ is too close to 1 relative to the RL discount, the memory might retain counts for regions the agent stopped visiting long ago, suppressing novelty signals in areas that have genuinely changed (e.g., due to environment stochasticity or procedural regeneration). The optimal $\gamma$ may depend on the environment's timescale of relevant change, not just on the representation drift that the paper identifies as the primary motivation for discounting. Without understanding this interaction, practitioners tuning $\gamma$ for a new domain have no principled way to coordinate it with the RL agent's temporal credit assignment horizon.

What evidence exists in the paper. The only direct evidence on $\gamma$ comes from the ablation in Figure 11 ( $\gamma = 1$ vs. $\gamma = 0.999$ on Atari) and the toy example in Figure 10 (showing that $\gamma$ interpolates between short-term and long-term memory). These experiments treat $\gamma$ in isolation, without varying the RL discount simultaneously. The DM-HARD-8 experiments use $\gamma = 0.997$ alongside an RL max discount of $0.997$ (Table 2) — the values happen to be equal, but the paper does not comment on this or test whether coordinated values matter. The Atari experiments use $\gamma = 0.999$ with an RL max discount of $0.9997$ (Table 1) — the memory discount is slightly lower (faster forgetting) than the RL discount, but again, this relationship is not discussed or tested.

Mitigation status. Not addressed. The paper frames $\gamma$ selection as a tradeoff between short-term and long-term memory (Section 3, Appendix C.2) and identifies its role in handling representation drift, but does not discuss its interaction with the RL discount factor. This is a conceptual gap — the exploration memory and the value function are both temporal credit assignment mechanisms operating at different timescales, and their discount parameters likely should be considered jointly rather than independently. Future work on "compute-optimal" exploration might study this interaction, but the current paper treats them as independent hyperparameters.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the landscape in two distinct ways, operating at different levels of abstraction. At the architectural level, it demonstrates that the dual-mechanism exploration architecture — which had become standard in state-of-the-art deep RL agents (NGU, Agent57, MEME) — is unnecessary. A single non-parametric clustering memory, properly designed to handle streaming non-stationary data through count discounting and probabilistic atom replacement, can cover both the short-term and long-term novelty estimation that previously required separate episodic and parametric components. This is not merely a simplification for its own sake: it removes the RND-based long-term component that was fundamentally incompatible with learned representations, thereby enabling the entire exploration pipeline to benefit from advances in representation learning. The consequence is that exploration methods can now be designed as a clean two-component stack — representation (what makes states similar) plus memory (how many times have I been here) — rather than as an ad-hoc assembly of mechanisms that evolved to patch each other's limitations.

At the conceptual level, the paper reframes exploration as an interface problem. Prior work asked "what exploration bonus should we use?" and answered with specific mechanisms (RND, ICM, NGU, BYOL-Explore) that bundled representation and count estimation together, or at least coupled them tightly. RECODE asks instead: given a good representation from anywhere, can we build a count estimator that works? This is a fundamentally different question — it separates the semantic question (what makes two states similar?) from the statistical question (how many times have I been here?) and argues they should be solved by different mechanisms optimized for different requirements. The paper's modular naming convention (AGENT-EXPLORATION-EMBEDDING) and its systematic ablation across all three axes (agent, exploration bonus, representation) make this reframing operational rather than philosophical — it provides a concrete template for how future exploration research should structure its comparisons.

This reframing reconciles a tension that had been building in the exploration literature. On one side, representation learning methods (ICM, BYOL-Explore, and now CASM) were producing increasingly powerful state embeddings that captured controllable, semantically meaningful features and filtered noise. On the other side, the exploration bonuses that could actually use these representations at scale were limited to short episodic horizons (NGU's EMM component) — the long-term component (RND) remained stuck on raw observations, creating an asymmetry where representation improvements only benefited part of the exploration signal. The Noisy Montezuma experiment (Section 5.3, Figure 6) makes this tension concrete: NGU already knew that AP representations filter noise, and its episodic memory uses them, yet its long-term RND component degenerates to random exploration under noise because it cannot use the representation. RECODE resolves this by making the entire exploration bonus (short-term and long-term) operate on the same embedding space, closing the gap between what representations can provide and what exploration mechanisms can leverage.

The paper also redirects research attention in specific ways. It makes the case that improving parametric density estimators for exploration (RND and its variants) is a less promising direction than improving non-parametric memory management — the RNDonAP ablation (Appendix K.2, Figure 21) shows that even when given access to AP embeddings, RND fails because it cannot disentangle representation drift from genuine novelty. Similarly, it suggests that designing ever-more-complex exploration bonus formulas may be unnecessary when the real bottleneck is memory horizon — the fact that RECODE matches NGU on Atari with a simpler bonus but a much longer memory horizon (thousands of episodes vs. one) implies that memory span, not bonus shape, was the binding constraint. This matters for research prioritization: time spent developing new intrinsic reward formulas may yield less return than time spent improving how long an agent can remember where it has been.

Finally, the paper establishes memory horizon as a first-class design parameter through its characterization of the discount factor γ. Prior work treated memory as binary: you either had it across episodes (via parametric methods) or you didn't (episodic-only). RECODE's demonstration that γ smoothly interpolates between short-term and long-term memory (Figures 10, 11, 12, 14) — and that the optimal horizon is both finite and environment-dependent — opens a new axis for exploration algorithm design. This is analogous to how the RL community treats the value discount factor: not as a fixed constant to be set once, but as a parameter that shapes the agent's effective planning horizon and should be tuned per domain or even adapted during training. Future exploration methods can build on this framing, treating the memory horizon as something to be optimized rather than accepted as a structural constraint of the chosen architecture.

Follow-Up Research This Work Enables

Characterizing the failure modes on the unsolved DM-HARD-8 tasks. Two of eight DM-HARD-8 tasks — Remember Sensor and Wall Sensor Stack — remain completely unsolved by RECODE-CASM (scores of 0.00, Table 5). The paper provides no analysis of why these tasks resist all tested exploration methods. A diagnostic study would systematically vary RECODE's key parameters (memory size, γ, κ, η) and representation (AP vs. CASM vs. alternative encoders) on these two tasks, measuring whether any configuration yields progress. The specific hypothesis to test: do these tasks require memory horizons longer than γ = 0.997 provides, or do they require state representations that CASM's action-prediction objective fails to capture (e.g., because the tasks require remembering sensor readings that are not action-relevant at the moment of observation)? If parameter sweeps fail to produce any improvement, the conclusion would be that novelty-based exploration is fundamentally insufficient for these tasks — perhaps they require model-based exploration, information gain, or explicit memory architectures beyond visitation counting. If a specific parameter change (e.g., γ → 0.9999) unlocks progress, it would characterize the memory horizon requirements of different exploration challenges. This experiment is directly enabled by RECODE's modular design: the representation and memory parameters can be varied independently without changing the base agent.

Scaling memory size with environment complexity. RECODE uses fixed memory sizes per domain (5 × 10^4 atoms for Atari, 2 × 10^5 for DM-HARD-8), but provides no evidence about whether these values are near-minimal, near-optimal, or could be improved with further scaling. A scaling study would train RECODE on Montezuma's Revenge and Pitfall! with memory sizes varying from 10^3 to 10^6 atoms (with appropriate adjustments to γ to maintain comparable effective horizons), measuring both final performance and the rate of exploration progress. The specific question: is there a phase transition where performance sharply degrades below a critical memory size (indicating that the state space requires a minimum number of clusters to cover), or does performance degrade gracefully? Relatedly, for a fixed memory size, how does performance vary with the insertion probability η? The paper's theoretical framing (Section 3, Appendix C) suggests that η controls the tradeoff between memory coverage (more atoms created) and stability (fewer outliers inserted) — quantifying this tradeoff across environments with different state-space sizes and observation noise levels would provide practical guidance for parameter selection. This experiment is enabled by the fact that RECODE's parameters are fully exposed and independently tunable, unlike NGU where the two novelty signals interact in complex ways.

Learning the discount factor γ during training. The paper shows that the optimal γ is environment-dependent (0.999 for Atari, 0.997 for DM-HARD-8) and that removing discount entirely (γ = 1) causes significant degradation when representations are trained concurrently (Figure 11). However, γ is set as a fixed hyperparameter. A natural extension is to make γ adaptive — for instance, by maintaining a running estimate of representation drift (measured as the average distance between embeddings of recently-visited states and their embeddings from earlier checkpoints) and adjusting γ to keep the effective memory horizon aligned with the rate of representation change. When the embedding function is changing rapidly (early training, or after a distribution shift in the policy), γ would decrease to forget stale atoms faster; when the embedding stabilizes (late training), γ would increase to extend memory horizon. The specific experiment: compare fixed-γ RECODE against adaptive-γ RECODE on Montezuma's Revenge and DM-HARD-8, measuring whether the adaptive variant achieves faster exploration progress or higher final performance without requiring per-domain tuning. A simpler variant would meta-learn γ across a distribution of procedurally-generated environments, testing whether a single learned schedule can generalize. This direction is enabled by RECODE's clean separation of the discount rate from other memory parameters — γ controls only the forgetting rate, making it a natural target for adaptation.

Combining RECODE with model-based exploration objectives. RECODE's intrinsic reward is purely count-based: r_t ∝ 1 / √(N_𝒦(M, e_t)). This drives uniform coverage of the state space but provides no incentive to seek out states that are informative about the environment's dynamics or reward structure — it treats all unvisited states as equally valuable. Model-based exploration methods (e.g., maximizing information gain about the transition model, or seeking states where the ensemble of dynamics models disagrees) target a different notion of novelty: not "have I been here?" but "will being here improve my understanding of how the world works?" A combined agent could use RECODE for long-term coverage (ensuring the entire state space is visited) while using a model-based bonus for local information-seeking (prioritizing which unvisited regions to explore first based on expected learning progress). The specific experiment: on DM-HARD-8's Push Blocks (where RECODE-CASM achieves only 4.06, Table 5), add an ensemble-of-dynamics-models disagreement bonus to RECODE's count-based reward and measure whether the combined bonus achieves higher performance than either alone. The hypothesis is that Push Blocks requires not just visiting all locations but actively experimenting with object interactions — a form of curiosity that pure count-based exploration does not incentivize. This experiment is enabled by RECODE's modularity: the count-based bonus can be summed with any other intrinsic reward without modifying RECODE's internals.

Stress-testing RECODE's noise robustness across noise types. The Noisy Montezuma experiment (Figure 6) demonstrates robustness to additive white noise, which AP embeddings can filter because the noise is uncorrelated with actions. But real-world sensor noise is rarely white — it can be structured (e.g., motion blur, lighting changes), action-correlated (e.g., sensor readings that fluctuate based on motor current), or occlusive (blocking task-relevant features). A systematic noise-robustness study would test RECODE on Montezuma's Revenge with multiple noise conditions: (a) structured noise (Gaussian blur, salt-and-pepper corruption), (b) action-correlated noise (e.g., screen shake proportional to agent velocity), (c) dynamic distractors (moving objects in the background that are uncontrollable but temporally structured), and (d) partial occlusion (random black rectangles overlaid on the observation). For each condition, compare RECODE-AP against NGU-AP and an oracle method that has access to noise-free observations. The specific question: under what noise conditions does the AP representation succeed at filtering, and under what conditions does it fail (allowing noise to enter the embedding space and corrupt RECODE's counts)? This would map the boundary of RECODE's noise robustness claim and identify which noise types require better representations (beyond AP) or fundamentally different exploration strategies. This experiment is enabled by the clean diagnostic provided by the NGU vs. RECODE comparison under noise — adding noise conditions systematically characterizes the robustness envelope.

Benchmarking RECODE against parametric long-term memory that can use learned representations. The paper's central architectural claim is that non-parametric clustering is better than parametric alternatives for long-term novelty estimation when representations are trained concurrently. The RNDonAP ablation (Appendix K.2) provides one data point supporting this claim, showing that simply feeding AP embeddings to RND does not work. But RND is a specific parametric method — its failure may be due to its particular design (random target network, prediction error as novelty) rather than an inherent limitation of parametric approaches. A stronger test would compare RECODE against alternative parametric long-term memory designs: (a) a variational autoencoder (VAE) trained on AP embeddings that computes novelty as reconstruction error, with periodic retraining to handle distribution shift, (b) an exemplar-based approach that stores a subsample of past embeddings and computes novelty as distance to the nearest stored exemplar (a parametric/non-parametric hybrid), (c) a normalizing flow that explicitly estimates density in the AP embedding space, updated with a replay buffer of recent and diverse embeddings. The specific experiment: on Pitfall! and Montezuma's Revenge, compare RECODE against these alternatives in terms of (i) final game score, (ii) rate of exploration progress (slope of the score curve), and (iii) computational cost (wall-clock time). If one of these parametric methods matches RECODE's performance, the paper's claim about the necessity of non-parametric memory would be weakened. If none do, the claim is strengthened — and the specific failure modes (catastrophic forgetting? slow adaptation?) would provide insight into what makes the parametric approach fundamentally limited for this problem. This experiment is enabled by RECODE's modular design: any density estimator operating on the same AP embeddings can be swapped in without changing the rest of the pipeline.

Practical Applications and Downstream Use Cases

On-device RL for mobile robots with noisy sensors. A mobile robot operating in a home or office environment must explore its surroundings to build a spatial understanding, but its sensors — cameras, lidar, microphones — are subject to varying lighting conditions, occlusions, sensor noise, and uncontrollable background changes (people walking, shadows moving). RECODE's robustness to observation noise (demonstrated in Figure 6 on Noisy Montezuma) combined with a representation trained via inverse dynamics on the robot's action-observation stream (the robot equivalent of AP) provides an exploration mechanism that will not get stuck on sensor artifacts. The shared-memory architecture maps naturally to multi-robot fleets, where each robot's exploration benefits all others — a room explored by one robot becomes less novel for all robots, encouraging coordinated coverage. The memory size (5 × 10^4 to 2 × 10^5 atoms) and per-step computational cost (nearest-neighbor queries over that many atoms) are well within the capability of onboard embedded processors, especially with approximate nearest-neighbor search. The concrete benefit: the robot explores its environment systematically rather than fixating on flickering lights or moving shadows, reducing the time to build a complete spatial map — a metric that could be measured directly in hours-to-full-coverage on standard robotics benchmarks (e.g., Habitat, Gibson environments).

Training data generation for self-improving agents in procedurally-generated games. A game studio developing a procedurally-generated 3D environment (e.g., a roguelike with randomized levels) needs to train NPCs or automated playtesters that can explore each generated level to discover bugs, balance issues, or unexpected interactions. RECODE's ability to track exploration progress over thousands of episodes (Figure 3, Figure 14) means an agent can systematically cover the state space of each new generated level without forgetting which areas it has already visited — something episodic-memory methods fail at (as demonstrated by EMM's failure on Montezuma's Revenge in Figure 4). The CASM representation provides robustness to the visual variation in procedurally-generated assets (randomized object shapes, colors, positions) by encoding the controllable, functionally relevant aspects of the state. The specific workflow: deploy RECODE-CASM agents to explore each procedurally-generated level, record trajectories that discover novel states or interactions, and use those trajectories to train the game's NPCs or to identify edge cases for human designers to review. The metric that matters is coverage — what fraction of the procedurally-generated state space does the agent visit within a fixed budget of environment steps? — and RECODE's 4× memory horizon advantage over episodic methods (thousands of episodes vs. one) directly translates to higher coverage for levels requiring persistent exploration across many attempts.

Pre-training exploration policies for sparse-reward robotics tasks via simulation. Training a robot arm to perform sparse-reward manipulation tasks (e.g., assembling furniture, sorting objects into bins) requires exploring a high-dimensional continuous action space where reward is only received upon task completion. Standard approaches use shaped rewards (which require engineering) or human demonstrations (which are expensive). RECODE combined with a suitable representation for proprioceptive observations (joint angles, end-effector position, tactile readings) could serve as a pre-training exploration policy: run RECODE in simulation to discover which regions of the state-action space lead to task-relevant outcomes (object grasped, peg inserted, bin reached), then use the collected trajectories to bootstrap a goal-conditioned policy or to train a reward model. The benefit over RND or NGU is that RECODE's count-based bonus does not saturate — even after millions of steps, the count discounting (γ = 0.999 corresponding to an effective horizon of ~10^4 steps) ensures that states visited long ago lose their counts and become novel again if the policy hasn't visited them recently, encouraging the agent to re-explore regions that might now be reachable with improved skills. This matters for manipulation because the ability to reach a state depends on having learned prerequisite skills — a state that was uninteresting early in training (because the agent couldn't interact with it) becomes interesting again once the agent has learned to grasp objects. The specific application: pre-train RECODE on a set of simulated manipulation tasks from the RLBench or Meta-World benchmarks, then fine-tune with sparse task rewards, measuring whether RECODE pre-training reduces the number of environment steps needed to achieve task success compared to RND pre-training or no pre-training.