ArXiv: 2605.11711

🎯 Pitch

Minimizing representation prediction errors does not guarantee meaningful dynamics encodingβ€”mutual information can actually drop. DR.Q fixes this by maximizing InfoNCE between state-action and next-state representations, combined with faded prioritized sampling, achieving up to 26.8% gains over strong baselines like MR.Q across 73 tasks.

1. Executive Summary

This paper proposes Debiased model-based Representations for Q-learning (DR.Q), an off-policy RL algorithm that explicitly maximizes the mutual information between the state-action representation and the next state representation β€” via an InfoNCE loss that acts as a lower bound on mutual information β€” while simultaneously adopting a faded prioritized experience replay sampling strategy that combines TD-error-based prioritization with a forget mechanism to favor recent, high-error transitions. Evaluated across 73 continuous control tasks spanning MuJoCo, the DMC suite, and HumanoidBench using a single set of hyperparameters, DR.Q matches or surpasses strong baselines including MR.Q, SimBaV2, and TDMPC2 β€” achieving gains such as a 15.5% improvement over SimBaV2 on DMC-Hard tasks and a 26.8% lead over MR.Q on visual DMC tasks β€” establishing that model-based representation learning benefits substantially from explicit mutual information maximization and decayed prioritization, but only when the base model already possesses sufficient capacity to extract useful patterns from the environment dynamics.

2. Context and Motivation

The Core Problem: Model-Based Representations Can Be Biased and Uninformative

The fundamental problem this paper tackles concerns how we learn useful state and action representations for reinforcement learning agents, specifically through a paradigm called model-based representation learning. In this framework, rather than explicitly building and planning through a world model (which is computationally expensive), the agent learns an encoder that maps raw observations into a latent space where the dynamics of the environment are implicitly captured. These learned representations are then fed to downstream actor and critic networks that learn the actual policy.

The specific gap the paper identifies is two distinct sources of bias in existing model-based representation methods that cause the learned representations to be suboptimal, ultimately degrading the performance of the policy and value functions that depend on them.

Bias source 1: The deviation-minimization objective does not guarantee informativeness. Existing methods like MR.Q (Fujimoto et al., 2025) train their encoders by minimizing the Euclidean distance between the current state-action representation zsaz_{sa} and the next-state representation zsβ€²z_{s'} β€” enforcing what they call "latent dynamics consistency." The intuition is straightforward: if the model can predict the next state representation from the current state-action representation with low error, then the representations must capture meaningful dynamics information.

The paper's key theoretical insight, formalized in Theorem 4.1, reveals that this intuition is flawed. Minimizing E[βˆ₯Zsaβˆ’Zsβ€²βˆ₯22]\mathbb{E}[\|Z_{sa} - Z_{s'}\|_2^2] does not necessarily increase the mutual information I(Zsa;Zsβ€²)I(Z_{sa}; Z_{s'}) between these representations. The authors construct two simple counterexamples to prove this:

  • First, when Zsa=XZ_{sa} = X and Zsβ€²=X+Ο΅Z_{s'} = X + \epsilon with independent noise ϡ∼N(0,Οƒ2Id)\epsilon \sim \mathcal{N}(0, \sigma^2 I_d), the MSE term scales with Οƒ2\sigma^2 while the mutual information scales with d2log⁑(1+1/Οƒ2)\frac{d}{2}\log(1 + 1/\sigma^2) β€” these move in opposite directions as Οƒ2\sigma^2 varies.
  • Second, when Zsa=XZ_{sa} = X and Zsβ€²=kXZ_{s'} = kX for some scaling factor k>1k > 1, the MSE term scales with (1βˆ’k)2(1-k)^2 while the mutual information scales with d2log⁑(2Ο€ek2)\frac{d}{2}\log(2\pi e k^2) β€” both increase as kk grows, showing no consistent relationship.

The practical consequence of this theorem is significant: an encoder can achieve low MSE by learning to predict irrelevant or redundant components of the state while neglecting features that are crucial for control. The representations become "falsely aligned" β€” numerically close in Euclidean space but not actually encoding information about each other in the information-theoretic sense. This problem is particularly acute in high-dimensional environments like HumanoidBench with dexterous hands, where the state space contains many task-irrelevant dimensions (e.g., precise finger positions matter less for running than for manipulation).

Bias source 2: Standard experience replay overfits to early, low-quality experiences. The second problem concerns how transitions are sampled from the replay buffer during training. Both uniform sampling (treating all transitions equally) and Prioritized Experience Replay (PER) (Schaul et al., 2015) (which samples transitions proportionally to their TD errors) suffer from what Nikishin et al. (2022) termed the primacy bias β€” the tendency of neural networks to overfit to early experiences in the replay buffer. Early in training, when the policy is poor, the agent generates low-quality trajectories with inaccurate value estimates. If these early experiences are repeatedly sampled (because they have high TD errors due to the agent's initial ignorance, or simply because the buffer is small), the encoder and the value function can become biased toward these obsolete data distributions, making it harder to adapt as the policy improves.

The forget mechanism (Novati & Koumoutsakos, 2019; Wang et al., 2020; Kang et al., 2025) addresses this by decaying the sampling probability of older transitions exponentially β€” (1βˆ’Ο΅)i(1-\epsilon)^i for transition index ii where i=0i=0 is the newest. However, this creates its own problem: it ignores potentially valuable older transitions that happen to have large TD errors and could contribute meaningfully to learning, especially if they represent rare but important state transitions.

The paper identifies that neither PER alone (which ignores recency) nor the forget mechanism alone (which ignores TD-error magnitude) is optimal. Both inject bias β€” PER toward early high-error experiences, forget toward potentially low-information recent experiences β€” and neither alone provides the right signal for representation learning.

Why This Problem Matters

The practical importance of addressing these two biases stems from the broader goal of sample-efficient reinforcement learning β€” getting agents to learn good policies from fewer environment interactions. This matters enormously for real-world deployment. In robotics, for instance, every interaction with the physical world is time-consuming, potentially damaging to equipment, and may require human supervision. An algorithm that learns a walking policy in 500K steps rather than 2M steps translates directly to reduced costs and faster experimentation cycles.

But there is a deeper architectural motivation that the paper implicitly addresses. Model-based representation learning occupies a strategic middle ground in the RL algorithm design space. Pure model-free methods (like SAC, TD3) are simpler and have lower per-step computational costs, but they learn exclusively from reward signals, which can be sparse and uninformative β€” especially in complex environments where most state transitions produce no immediate reward. Pure model-based methods (like DreamerV3, TDMPC2) learn explicit world models and can extract rich learning signals from every transition, but they incur substantial computational overhead from model training and planning.

Model-based representations promise the best of both worlds: richer learning signals than pure model-free methods (since every transition provides a dynamics prediction target, not just reward-bearing transitions), but without the computational burden of explicit planning. However, this promise only materializes if the representations actually capture useful information. If the encoder learns to minimize MSE by attending to irrelevant features, the downstream actor-critic receives representations that are no better β€” and potentially worse β€” than raw states. The paper's contribution is therefore not just incremental performance improvement, but a fix to a fundamental flaw that limits the entire paradigm.

Prior Approaches and Where They Fall Short

The paper situates itself within a rich literature of dynamics-based representation learning, spanning both model-based and model-free methods:

Model-based dynamics learning. Classic model-based RL methods learn explicit dynamics models s^β€²=g(s,a)\hat{s}' = g(s, a) by minimizing E[βˆ₯g(s,a)βˆ’sβ€²βˆ₯2]\mathbb{E}[\|g(s, a) - s'\|^2], then use these models for planning (Hansen et al., 2022; 2024) or data augmentation (Janner et al., 2019). These methods achieve strong sample efficiency but at high computational cost. More recent approaches like DreamerV3 (Hafner et al., 2023) and TDMPC2 (Hansen et al., 2024) learn latent dynamics models β€” predicting in a compressed representation space rather than raw state space β€” which reduces computational burden but still requires explicit model rollouts and planning.

Model-free dynamics-based representations. A parallel line of work embeds dynamics learning into model-free algorithms by training encoders to predict future latent states or enforce temporal consistency, without ever using these predictions for planning. DeepMDP (Gelada et al., 2019) provided theoretical foundations, showing that value prediction error is upper-bounded by latent transition and reward modeling errors. Subsequent methods built on this: SPR (Schwarzer et al., 2020) used self-predictive representations for Atari; TD7 (Fujimoto et al., 2023) incorporated latent consistency losses alongside SALE (state-action representation learning); and MR.Q (Fujimoto et al., 2025) unified these ideas into a general-purpose algorithm that achieved strong performance across diverse benchmarks with a single set of hyperparameters.

Where these methods fall short on bias 1 (the informativeness gap). The critical oversight in all these prior model-based representation methods β€” including MR.Q, TD7, and the representation learning components of model-based systems like TDMPC2 β€” is that they treat the latent dynamics consistency loss as sufficient. The implicit assumption is: if zsaz_{sa} can predict zsβ€²z_{s'} accurately (low MSE), then zsaz_{sa} must contain all the information needed for control. Theorem 4.1 proves this assumption false. The MSE loss only constrains the first moment of the conditional distribution P(Zsβ€²βˆ£Zsa)P(Z_{s'}|Z_{sa}) β€” it says nothing about whether ZsaZ_{sa} captures the full distribution of possible next states, or whether irrelevant dimensions of Zsβ€²Z_{s'} are being predicted at the expense of control-relevant ones.

The paper further strengthens this argument with Lemma 4.2: maximizing I(Zsa;Zsβ€²)I(Z_{sa}; Z_{s'}) strictly reduces the conditional entropy H(Zsβ€²βˆ£Zsa)H(Z_{s'}|Z_{sa}). This means that adding a mutual information objective makes the latent dynamics model more deterministic and discriminative β€” it reduces uncertainty about what Zsβ€²Z_{s'} will be given ZsaZ_{sa}. Since DeepMDP and MR.Q already established that tighter latent dynamics bounds lead to tighter value error bounds, this directly motivates why maximizing mutual information should improve downstream policy performance. Prior methods simply never optimized this quantity.

Where these methods fall short on bias 2 (the sampling problem). The replay buffer sampling problem has been studied extensively, but prior solutions address only half the issue:

  • PER (Schaul et al., 2015) and LAP (Fujimoto et al., 2020) prioritize transitions with high TD errors, ensuring that "surprising" or poorly-estimated transitions are visited more frequently. This accelerates learning but exacerbates primacy bias β€” early transitions with spuriously high TD errors get over-sampled, anchoring the value function to outdated estimates.

  • Forget mechanisms (Novati & Koumoutsakos, 2019; Kang et al., 2025) decay sampling probability with age, reducing the influence of obsolete experiences. But they treat all old transitions equally, potentially discarding rare but valuable experiences that happen to be old. Kang et al. (2025) showed that FoG (Forget and Grow) achieves strong results with this approach, but their method applies uniform decay regardless of TD error magnitude.

Neither approach alone solves the full problem: PER over-emphasizes old high-error transitions; forget mechanisms under-emphasize old transitions regardless of their informational value. The paper's faded PER proposal β€” multiplying the PER probability by the forget decay β€” is a natural synthesis, but one that, to the authors' knowledge, had not been systematically studied for representation learning.

How DR.Q Positions Itself

DR.Q positions itself not as a fundamentally new paradigm, but as a correction to existing model-based representation methods that addresses two specific, theoretically-grounded sources of bias. The paper builds directly on MR.Q's architecture and training pipeline β€” the same encoder structure, the same short-horizon rollout training, the same separation between encoder training and actor-critic training β€” but modifies two components:

  1. Adds an InfoNCE loss (Oord et al., 2018) to the encoder objective that serves as a variational lower bound on I(Zsa;Zsβ€²)I(Z_{sa}; Z_{s'}), explicitly optimizing for what the MSE loss only implicitly hopes to achieve. This is the primary conceptual contribution β€” the recognition that latent dynamics consistency requires both predictive accuracy (MSE) and informational coverage (mutual information), and that prior methods only addressed the former.

  2. Replaces standard LAP sampling with faded PER, which multiplicatively combines TD-error prioritization and temporal decay. This is a pragmatic synthesis that acknowledges both the value of prioritizing high-error transitions and the danger of overfitting to obsolete data.

The paper is careful to frame these as debiasing operations β€” removing systematic errors in the learning process rather than adding entirely new capabilities. The title "Debiased Model-based Representations" reflects this framing: the goal is to correct distortions in existing methods, not to propose a completely different approach to representation learning.

Crucially, DR.Q does not challenge the model-based representation paradigm itself. It accepts that learning latent dynamics is valuable, that the encoder-actor-critic separation is effective, and that short-horizon rollouts provide useful training signals. The contribution is in recognizing why the paradigm sometimes underperforms (biased representations from insufficient objectives; biased sampling from inappropriate prioritization) and providing targeted fixes. This positioning is strategic: it means DR.Q inherits the broad applicability and simplicity of MR.Q (single hyperparameter set, no task-specific tuning) while addressing documented failure modes β€” particularly in high-dimensional environments where the biases are most pronounced.

3. Technical Approach

3.1 Reader Orientation

DR.Q is an off-policy reinforcement learning algorithm that learns compressed state and state-action representations by enforcing both predictive accuracy (via mean-squared error) and information-theoretic coverage (via mutual information maximization) in a learned latent dynamics model, while simultaneously using a hybrid sampling strategy that combines TD-error prioritization with temporal decay to select training transitions. The system solves the problem of biased model-based representations β€” representations that appear accurate under the MSE metric but fail to encode sufficient information about environment dynamics for downstream policy learning β€” by adding an explicit mutual information objective to the encoder loss and correcting the sampling distribution to emphasize recent, high-information transitions.

3.2 Big-Picture Architecture (Diagram in Words)

The DR.Q system comprises four major components that operate in a coordinated training loop:

  1. State Encoder $f_\omega(s)$ β€” a neural network that maps raw observations $s$ to a compressed state representation $z_s \in \mathbb{R}^{512}$. This is the primary interface between the environment's high-dimensional observations and the downstream networks.

  2. State-Action Encoder $g_\omega(z_s, a)$ β€” a neural network that takes the state representation $z_s$ and the action $a$, and produces a state-action representation $z_{sa} \in \mathbb{R}^{512}$. This representation is the central object that all other components operate on: the latent dynamics model predicts from it, the critic evaluates it, and the mutual information objective aligns it with future states.

  3. Linear MDP Predictor $M(z_{sa})$ β€” a lightweight linear mapping that takes the state-action representation $z_{sa}$ as input and produces two outputs: a predicted next-state representation $\hat{z}_{s'} \in \mathbb{R}^{512}$ and a predicted reward $\hat{r}$. This component enforces the latent dynamics consistency: it is trained to make $\hat{z}_{s'}$ match the actual next-state representation $\tilde{z}_{s'}$ produced by the target state encoder.

  4. Actor-Critic Networks β€” a deterministic policy $\pi_\phi(z_s)$ that maps state representations to actions, and two critic networks $Q_{\theta_i}(z_{sa})$ that estimate Q-values from state-action representations. These are standard off-policy components trained via clipped double Q-learning and deterministic policy gradients, but they operate entirely on the learned representations rather than raw states.

  5. Faded Prioritized Experience Replay Buffer β€” the sampling mechanism that selects which transitions to train on. It stores transitions $(s, a, r, s', d)$ and assigns each a sampling probability that multiplicatively combines its TD-error magnitude (via LAP prioritization) with a temporal decay factor $(1-\epsilon)^i$ where $i$ is the transition's age index (0 being newest).

Information flow during training. The process begins by sampling a batch of transitions using the faded PER mechanism. For each transition, the state $s$ passes through the state encoder $f_\omega$ to produce $z_s$, which then flows to both the state-action encoder (along with $a$) to produce $z_{sa}$, and to the actor network for policy updates. The state-action representation $z_{sa}$ is the hub: it goes to the linear MDP predictor (producing $\hat{z}_{s'}$ and $\hat{r}$, feeding into the encoder loss), and to the critic networks (producing Q-value estimates, feeding into the critic loss). The encoder is trained using Equation 9, which combines three losses β€” reward prediction error, latent dynamics MSE, and InfoNCE mutual information maximization β€” summed over a 5-step unrolled horizon. The actor and critic are trained on the same batch using standard deterministic policy gradient and clipped double Q-learning objectives.

3.3 Roadmap for the Deep Dive

  • First, the formal problem setup β€” what it means to learn model-based representations, the specific components that must be trained (Equation 2), and how they differ from standard model-free and model-based approaches. This establishes the vocabulary and the architectural assumptions DR.Q inherits from MR.Q.

  • Second, the mutual information loss (Section 4.1) β€” Theorem 4.1 (why MSE alone is insufficient), Lemma 4.2 (why maximizing mutual information helps), and the InfoNCE objective (Equation 8) that serves as a tractable lower bound. This is the primary theoretical contribution and the most novel component of the encoder loss.

  • Third, the faded prioritized experience replay (Section 4.2) β€” why uniform sampling and standard PER both suffer from primacy bias, why forget mechanisms alone are insufficient, and how the multiplicative combination in Equation 10 creates the desired sampling distribution. Theorem 4.3 formalizes the properties of this distribution.

  • Fourth, the full encoder training procedure (Section 4.3.1) β€” how the three loss terms (reward, dynamics, InfoNCE) are combined, the horizon-based unrolling mechanism, the target network update schedule, and all hyperparameter values that govern this phase of training.

  • Fifth, the downstream actor-critic training (Section 4.3.2) β€” how the policy and critics are updated using the learned representations, the specific choices (clipped double Q-learning, Huber loss, multi-step returns, target policy smoothing) and their justifications.

  • Sixth, the complete training loop and hyperparameter table β€” how encoder training, critic training, and actor training are interleaved, the update frequencies, and every numerical value that defines DR.Q's behavior.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a representation learning paper whose core idea is that model-based representations for RL suffer from two correctable biases: (i) the standard MSE latent dynamics loss does not optimize for mutual information between current and future representations, leading to representations that are numerically close but informationally impoverished; and (ii) standard experience replay strategies either over-emphasize old high-error transitions (PER) or under-emphasize all old transitions uniformly (forget mechanism), leading to sampling distributions that do not maximize the value of each training step. DR.Q corrects both biases by adding an InfoNCE loss to the encoder objective and replacing the sampling strategy with a multiplicative combination of error-based and recency-based prioritization.

The Model-Based Representation Learning Framework

DR.Q inherits its architectural decomposition from MR.Q (Fujimoto et al., 2025), which itself builds on the TD7 and SALE frameworks. The central design principle is separation of concerns: the networks responsible for learning representations (the encoders) are trained with a different objective and on a different schedule than the networks responsible for learning the policy and value function (the actor and critic). This separation is motivated by the observation that representation learning benefits from modeling environment dynamics (which are stable across policy improvements), while policy learning benefits from stable representations (which change slowly once the dynamics are captured).

The system must train the following functions, as formalized in Equation 2:

f:sβ†’zsf: s \rightarrow z_s g:(s,a)β†’zsag: (s, a) \rightarrow z_{sa} r^:zsaβ†’R\hat{r}: z_{sa} \rightarrow \mathbb{R} πϕ:zsβ†’a\pi_\phi: z_s \rightarrow a QΞΈ:zsaβ†’RQ_\theta: z_{sa} \rightarrow \mathbb{R}

where $f$ is the state encoder, $g$ is the state-action encoder, $\hat{r}$ is the reward predictor (a component of the linear MDP predictor $M$), $\pi_\phi$ is the deterministic policy network, and $Q_\theta$ represents the two critic networks.

What this decomposition accomplishes. The functions $f$ and $g$ (collectively, the "encoders") are responsible for compressing raw observations into a representation space where the environment's dynamics are simple β€” specifically, where the next state representation can be predicted from the current state-action representation by a linear mapping. The linearity constraint is deliberate: it forces the encoders to learn representations where the dynamics are approximately linear, which makes the representation space well-behaved for downstream learning. If the dynamics were highly nonlinear in the representation space, the critic networks would need to be correspondingly more complex to estimate values accurately.

The functions $\pi_\phi$ and $Q_\theta$ are standard actor-critic components that operate entirely on the learned representations. They never see raw states or actions β€” they receive $z_s$ and $z_{sa}$ as inputs. This means the quality of the representations directly determines the ceiling on policy performance: if $z_{sa}$ does not distinguish between states that require different actions, no amount of critic or policy optimization can compensate.

How this differs from pure model-based RL. In model-based methods like DreamerV3 or TDMPC2, the latent dynamics model is used explicitly for planning β€” the agent simulates future trajectories in the latent space and selects actions that maximize predicted return. This requires the dynamics model to be accurate over long horizons and to support both forward simulation and backward credit assignment. DR.Q's encoders do not support planning; they only need to be accurate enough to provide useful representations for the critic, which learns its own value predictions from reward signals. This is a substantially weaker requirement, which is why model-based representation methods can be more computationally efficient than full model-based planning.

How this differs from pure model-free RL. In model-free methods like SAC or TD3, there is no explicit dynamics learning β€” the networks operate directly on raw states and actions, and all learning comes from the TD error (the mismatch between predicted and actual returns). The advantage of adding model-based objectives is that they provide an additional learning signal on every transition, not just those that happen to yield informative TD errors. Even a transition with zero TD error (because the value function is already accurate) still provides a training signal for the encoder: the next state representation should be predictable from the current state-action representation. This is particularly valuable in sparse-reward environments or early in training when the value function is inaccurate and TD errors are unreliable.

The linear MDP predictor $M$. The paper describes $M(z_{sa})$ as a "linear MDP predictor" that outputs both $\hat{r}$ and $\hat{z}_{s'}$, as shown in Equation 5:

r^,z^s′=M(zsa)\hat{r}, \hat{z}_{s'} = M(z_{sa}) zsa=gω(zs,a)z_{sa} = g_\omega(z_s, a) zs=fω(s)z_s = f_\omega(s)

where $\hat{r}$ is the predicted reward, $\hat{z}_{s'}$ is the predicted next-state representation, $z_{sa}$ is the state-action representation from the current state-action encoder, and $f_\omega$, $g_\omega$ share parameters $\omega$.

What this computation does. Given a raw state $s$, the state encoder $f_\omega$ compresses it to a 512-dimensional vector $z_s$. This compressed state, along with the action $a$, is fed to $g_\omega$ to produce another 512-dimensional vector $z_{sa}$ that encodes both the current situation and the chosen action. The linear predictor $M$ then maps $z_{sa}$ to two outputs: a scalar reward prediction $\hat{r}$ and a 512-dimensional vector $\hat{z}_{s'}$ that should match the actual next-state representation (computed by applying $f_\omega$ to the actual next state $s'$). The linearity of $M$ is important β€” it is not a deep network but a single linear transformation, which forces the encoders to learn representations where the dynamics are approximately linear.

Why separate state and state-action encoders. The state encoder $f_\omega$ and state-action encoder $g_\omega$ are separate modules (though they share parameters $\omega$ and are trained jointly). The state encoder produces $z_s$ from $s$ alone β€” this is the representation used by the actor network to select actions. The state-action encoder produces $z_{sa}$ from $z_s$ and $a$ β€” this is the representation used by the critic to evaluate state-action pairs and by the MDP predictor to model dynamics. This separation mirrors the natural structure of MDPs: the current state is sufficient for choosing an action (Markov property), but the next state depends on both the current state and the chosen action. By keeping separate representations, the architecture encodes this structural prior, which likely improves sample efficiency.

The encoder horizon $H$. Following prior work (MR.Q, TDMPC2, DreamerV3), DR.Q does not train the encoder on single transitions in isolation. Instead, it samples subsequences of length $H = 5$ from the replay buffer: $(s_0, a_0, r_1, s_1, a_1, \ldots, r_H, s_H)$. Starting from $s_0$, the encoder produces $z_{s_0}$ and $z_{sa,0}$, which the MDP predictor maps to $\hat{z}_{s,1}$ and $\hat{r}_1$. The predicted next-state representation $\hat{z}_{s,1}$ is then treated as the input to the state-action encoder for the next step (along with $a_1$), producing $z_{sa,1}$, and so on for $H$ steps. This latent rollout trains the encoder to be consistent over multiple timesteps, not just single transitions β€” it must learn representations where the dynamics are not only predictable but compositionally predictable, meaning that chaining predictions together does not cause rapid divergence. The losses (reward, dynamics MSE, InfoNCE) are computed at each step $t = 1, \ldots, H$ and summed.

The Mutual Information Objective: Why MSE Alone Is Insufficient

The core theoretical insight of the paper is stated in Theorem 4.1 and proved in Appendix A.1:

"Minimizing $\mathbb{E}[\|Z_{sa} - Z_{s'}\|_2^2]$ does not necessarily increase the mutual information $I(Z_{sa}; Z_{s'})$."

What this theorem means in operational terms. The theorem establishes that there is no guarantee that making $z_{sa}$ and $z_{s'}$ numerically close in Euclidean distance forces them to share information. The proof constructs two counterexamples:

  • Example 1 (independent noise): Let $Z_{sa} = X$ and $Z_{s'} = X + \epsilon$ where $X \sim \mathcal{N}(0, I_d)$ and $\epsilon \sim \mathcal{N}(0, \sigma^2 I_d)$ independently. The MSE is $\mathbb{E}[\|\epsilon\|^2] = \sigma^2 d$, which grows with $\sigma^2$. The mutual information is $I(Z_{sa}; Z_{s'}) = \frac{d}{2}\log(1 + 1/\sigma^2)$, which shrinks as $\sigma^2$ grows. So when MSE is large, mutual information is small, and vice versa β€” they move in opposite directions.

  • Example 2 (deterministic scaling): Let $Z_{sa} = X$ and $Z_{s'} = kX$ for some $k > 1$. The MSE is $(1-k)^2 d$, which is small when $k \approx 1$. The mutual information is $\frac{d}{2}\log(2\pi e k^2)$, which is also small when $k \approx 1$ (since $\log(2\pi e) \approx 2.84$, but the key is that it decreases as $k \to 1$ from above). So both MSE and mutual information are small when $k$ is near 1 β€” they move in the same direction. This pair of examples proves there is no consistent monotonic relationship.

The practical failure mode this reveals. In high-dimensional environments, many dimensions of the state space may be irrelevant to the control task. For example, in HumanoidBench with dexterous hands, the precise joint angles of individual fingers may be largely irrelevant for a running task β€” what matters is the overall body pose and velocity. The MSE objective provides no mechanism to distinguish between relevant and irrelevant dimensions: the encoder can achieve low MSE by learning to predict the irrelevant dimensions accurately while neglecting the control-relevant ones. The resulting representations $z_{sa}$ and $z_{s'}$ are numerically close (low MSE) but do not encode the information needed for the critic to distinguish between good and bad states, or for the actor to select appropriate actions.

Why mutual information fixes this. Mutual information $I(Z_{sa}; Z_{s'})$ measures how much knowing $Z_{sa}$ reduces uncertainty about $Z_{s'}$, or equivalently, how much knowing $Z_{s'}$ reduces uncertainty about $Z_{sa}$. Maximizing mutual information forces the encoder to ensure that $z_{sa}$ and $z_{s'}$ are not just numerically close but statistically dependent in a way that makes each informative about the other. If a dimension of the state is irrelevant to the dynamics, the mutual information objective provides no incentive to encode it (since it doesn't help predict the next state), whereas the MSE objective does (since predicting it reduces the Euclidean error). This means the mutual information objective acts as an implicit information bottleneck β€” it encourages the encoder to focus representational capacity on the dimensions that are actually predictive of future states.

Lemma 4.2 formalizes the downstream benefit. The lemma states that if $I(Z_{s'}; Z_{sa})$ increases, then the conditional entropy $H(Z_{s'} \mid Z_{sa})$ strictly decreases. This follows directly from the identity:

H(Zsβ€²βˆ£Zsa)=H(Zsβ€²)βˆ’I(Zsβ€²;Zsa)H(Z_{s'} \mid Z_{sa}) = H(Z_{s'}) - I(Z_{s'}; Z_{sa})

where $H(Z_{s'})$ is the marginal entropy of the next-state representation (a property of the environment and the encoder, fixed during optimization of the mutual information term). As $I(Z_{s'}; Z_{sa})$ increases, the conditional entropy $H(Z_{s'} \mid Z_{sa})$ β€” the remaining uncertainty about $Z_{s'}$ given $Z_{sa}$ β€” must decrease.

Why this matters for RL. Lower conditional entropy $H(Z_{s'} \mid Z_{sa})$ means the latent dynamics are more deterministic: given $z_{sa}$, the next-state representation $z_{s'}$ is more predictable. This directly tightens the theoretical bounds from DeepMDP (Gelada et al., 2019) and MR.Q, which show that the value prediction error is upper-bounded by a function of the latent transition and reward modeling errors. A more deterministic latent dynamics model means smaller transition modeling error, which means a tighter bound on value error, which ultimately enables more accurate value estimation and better policy performance.

The key design choice: maximizing mutual information while also minimizing MSE. DR.Q does not replace the MSE loss with the mutual information objective β€” it adds the mutual information objective alongside the MSE loss, as shown in Equation 3:

L=min⁑E[(Zsaβˆ’Zsβ€²)2]⏟latentΒ consistencyΒ lossβˆ’MI(Zsa;Zsβ€²)⏟mutualΒ informationΒ loss\mathcal{L} = \min \underbrace{\mathbb{E}[(Z_{sa} - Z_{s'})^2]}_{\text{latent consistency loss}} - \underbrace{MI(Z_{sa}; Z_{s'})}_{\text{mutual information loss}}

where the first term is the standard latent dynamics MSE and the second term is the mutual information (with a negative sign because the overall objective is minimized). The MSE term ensures predictive accuracy (the predicted next-state representation is numerically close to the actual one), while the mutual information term ensures informational coverage (the representations capture the statistical dependency structure of the dynamics). The combination addresses both failure modes: MSE alone can achieve low error by predicting irrelevant dimensions; mutual information alone could achieve high dependency without numerical accuracy (e.g., by making $z_{sa}$ and $z_{s'}$ perfectly correlated but in completely different regions of the representation space). Together, they require both accuracy and informativeness.

The InfoNCE Loss: A Tractable Lower Bound on Mutual Information

Computing mutual information $I(Z_{sa}; Z_{s'})$ directly is intractable for high-dimensional continuous representations because it requires integrating over the joint and marginal distributions, which are unknown and must be estimated from finite samples. DR.Q instead uses the InfoNCE loss (Oord et al., 2018), which provides a variational lower bound on mutual information. The specific form used is given in Equation 8:

LI(z^sβ€²,z~sβ€²)=βˆ’1Nβˆ‘i=1Nlog⁑[exp⁑(cos⁑(z^sβ€²i,z~sβ€²i)/Ο„)βˆ‘k=1Nexp⁑(cos⁑(z^sβ€²i,z~sβ€²k)/Ο„)]L_I(\hat{z}_{s'}, \tilde{z}_{s'}) = -\frac{1}{N} \sum_{i=1}^N \log \left[ \frac{\exp(\cos(\hat{z}_{s'}^i, \tilde{z}_{s'}^i) / \tau)}{\sum_{k=1}^N \exp(\cos(\hat{z}_{s'}^i, \tilde{z}_{s'}^k) / \tau)} \right]

where $\hat{z}_{s'}^i$ is the predicted next-state representation for the $i$-th sample in the batch, $\tilde{z}_{s'}^i$ is the actual (target) next-state representation for the same sample, $\tilde{z}_{s'}^k$ for $k \neq i$ are the actual next-state representations for other samples in the batch (the "negative" samples), $\cos(X, Y) = \frac{X \cdot Y}{\|X\| \|Y\|}$ is the cosine similarity, $\tau$ is a temperature hyperparameter (set to 0.1 in all experiments), and $N$ is the batch size (256).

What this loss computes. For each sample $i$ in the batch, the loss computes the cosine similarity between the predicted next-state representation $\hat{z}_{s'}^i$ and the true next-state representation $\tilde{z}_{s'}^i$ (the "positive pair"), divides by the temperature $\tau$, and exponentiates. It then divides this by the sum of similarly computed scores for all $N$ possible pairings between $\hat{z}_{s'}^i$ and every $\tilde{z}_{s'}^k$ in the batch (including the true pairing $k=i$ and all "negative" pairings $k \neq i$). The negative logarithm of this ratio is the loss for sample $i$. The final loss is the average over all $N$ samples.

What this operationalizes. The InfoNCE loss is a classification objective in disguise. It asks: given the predicted representation $\hat{z}_{s'}^i$, can the model correctly identify which of the $N$ candidate next-state representations $\{\tilde{z}_{s'}^1, \ldots, \tilde{z}_{s'}^N\}$ is the true one? The numerator measures compatibility between the prediction and the true next state; the denominator measures compatibility between the prediction and all candidate next states (including the true one). Maximizing the log-ratio (equivalently, minimizing the negative log-ratio) encourages the model to assign high compatibility to the true pairing and low compatibility to all false pairings.

Why cosine similarity and not dot product. Cosine similarity normalizes the representations to unit length before computing the dot product. This means the compatibility score depends only on the direction of the representations, not their magnitudes. This is important because the MSE loss already constrains the magnitudes (encouraging $\|\hat{z}_{s'}\| \approx \|\tilde{z}_{s'}\|$). By using cosine similarity, the InfoNCE loss focuses exclusively on the angular alignment β€” it asks whether the predicted and true next-state representations point in the same direction in representation space, independent of their lengths. This prevents the two losses from interfering: MSE handles magnitude, InfoNCE handles direction.

Why this is a lower bound on mutual information. The theoretical result from Oord et al. (2018) and Poole et al. (2019) states that $I(X; Y) \geq \log(N) - L_{\text{InfoNCE}}$ for some random variables $X, Y$. This means that minimizing the InfoNCE loss maximizes a lower bound on the mutual information. The bound becomes tighter as $N$ (the number of negative samples) increases. In DR.Q, $N = 256$ (the batch size), which provides 255 negative samples per positive pair β€” sufficient to make the bound informative in practice.

The temperature $\tau$ controls concentration. The temperature parameter $\tau = 0.1$ governs how sharply the loss distinguishes between positive and negative pairs. A smaller $\tau$ makes the softmax distribution peakier β€” small differences in cosine similarity get amplified, making the loss focus more on "hard negatives" (negative samples that are similar to the positive). A larger $\tau$ makes the distribution more uniform. The value 0.1 is relatively small, indicating that DR.Q uses a "hard" contrastive objective that strongly penalizes confusing nearby but incorrect next-state representations.

The computation is performed in the batch. A crucial implementation detail: the negative samples are the other samples in the same minibatch, not samples from a separate memory bank. This means no additional storage or computation is required beyond what is already available from the standard training loop. For each of the $N$ samples in the batch, we have both $\hat{z}_{s'}^i$ (from the MDP predictor) and $\tilde{z}_{s'}^i$ (from the target state encoder applied to the actual next state). The InfoNCE loss is computed entirely from these $N$ pairs, using all $N^2$ possible pairings. The computational cost is $O(N^2 \cdot d)$ where $d = 512$ is the representation dimension, which is negligible compared to the forward and backward passes through the encoder and critic networks.

The loss weight $\lambda_m = 0.1$. The InfoNCE loss is not added at full strength β€” it is multiplied by a coefficient $\lambda_m = 0.1$ in the overall encoder loss (Equation 9). This ensures that the mutual information objective does not dominate the MSE objective. If $\lambda_m$ were too large, the encoder might learn representations that are highly mutually informative but numerically inaccurate (the complement of the failure mode DR.Q is designed to fix). The value 0.1 was presumably chosen through empirical tuning to balance the two objectives, though the paper does not report a sweep over this hyperparameter.

Relationship to contrastive learning in computer vision. The InfoNCE loss used here is closely related to the contrastive objectives popular in self-supervised visual representation learning (SimCLR, Chen et al., 2020; CPC, Oord et al., 2018). The difference is that in vision, the positive pairs are typically two augmented views of the same image, and the negative pairs are different images. Here, the positive pair is a predicted next state and the actual next state from the same transition, and the negative pairs are predicted-actual pairings from different transitions. This is a natural fit for temporal data: the dynamics of the environment provide a natural notion of which representations should be similar (those from consecutive timesteps of the same trajectory) and which should be dissimilar (those from different trajectories or well-separated timesteps).

Faded Prioritized Experience Replay

The second major component of DR.Q is the faded prioritized experience replay (faded PER), a sampling strategy that combines TD-error-based prioritization with temporal decay to address the primacy bias in representation learning.

Standard PER and its limitation. In standard prioritized experience replay (Schaul et al., 2015), each transition $e_i$ in the replay buffer is assigned a sampling probability proportional to its TD error magnitude:

p(i)=∣δ(i)∣α+ΞΊβˆ‘j(∣δ(j)∣α+ΞΊ)p(i) = \frac{|\delta(i)|^\alpha + \kappa}{\sum_j (|\delta(j)|^\alpha + \kappa)}

where $\delta(i)$ is the TD error of transition $e_i$, $\alpha$ smooths out extremes (set to 0.4 in DR.Q), and $\kappa$ is a small constant preventing zero probability (effectively 0 in LAP, the variant DR.Q uses). The intuition is that transitions with large TD errors are "surprising" β€” the value function's prediction was far from the observed outcome β€” and therefore deserve more attention.

Why standard PER exacerbates primacy bias. Early in training, when the value function is poorly initialized and the policy is essentially random, many transitions will have large TD errors simply because the agent has not yet learned the environment dynamics. If the replay buffer is large (1 million transitions in DR.Q) and training continues for hundreds of thousands of steps, these early high-error transitions will persist in the buffer and continue to be sampled frequently long after they have become obsolete β€” the value function has converged past them, but the stored TD error (computed at the time the transition was added) still marks them as high-priority. This creates a feedback loop where outdated experiences dominate training, slowing adaptation to the current policy distribution.

The forget mechanism and its limitation. The forget mechanism (Novati & Koumoutsakos, 2019; Kang et al., 2025) addresses this by assigning sampling probability based on recency:

P(i)=(1βˆ’Ο΅)iβˆ‘j(1βˆ’Ο΅)jP(i) = \frac{(1-\epsilon)^i}{\sum_j (1-\epsilon)^j}

where $i = 0$ is the newest transition and $\epsilon \in (0, 1)$ controls the decay rate (set to 0.0001 in DR.Q). The probability decays exponentially with age, so old transitions are gradually "forgotten" regardless of their TD error. This successfully mitigates primacy bias β€” old, obsolete experiences stop being sampled β€” but creates a new problem: it ignores TD error magnitude entirely. A recent transition with a tiny TD error (the value function already fits it well) gets sampled more frequently than an older transition with a large TD error that might still provide useful learning signal, especially for rare but important state transitions.

Faded PER: the multiplicative combination. DR.Q's faded PER combines both signals multiplicatively, as shown in Equation 4:

P(i)=∣δ(i)∣α(1βˆ’Ο΅)iβˆ‘j∣δ(j)∣α(1βˆ’Ο΅)jP(i) = \frac{|\delta(i)|^\alpha (1-\epsilon)^i}{\sum_j |\delta(j)|^\alpha (1-\epsilon)^j}

where $|\delta(i)|^\alpha$ is the TD-error-based priority (the numerator of standard PER with $\kappa=0$), and $(1-\epsilon)^i$ is the temporal decay factor.

What this accomplishes. The sampling probability is the product of two independent signals: one measuring how informative the transition is for value learning (TD error), and one measuring how relevant it is to the current policy distribution (recency). This means:

  • A recent transition with large TD error gets the highest probability (both factors are large).
  • An old transition with large TD error gets intermediate probability (the TD-error factor is large, but the temporal decay factor is small) β€” it is still sampled, but less frequently than it would be under standard PER.
  • A recent transition with small TD error gets low-to-intermediate probability (the recency factor is large, but the TD-error factor is small) β€” it is sampled less frequently than it would be under a pure forget mechanism.
  • An old transition with small TD error gets the lowest probability (both factors are small).

Why multiplicative rather than additive. An additive combination $c_1 |\delta(i)|^\alpha + c_2 (1-\epsilon)^i$ would allow a transition to be sampled frequently if either its TD error is large or it is recent, even if the other factor is near zero. This would not solve the problem: old high-error transitions would still be over-sampled (due to the additive contribution of the TD-error term), and recent low-error transitions would still be over-sampled (due to the additive contribution of the recency term). The multiplicative form requires both factors to be non-negligible for a transition to receive high priority, which is exactly the desired behavior.

Practical modifications for numerical stability (Equation 10). The theoretical form in Equation 4 has two practical issues:

  1. If $|\delta(i)| = 0$, the priority is zero β€” the transition would never be sampled, which is undesirable because even transitions with zero current TD error might become relevant as the value function changes.
  2. The exponential decay $(1-\epsilon)^i$ can become extremely small for large $i$, potentially underflowing to zero in floating-point arithmetic.

DR.Q addresses both issues with the practical sampling probability given in Equation 10:

P(i)=max⁑(∣δ(i)∣α,1)β‹…max⁑(Ο΅low,(1βˆ’Ο΅)i)βˆ‘jmax⁑(∣δ(j)∣α,1)β‹…max⁑(Ο΅low,(1βˆ’Ο΅)j)P(i) = \frac{\max(|\delta(i)|^\alpha, 1) \cdot \max(\epsilon_{\text{low}}, (1-\epsilon)^i)}{\sum_j \max(|\delta(j)|^\alpha, 1) \cdot \max(\epsilon_{\text{low}}, (1-\epsilon)^j)}

where $\alpha = 0.4$ is the LAP smoothing coefficient, $\epsilon_{\text{low}} = 0.1$ is a lower bound on the forget weight, and $\epsilon = 0.0001$ is the decay rate.

What the modifications do. The $\max(|\delta(i)|^\alpha, 1)$ term ensures a minimum priority of 1 (following LAP, Fujimoto et al., 2020), which guarantees that every transition has a non-zero probability of being sampled. The $\max(\epsilon_{\text{low}}, (1-\epsilon)^i)$ term clips the forget weight at 0.1, preventing it from decaying below this threshold. This ensures that even very old transitions retain at least 10% of their original temporal weight, so they are not completely forgotten β€” they can still be sampled if their TD error is large enough. The choice of $\epsilon_{\text{low}} = 0.1$ means that the oldest transition in the buffer has a temporal weight exactly 0.1, while the newest transition has temporal weight 1.0 (a 10Γ— ratio).

The decay rate $\epsilon = 0.0001$ controls the rate of forgetting. With this decay rate, the temporal weight halves after approximately $\ln(2)/\epsilon \approx 6931$ steps. Since the replay buffer has capacity 1 million and the agent takes 1 million environment steps in total, the oldest transitions have temporal weight $(1 - 0.0001)^{1000000} \approx e^{-100} \approx 3.7 \times 10^{-44}$, which would be effectively zero without the $\epsilon_{\text{low}} = 0.1$ clipping. The clipping ensures that even the oldest transitions remain accessible (at 10% weight) throughout training.

Theorem 4.3 formalizes the properties of faded PER. The theorem establishes three guarantees:

  • (i) Recency dominance for equal TD errors: If two transitions have the same TD error magnitude, the newer one has strictly higher sampling probability. This follows directly from $(1-\epsilon)^{i_1} > (1-\epsilon)^{i_2}$ when $i_1 < i_2$.
  • (ii) Relationship to standard PER: The faded PER probability $P(i)$ is lower-bounded by the standard PER probability $\hat{P}(i) = |\delta(i)|^\alpha / \sum_j |\delta(j)|^\alpha$ multiplied by $(1-\epsilon)^i$, and upper-bounded by $1 / (1 + C(1-\epsilon)^{k-i})$ for some $C > 0, k \in \mathbb{N}$. The lower bound shows that faded PER never samples a transition more frequently than standard PER would (the temporal decay always reduces probability). The upper bound shows that the probability is always strictly less than 1, and that it decreases as the transition ages (as $k-i$ increases).
  • (iii) Bounded expected sample count: The expected number of times a transition $e_i$ is sampled in a batch of size $N$ satisfies $0 < \mathbb{E}[n_i] \leq N / (1 + C(1-\epsilon)^{k-i}) < N$. This guarantees that no single transition can dominate the sampling distribution β€” even the newest, highest-TD-error transition has expected sample count strictly less than $N$.

Why these guarantees matter. Theorem 4.3 provides theoretical reassurance that faded PER does not degenerate: no transition is ever completely excluded (positive lower bound on expected sample count), no transition can monopolize the batch (expected sample count strictly less than batch size), and the temporal ordering is respected when TD errors are equal. This makes the sampling strategy well-behaved and predictable, which is important for training stability.

Encoder Training: The Full Objective

The encoder training in DR.Q combines three loss terms and operates on 5-step latent rollouts. The full encoder loss is given in Equation 9:

LencDR.Q=βˆ‘t=1HΞ»rLreward(r^t,rt)+Ξ»dLdynamics(z^sβ€²,t,z~sβ€²,t)+Ξ»mLI(z^sβ€²,t,z~sβ€²,t)L^{\text{DR.Q}}_{\text{enc}} = \sum_{t=1}^H \lambda_r L_{\text{reward}}(\hat{r}_t, r_t) + \lambda_d L_{\text{dynamics}}(\hat{z}_{s',t}, \tilde{z}_{s',t}) + \lambda_m L_I(\hat{z}_{s',t}, \tilde{z}_{s',t})

where $H = 5$ is the encoder horizon, $\lambda_r = 0.1$ is the reward loss weight, $\lambda_d = 1$ is the dynamics loss weight, $\lambda_m = 0.1$ is the InfoNCE loss weight, $t$ indexes the step in the unrolled horizon, $\hat{r}_t$ is the predicted reward at step $t$, $r_t$ is the true reward, $\hat{z}_{s',t}$ is the predicted next-state representation, and $\tilde{z}_{s',t}$ is the actual next-state representation from the target state encoder.

The three loss components in detail:

1. Reward loss (Equation 6):

Lreward(r^,r)=CE(r^,TwoHot(r))L_{\text{reward}}(\hat{r}, r) = \text{CE}(\hat{r}, \text{TwoHot}(r))

where $\hat{r}$ is the predicted reward (a vector of logits over 65 bins), $\text{TwoHot}(r)$ is the two-hot encoding of the true scalar reward $r$ into a probability distribution over the same 65 bins, and CE is the cross-entropy loss.

What this computes. The true scalar reward $r$ is converted to a "two-hot" encoding β€” a vector of length 65 where at most two adjacent elements are non-zero, with values summing to 1, representing the reward magnitude via a soft assignment to reward bins. The bins are non-uniformly spaced according to the symexp transformation: $\text{symexp}(x) = \text{sign}(x)(\exp(|x|) - 1)$. This means that bins near zero are more densely packed than bins far from zero, providing higher precision for small rewards (which are common) while still covering a wide range (the paper specifies the reward range as $[-10, 10]$). The predicted reward $\hat{r}$ is a vector of 65 logits (one per bin), and the cross-entropy loss encourages it to match the two-hot distribution.

Why two-hot encoding. Two-hot encoding provides a middle ground between regression (predicting a scalar directly, which can be unstable with varying reward scales) and full categorical classification (predicting a one-hot bin, which loses information about reward magnitude within a bin). The two-hot encoding retains some of the "softness" of a continuous distribution β€” the reward is represented as a mixture of two adjacent bins, encoding both the approximate magnitude and the precise offset within the bin. This has been shown to improve robustness to reward scale variations (Hafner et al., 2023) and is adopted from MR.Q.

2. Dynamics loss (Equation 7):

Ldynamics(z^sβ€²,z~sβ€²)=E[(z^sβ€²βˆ’SG(z~sβ€²))2]L_{\text{dynamics}}(\hat{z}_{s'}, \tilde{z}_{s'}) = \mathbb{E}[(\hat{z}_{s'} - \text{SG}(\tilde{z}_{s'}))^2]

where $\hat{z}_{s'}$ is the predicted next-state representation (from the MDP predictor applied to $z_{sa}$), $\tilde{z}_{s'}$ is the actual next-state representation from the target state encoder $f_{\omega'}$, and SG is the stop-gradient operator.

What this computes. This is a standard mean-squared error between the predicted and actual next-state representations, but with a crucial asymmetry: the gradient flows only through $\hat{z}_{s'}$, not through $\tilde{z}_{s'}$. The stop-gradient on $\tilde{z}_{s'}$ means that the encoder parameters are updated to make the prediction $\hat{z}_{s'}$ match the target $\tilde{z}_{s'}$, but the target encoder is never updated to make its output easier to predict. This prevents a degenerate solution where both the predictor and the target encoder collapse to a trivial representation (e.g., always outputting zero).

Why the target encoder is separate. The target state encoder $f_{\omega'}$ is a lagged copy of the online state encoder $f_\omega$, updated every $T_{\text{target}} = 250$ environment steps by copying $\omega \to \omega'$. This is the same technique used in DQN and DDPG to stabilize value function learning. Here, it serves the analogous purpose of providing a stable target for the dynamics prediction: if the target changed every gradient step, the prediction objective would be chasing a moving target, potentially causing oscillations or collapse.

3. InfoNCE loss (Equation 8): Already described in detail above.

Why the loss weights are $(1.0, 0.1, 0.1)$. The dynamics loss has weight $\lambda_d = 1.0$, making it the dominant term. The reward loss and InfoNCE loss both have weight $\lambda_r = \lambda_m = 0.1$, making them auxiliary objectives that modulate the primary dynamics-consistency signal. This weighting reflects the paper's design philosophy: the dynamics consistency (MSE) is the primary objective inherited from prior work; the mutual information and reward prediction are auxiliary signals that improve the quality of the learned representations without dominating the optimization. The fact that the same weights work across all 73 tasks (from simple MuJoCo environments to complex HumanoidBench with dexterous hands) is remarkable and speaks to the robustness of this balance.

The horizon-based unrolling. The loss is computed at each step $t = 1, \ldots, 5$ of the latent rollout, and the terms are summed. This means the encoder receives gradients from predictions at multiple timescales: the 1-step dynamics (predicting the immediate next state), 2-step dynamics (predicting two steps ahead through the latent model), and so on up to 5 steps. This multi-step training encourages the encoder to learn representations that support compositional prediction β€” representations where chaining the dynamics model multiple times does not cause rapid divergence. This is important because the critic in DR.Q uses multi-step returns ($H_Q = 3$, see Section 4.3.2), which implicitly relies on the latent dynamics being accurate over multiple steps.

Encoder network architecture details. The state encoder $f_\omega$ and state-action encoder $g_\omega$ share parameters $\omega$. The paper specifies (Table 5): hidden dimension 750 (meaning the intermediate layers of the encoder networks have width 750), output dimensions 512 for both $z_s$ and $z_{sa}$, and an additional $z_a$ dimension of 256 (used only internally in the architecture β€” likely an intermediate representation of the action before combining with the state representation). The activation function is ELU (Clevert et al., 2015). The optimizer is AdamW (Loshchilov & Hutter, 2019) with learning rate $3 \times 10^{-4}$ and weight decay 0.01. Weights are initialized with Xavier uniform (Glorot & Bengio, 2010) and biases are initialized to zero.

Target network updates. The target state encoder $f_{\omega'}$ is updated every $T_{\text{target}} = 250$ environment steps, along with the target critic networks and target policy network. This periodic "hard" update (full copy, no Polyak averaging) is less common in modern actor-critic methods (which typically use soft updates $\theta' \leftarrow \tau \theta + (1-\tau)\theta'$ with $\tau \ll 1$), but follows the practice established in TD7 and MR.Q. The justification is likely that with a replay ratio of 1 (one gradient step per environment step), hard updates at 250-step intervals provide sufficient stability without the complexity of managing a moving average.

Actor-Critic Training

Once representations are learned, the actor and critic networks operate entirely on the encoded representations. The training follows standard off-policy actor-critic methods with several specific design choices documented in Section 4.3.2.

Critic architecture and objective. DR.Q uses two critic networks $Q_{\theta_1}, Q_{\theta_2}$ with identical architecture: hidden dimension 512, ELU activation, trained with AdamW (learning rate $3 \times 10^{-4}$, no weight decay specified for critics). The critic loss is given in Equation 13:

Lcritics=Huber(QΞΈi,βˆ‘t=0HQβˆ’1Ξ³trt+Ξ³HQmin⁑j=1,2QΞΈjβ€²)L_{\text{critics}} = \text{Huber}\left(Q_{\theta_i}, \sum_{t=0}^{H_Q-1} \gamma^t r_t + \gamma^{H_Q} \min_{j=1,2} Q_{\theta'_j}\right)

where $Q_{\theta_i} = Q_{\theta_i}(z_{s_0 a_0})$ is the critic's prediction for the current state-action representation, $H_Q = 3$ is the multi-step return horizon, $\gamma = 0.99$ is the discount factor, $\min_j Q_{\theta'_j}$ is the clipped double Q-learning target (taking the minimum over the two target critics evaluated at the $H_Q$-step future state-action pair), and Huber is the Huber loss (squared error for small residuals, absolute error for large residuals).

Why clipped double Q-learning. The minimum over two critics is a standard technique (Fujimoto et al., 2018) to combat overestimation bias β€” the tendency of Q-learning to systematically overestimate values because the max operator in the Bellman backup amplifies positive errors. By taking the minimum of two independently initialized critics, the target value is biased downward rather than upward, which counteracts the overestimation tendency. DR.Q uses two critics (not an ensemble of more), which provides sufficient bias reduction without excessive computational cost.

Why Huber loss. The Huber loss is used instead of MSE because LAP (the prioritized replay variant DR.Q uses) introduces non-uniform sampling, which changes the effective loss function that is being optimized. Fujimoto et al. (2020) showed that under non-uniform sampling, the Huber loss provides better statistical properties than MSE for value function learning. The Huber loss is quadratic for errors below a threshold (typically 1.0) and linear for errors above, making it more robust to outliers than MSE while still providing strong gradients for small errors.

Why multi-step returns ($H_Q = 3$). Multi-step returns compute the target value using the sum of $H_Q$ actual rewards plus the bootstrapped value at step $H_Q$, rather than the standard 1-step TD target $r + \gamma V(s')$. This reduces the dependence on the (potentially inaccurate) value function by relying more on actual observed rewards, which accelerates learning when the value function is poor. The horizon of 3 steps is a moderate choice that balances the bias of 1-step returns (high dependence on value function accuracy) against the variance of long multi-step returns (more reward terms mean more noise).

Actor architecture and objective. The actor network $\pi_\phi$ is a deterministic policy with hidden dimension 512, ReLU activation, trained with AdamW (learning rate $3 \times 10^{-4}$). The actor objective is given in Equation 12:

Lactor=βˆ’12βˆ‘i=1,2QΞΈi(zsaΟ€)L_{\text{actor}} = -\frac{1}{2} \sum_{i=1,2} Q_{\theta_i}(z_{sa_\pi})

where $z_{sa_\pi} = g_\omega(z_s, a_\pi)$ is the state-action representation for the actor's chosen action $a_\pi$. This is the standard deterministic policy gradient objective: maximize the average Q-value of the actions selected by the policy. Both critics are averaged (not minimum) to provide a more stable gradient signal β€” the minimum would be overly conservative for policy improvement.

Exploration via target policy smoothing. During both training and evaluation, Gaussian noise is added to the policy's output before execution, as specified in Equation 11:

aΟ€=clip(aβ€²,βˆ’1,1),aβ€²=πϕ′(zs)+clip(ψ,βˆ’c,c)a_\pi = \text{clip}(a', -1, 1), \quad a' = \pi_{\phi'}(z_s) + \text{clip}(\psi, -c, c)

where $\pi_{\phi'}$ is the target policy network, $\psi \sim \mathcal{N}(0, 0.2^2)$ is Gaussian exploration noise with standard deviation 0.2, and $c = 0.3$ clips the noise to $[-0.3, 0.3]$. The action is then clipped to $[-1, 1]$ (the valid action range for all environments). This technique, introduced in TD3 (Fujimoto et al., 2018), serves two purposes: during training, it provides exploration by injecting randomness into action selection; during critic updates, it smooths the value function by ensuring the critic is evaluated at actions near (but not exactly at) the policy's output, which reduces sensitivity to narrow peaks in the Q-function.

The exploration phase. For the first $T_{\text{explore}} = 10^4$ environment steps, the agent takes purely random actions (sampled from $\mathcal{N}(0, 0.2^2)$, clipped to $[-1, 1]$) to populate the replay buffer with diverse initial experiences. After this phase, the agent switches to the policy with exploration noise as described above. No training occurs during the exploration phase.

The Complete Training Loop

Algorithm 1 (Appendix B) specifies the full DR.Q training procedure. The main loop runs for $T$ total environment steps (1 million for MuJoCo, 500K for DMC and HumanoidBench, corresponding to 1M environment frames due to action repeat of 2 on the latter two benchmarks):

  1. Action selection: At each step, select action using the target policy with exploration noise (Equation 11). Execute the action, observe reward $r$, next state $s'$, and done flag $d$. Store the transition in the replay buffer.

  2. Encoder training (every $T_{\text{target}} = 250$ steps): After the exploration phase, every 250 environment steps:

    • Update target networks $\theta'_1, \theta'_2, \phi', \omega' \leftarrow \theta_1, \theta_2, \phi, \omega$ (hard copy).
    • For 250 gradient steps (matching the update interval β€” this means one gradient step per environment step on average, giving a replay ratio of 1):
      • Sample a batch of transitions using faded PER (Equation 10).
      • Compute the encoder loss via Equation 9 (5-step horizon rollout, reward loss, dynamics loss, InfoNCE loss summed).
      • Update encoder parameters $\omega$ via gradient descent.

  3. Critic training (every step after exploration): At each environment step after the exploration phase:

    • Sample a batch of transitions using faded PER.
    • Compute the critic loss via Equation 13 (clipped double Q-learning with 3-step returns and Huber loss).
    • Update critic parameters $\theta_1, \theta_2$ via gradient descent.
    • Update the LAP priorities in the replay buffer using the new TD errors.

  4. Actor training (every step after exploration): At each environment step after the exploration phase:

    • Using the same batch as the critic update, compute the actor loss via Equation 12 (average of both critics' Q-values).
    • Update actor parameters $\phi$ via gradient descent.

Why staggered updates. The encoder targets are updated every 250 steps, while the encoder parameters are updated every step (but at a rate of 1 update per step, not 250 updates per 250 steps β€” the encoder training loop runs for 250 gradient steps every 250 environment steps, which averages to 1 gradient step per environment step). The critic and actor are updated every step. This design ensures that the encoder has a stable target for its dynamics predictions (updated infrequently) while the actor and critic can adapt quickly to the latest representations.

Why replay ratio equals 1. Many recent sample-efficient RL algorithms use high replay ratios (UTD >> 1) β€” taking multiple gradient steps per environment step to extract more learning from each collected transition (e.g., REDQ uses UTD=20). DR.Q maintains a replay ratio of 1 (one gradient step per environment step for each of encoder, critic, and actor). This makes DR.Q computationally efficient β€” its wall-clock time per environment step is comparable to standard algorithms like TD3 β€” while still achieving high sample efficiency through better representations rather than more gradient steps. This is a deliberate design choice: the paper positions DR.Q as an improvement in learning efficiency (what you can learn from each transition) rather than computational efficiency (how many times you can process each transition).

The complete hyperparameter table (Table 5). Every numerical value defining DR.Q's behavior is specified in this table. Key values not yet covered:

  • Replay buffer size: 1,000,000 transitions
  • Batch size: 256
  • Discount factor $\gamma$: 0.99
  • Encoder $z_s$ dim: 512, $z_{sa}$ dim: 512, $z_a$ dim: 256
  • Encoder reward bins: 65, reward range: $[-10, 10]$
  • Encoder horizon $H$: 5
  • Critic multi-step return horizon $H_Q$: 3
  • Optimizer: AdamW for all networks, learning rate $3 \times 10^{-4}$
  • Weight initialization: Xavier uniform, bias initialization: 0

What DR.Q notably does NOT include. The paper explicitly states (Section 4.3.2):

"DR.Q does not include components such as normalizing target values or input states, parameter reset, regularizing hidden embeddings, etc."

This is significant because many competing algorithms (BRO, SimBaV2, FoG) rely on auxiliary techniques like periodic network resets, layer normalization, or distributional value functions to achieve their performance. DR.Q achieves competitive or superior results with a relatively simple recipe: model-based representation learning with mutual information maximization, faded PER, and standard clipped double Q-learning. The simplicity makes the contribution clearer β€” the gains can be attributed to the two proposed mechanisms (InfoNCE loss and faded PER) rather than a tangle of interacting components.

4. Key Insights and Innovations

Innovation 1: Reframing Model-Based Representation Learning as an Information-Theoretic Problem

The paper's most fundamental intellectual move is diagnosing why the standard latent dynamics consistency objective fails β€” not just observing that it sometimes underperforms, but providing a precise mathematical explanation (Theorem 4.1) for a failure mode that had gone unrecognized in prior work. Before DR.Q, the dominant assumption in model-based representation learning β€” spanning DeepMDP (Gelada et al., 2019), TD7 (Fujimoto et al., 2023), MR.Q (Fujimoto et al., 2025), and the representation-learning components of model-based systems like TDMPC2 (Hansen et al., 2024) β€” was that minimizing the Euclidean distance between a predicted next-state representation and the actual next-state representation was sufficient to produce useful representations. The reasoning was intuitively appealing: if the encoder can predict where the system will go next in latent space, it must have captured the relevant dynamics.

DR.Q proves this intuition is mathematically unfounded. The two counterexamples in Theorem 4.1 β€” the independent-noise case where MSE and mutual information move in opposite directions, and the deterministic-scaling case where they move together β€” establish that there is no monotonic relationship between the two quantities. The practical implication is profound: an encoder can achieve low MSE by learning to predict task-irrelevant state dimensions (e.g., precise finger joint angles during a running task) while neglecting control-critical features (e.g., overall body orientation and velocity). The resulting representations are numerically accurate but informationally impoverished β€” they don't help the critic distinguish good states from bad ones or help the actor select appropriate actions.

What distinguishes this from a routine architectural tweak is that it shifts the success criterion for representation learning from a geometric notion (distance in latent space) to an information-theoretic one (mutual information). This is a conceptual reframing, not just a new loss function. The InfoNCE loss is the implementation, but the insight is that representation learning for control requires statistical dependency between current and future representations, not just numerical proximity. Lemma 4.2 operationalizes why this matters for RL specifically: maximizing mutual information reduces the conditional entropy of the next-state representation given the current one, which tightens the theoretical value-error bounds from DeepMDP and MR.Q. The paper thus provides both a negative result (MSE alone is insufficient β€” a diagnostic insight) and a constructive response (mutual information maximization fills the gap β€” a prescriptive insight), forming a complete conceptual package rather than an isolated technique.

The significance extends beyond the specific InfoNCE implementation. By identifying that the MSE objective has a blind spot β€” it cannot distinguish between informative and uninformative dimensions β€” the paper opens a broader research direction: what other representation-learning objectives might suffer from similar information-theoretic gaps? Could dynamics consistency be better enforced through optimal transport, Wasserstein distances, or other measures that capture distribution-level alignment rather than point-wise error? The paper doesn't answer these questions, but its reframing makes them natural to ask.

Innovation 2: The Concept of "Debiasing" as a Unifying Principle for Representation and Replay Design

DR.Q's second distinctive contribution is framing both of its technical components β€” the InfoNCE loss and the faded PER β€” as instances of bias correction, not as ad-hoc improvements. This framing is more than rhetorical: it reveals a structural connection between two seemingly unrelated problems (the encoder objective and the sampling strategy) that prior work had treated independently.

Consider the two biases the paper identifies. Bias 1 (the informativeness gap) arises because the MSE objective systematically underestimates the importance of statistical dependency between representations β€” it "biases" the encoder toward representing predictable-but-irrelevant dimensions of the state space. Bias 2 (the primacy bias) arises because standard replay strategies systematically over-represent either old high-error transitions (PER) or recent low-error transitions (forget mechanism alone) β€” they "bias" the training distribution away from the mixture of recency and informativeness that would be optimal. Both biases share a common structure: they are systematic distortions introduced by design choices in the learning algorithm, not random noise or environment stochasticity. And both are corrected not by adding complexity but by restoring balance β€” the InfoNCE loss balances the MSE's geometric focus with information-theoretic coverage; faded PER balances TD-error prioritization with temporal decay.

What makes this framing intellectually productive is that it unifies the evaluation criteria for representation learning components. Rather than asking "does this new loss function improve performance?" (an empirical question), the debiasing lens asks "what systematic distortion does the current objective introduce, and what complementary signal would correct it?" This turns algorithm design from a search over components into a diagnostic process: identify the bias, then design the correction. The paper demonstrates this process explicitly for two biases, but the framework suggests a methodology for future work β€” look for other places where the learning algorithm's inductive biases systematically distort the learned representations or the training distribution, and design targeted corrections.

The debiasing framing also explains why DR.Q's components are additive rather than replacement-based. DR.Q does not discard the MSE loss (replacing it with InfoNCE) or discard TD-error prioritization (replacing it with temporal decay). It adds the complementary signal in each case. This is philosophically distinct from the "our new method beats the old method" narrative common in RL papers: DR.Q argues that MR.Q's components are not wrong but incomplete, and that the fix is to supplement rather than supplant. This is a more nuanced and intellectually honest position β€” it acknowledges the value of prior work while precisely identifying its limitations.

Innovation 3: Empirical Demonstration That Mutual Information Maximization and Decayed Prioritization Are Complementary Scaling Axes

The paper's third contribution is the empirical finding that the InfoNCE loss and faded PER provide complementary benefits, with each addressing distinct failure modes that are most pronounced in different environments. This is not a claim the paper makes explicitly in theoretical terms, but it emerges clearly from the ablation studies (Figure 4, Figures 11-12 in Appendix E.1).

The ablation results reveal a pattern: the InfoNCE loss matters most in high-dimensional environments with redundant state information, while faded PER matters most when early experiences are poor and diverse exploration is needed. Specifically, removing the InfoNCE loss (Figure 4, top row; Figure 11) causes the largest performance drops on HumanoidBench tasks β€” particularly those with dexterous hands (h1hand-walk, h1hand-stand), where the state space contains finger joint angles that are largely irrelevant for locomotion. This aligns with the theoretical motivation: in high-dimensional environments with many task-irrelevant dimensions, the MSE objective has the most "room" to fit irrelevant features, making the mutual information correction most valuable. Conversely, removing LAP or the forget mechanism (Figure 4, bottom row; Figure 12) causes the largest drops on tasks requiring sustained exploration β€” h1hand-walk, humanoid-run β€” where early random experiences are poor and the agent needs to balance learning from rare informative transitions (which PER helps with) against overfitting to obsolete early data (which the forget mechanism prevents).

This complementarity is significant because it suggests that DR.Q's two components are not redundant β€” they solve different problems that manifest to different degrees across environments. A method that only added the InfoNCE loss (but kept standard PER) would still suffer from primacy bias in exploration-heavy tasks. A method that only used faded PER (but kept only the MSE loss) would still learn impoverished representations in high-dimensional environments. The fact that both components contribute independently to performance, and that their contributions are most visible in different types of tasks, validates the paper's diagnosis of two distinct biases rather than two symptoms of the same underlying issue.

This finding also has implications for algorithm selection in practice. For practitioners working in low-dimensional environments (e.g., classic MuJoCo tasks where the state space is under 20 dimensions and all dimensions are control-relevant), the InfoNCE loss may provide only marginal benefits β€” the MSE loss alone may be sufficient because there are few irrelevant dimensions to overfit. The paper's results on MuJoCo (Table 6: DR.Q slightly underperforms SimBaV2 on HalfCheetah and Walker2d) hint at this. Conversely, for high-dimensional robotics tasks (HumanoidBench with hands), the InfoNCE loss is critical. Similarly, for tasks where the agent quickly converges to a good policy (e.g., simple DMC tasks), faded PER may matter less because early experiences are not catastrophically poor. This task-dependent sensitivity is an empirical finding that enriches the conceptual debiasing framework β€” it shows when each bias correction matters, not just that it matters in aggregate.

Innovation 4: A Practical Demonstration of Single-Hyperparameter Generality Across Diverse Continuous Control Domains

While not conceptually novel in the abstract (many papers claim generality), DR.Q's demonstration of competitive-or-better performance across 73 tasks spanning three benchmarks with zero algorithmic changes is an empirical contribution that distinguishes it from prior work with similar claims. The paper's comparison table (Table 4 in Appendix C.2) reveals that several strong baselines that claim single-hyperparameter generality actually modify algorithmic configurations across tasks: SimBa and SimBaV2 switch between clipped double Q-learning and single-Q depending on the domain; FoG modifies batch size and reset schedules. DR.Q genuinely uses the same hyperparameters and the same algorithmic choices across all 73 tasks.

What makes this intellectually significant is that it validates the debiasing hypothesis in a way that task-specific tuning cannot. If DR.Q required different loss weights for the InfoNCE term on HumanoidBench versus MuJoCo, that would undermine the claim that the InfoNCE loss addresses a general bias in the MSE objective β€” it might instead be compensating for some domain-specific idiosyncrasy. The fact that a single InfoNCE weight (Ξ»m = 0.1) works across low-dimensional MuJoCo tasks (17-376 observation dimensions), moderately complex DMC tasks, and high-dimensional HumanoidBench tasks with dexterous hands (up to 308 observation dimensions) supports the claim that the bias is structural, not environment-specific. Similarly, the single decay rate (Ξ΅ = 0.0001) and forget weight threshold (Ξ΅low = 0.1) work across all environments, suggesting that the primacy bias operates similarly regardless of state dimensionality or task complexity.

The practical implication is that DR.Q lowers the barrier to entry for applying model-based representation learning to new domains. MR.Q already demonstrated that a single hyperparameter set could work across diverse benchmarks; DR.Q shows that the addition of debiasing components does not compromise this generality β€” it enhances it, achieving stronger performance without introducing new per-task tuning knobs. This is a non-trivial empirical finding: many proposed improvements to RL algorithms (distributional critics, normalization schemes, architecture modifications) require per-task adjustment of their corresponding hyperparameters to realize their benefits. DR.Q's components do not.

This generality also makes DR.Q a strong baseline for future research. A new algorithm that beats DR.Q on a subset of tasks but requires task-specific hyperparameters has a higher burden of proof β€” it must demonstrate that the performance gain is due to algorithmic improvements rather than hyperparameter overfitting. DR.Q sets a high bar: single-configuration, simple (no parameter resets, no distributional critics beyond two-hot rewards, no layer normalization), and broadly competent.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on 73 continuous control tasks drawn from three standard benchmarks: Gym MuJoCo (5 tasks, v4 versions: Ant, HalfCheetah, Hopper, Humanoid, Walker2d; 1M environment steps with no action repeat), the DMC suite (28 proprioceptive tasks split into DMC-Easy [21 tasks] and DMC-Hard [7 dog and humanoid tasks], plus 12 visual DMC tasks; 500K agent steps with action repeat 2, equivalent to 1M environment frames), and HumanoidBench (28 tasks: 14 without dexterous hands and 14 with dexterous hands, using the Unitree H1 humanoid; 500K agent steps with action repeat 2, equivalent to 1M environment frames). The tasks span wide complexity β€” from low-dimensional MuJoCo environments (11–376 observation dimensions) to high-dimensional humanoid manipulation with dexterous hands (up to 308 observation dimensions). The MuJoCo and DMC tasks use standard reward structures; HumanoidBench tasks vary in reward scales and are normalized by task success scores (Appendix C.1, Tables 1–3).

  • Base model(s). DR.Q uses a standard deterministic actor-critic architecture with two critic networks, a state encoder f_Ο‰ (output dimension 512, hidden dimension 750, ELU activation), a state-action encoder g_Ο‰ (output dimension 512, with an additional internal action representation of dimension 256), and a linear MDP predictor M. All networks use AdamW optimizer with learning rate 3 Γ— 10⁻⁴, Xavier uniform initialization, and bias initialization to 0. The architecture is built directly on MR.Q (Fujimoto et al., 2025) and is described as "representative of the capabilities of many contemporary deep RL architectures" β€” the paper chooses this base because MR.Q already established strong single-hyperparameter-set performance, allowing DR.Q's two modifications (InfoNCE loss and faded PER) to be evaluated cleanly without confounding architectural innovations.

  • Metrics. The primary metric is average undiscounted episode return over 10 random seeds, reported with 95% bootstrap confidence intervals. For cross-task aggregation, returns are normalized: MuJoCo tasks use TD3-normalized scores (x βˆ’ random_score) / (TD3_score βˆ’ random_score), DMC tasks divide the raw return by 1000, and HumanoidBench tasks use success-normalized scores (x βˆ’ random_score) / (task_success_score βˆ’ random_score). The paper reports both per-task learning curves and aggregate statistics (mean, median, interquartile mean [IQM]) computed over normalized scores using the rliable library (Agarwal et al., 2021) with bootstrapped confidence intervals over n Γ— T samples where n is the number of seeds and T is the number of tasks.

  • Baselines. The paper compares against a comprehensive set of model-free and model-based algorithms: PPO (Schulman et al., 2017), TD3+OFE (Ota et al., 2020), TQC (Kuznetsov et al., 2020), REDQ (Chen et al., 2021), DroQ (Hiraoka et al., 2021), DreamerV3 (Hafner et al., 2023), DrQ-v2 (Yarats et al., 2021a), TD7 (Fujimoto et al., 2023), TDMPC2 (Hansen et al., 2024), CrossQ (Bhatt et al., 2024), iQRL (Scannell et al., 2024a), BRO (Nauman et al., 2024), MAD-TD (Voelcker et al., 2025), SimBa (Lee et al., 2025a), SimBaV2 (Lee et al., 2025b), MR.Q (Fujimoto et al., 2025), and FoG (Kang et al., 2025). Not all baselines are run on all benchmarks: the paper uses previously published results where available (with appropriate citation) and runs MR.Q, SimBa, SimBaV2, and FoG from their official codebases for tasks not covered in original publications. For HumanoidBench with hands, the paper runs MR.Q, SimBa, SimBaV2, and FoG across 10 seeds using default hyperparameters. For DMC-Visual tasks, it runs MR.Q and TDMPC2 for comparison due to computational expense.

  • Generation budget / compute accounting. The primary compute budget is measured in environment steps: 1M steps for MuJoCo (no action repeat) and 500K agent steps for DMC and HumanoidBench (with action repeat 2, yielding 1M total environment frames). All methods are compared at the same step budget to evaluate sample efficiency β€” how much reward the agent accumulates after a fixed number of interactions. DR.Q uses a replay ratio (UTD) of 1 across all tasks β€” one gradient step per environment step for each of encoder, critic, and actor β€” which is lower than baselines like BRO (UTD=2), FoG (UTD=10), and REDQ (UTD=20). This makes DR.Q computationally efficient in wall-clock time while still competitive or superior in sample efficiency. The paper does not perform FLOPs-matched comparisons (unlike some pretraining-vs-inference scaling papers) β€” the comparison is purely on environment interaction budget.

  • Cross-validation / statistical protocol. All results are reported with 95% bootstrap confidence intervals computed over 10 random seeds per task. For aggregate statistics (mean, median, IQM) across tasks, confidence intervals are computed over the full n Γ— T sample using rliable with stratified bootstrap sampling (Agarwal et al., 2021). The paper does not use cross-validation for hyperparameter selection β€” it uses a single fixed set of hyperparameters across all 73 tasks without any per-task tuning (Table 5). Baseline results are taken from original publications (which typically use 3–10 seeds depending on the method) or rerun using the authors' official code with default hyperparameters and 10 seeds. The paper explicitly notes when baseline results are unavailable for certain tasks and describes which baselines were run locally (Appendix C.2, Table 4 compares hyperparameter and algorithmic configuration consistency across methods).

Main Quantitative Results

The paper organizes results by benchmark, each with a dedicated table and figure showing per-task learning curves. We present the headline comparisons, organized by domain.

Gym MuJoCo Results

DR.Q achieves an aggregate IQM normalized score of 1.691 across the 5 MuJoCo tasks, compared to 1.637 for SimBaV2, 1.499 for MR.Q, 1.570 for TD7, 1.242 for FoG, 1.135 for REDQ, and 1.540 for TDMPC2 (Table 6). However, the aggregate scores obscure substantial per-task variation. DR.Q outperforms SimBaV2 on 3 of 5 tasks: Ant-v4 (8138 vs. 7429), HalfCheetah-v4 (14775 vs. 12022), and Humanoid-v4 (11239 vs. 10546). But it underperforms on Hopper-v4 (2504 vs. 4054) and Walker2d-v4 (6422 vs. 6938). The poor Hopper-v4 performance is shared by other recent methods β€” MR.Q achieves 2692, FoG achieves 1822 β€” suggesting this is a challenging environment for model-based representation methods, not a DR.Q-specific failure. The paper attributes the Hopper-v4 weakness to "a side effect of adopting unified hyperparameters across all benchmarks" (Section 6, Limitations).

Figure 5 shows learning curves: DR.Q exhibits faster initial learning and higher asymptotic performance than MR.Q on Ant-v4, HalfCheetah-v4, and Humanoid-v4, with the Humanoid-v4 gap being particularly pronounced (DR.Q reaches ~11000 while MR.Q plateaus around ~10000). On Walker2d-v4, DR.Q's final performance is comparable to MR.Q but with higher variance. On Hopper-v4, all methods show high variance and DR.Q underperforms SimBaV2.

DMC-Easy Results

DR.Q achieves an aggregate IQM of 0.937 (normalized by dividing raw return by 1000) across the 21 DMC-Easy tasks, compared to 0.935 for FoG, 0.936 for MR.Q, 0.933 for SimBaV2, and 0.941 for TDMPC2 (Table 7). The median scores are 0.885 for DR.Q vs. 0.875 for SimBaV2 and 0.876 for MR.Q. These aggregates are close, reflecting the fact that many DMC-Easy tasks are near-saturation for all strong methods (e.g., cartpole-balance achieves 999–1000 for all methods; reacher-easy achieves 975–983).

DR.Q's most visible advantages appear on tasks where baselines have not saturated. On acrobot-swingup, DR.Q achieves 569 vs. 436 for SimBaV2 (a 30% improvement) and 414 for FoG. On quadruped-run, DR.Q achieves 953 vs. 935 for SimBaV2 and 918 for FoG. On quadruped-walk, DR.Q achieves 969 vs. 962 for SimBaV2 and 963 for MR.Q. On hopper-hop, DR.Q achieves 384, substantially above SimBaV2 (290) and MR.Q (251), though below TDMPC2 (425). Learning curves in Figure 6 show DR.Q matching or exceeding baselines on most tasks, with particularly clear advantages on acrobot-swingup (where all baselines plateau lower), hopper-hop (where DR.Q's final performance and learning speed exceed MR.Q and SimBaV2), and hopper-stand (where DR.Q converges faster than MR.Q).

DMC-Hard Results

The DMC-Hard tasks (7 dog and humanoid locomotion tasks) reveal DR.Q's strongest relative gains. DR.Q achieves an aggregate IQM of 0.917, substantially above SimBaV2 (0.808), MR.Q (0.796), FoG (0.880), TDMPC2 (0.464), and BRO (0.772). The mean normalized score is 0.842 for DR.Q vs. 0.729 for SimBaV2 β€” a 15.5% improvement (Table 8).

The largest gains appear on the three most challenging tasks. On dog-run, DR.Q achieves 721, far exceeding FoG (613), MR.Q (569), and SimBaV2 (562). The paper claims this is "the first that achieves an average return that exceeds 700 on the challenging dog-run task under 1M environment steps." On humanoid-run, DR.Q achieves 465 vs. FoG (292), MR.Q (200), and SimBaV2 (194) β€” more than doubling the performance of MR.Q and SimBaV2. On humanoid-walk, DR.Q achieves 925 vs. FoG (878), MR.Q (662), and SimBaV2 (651) β€” a 40% improvement over MR.Q. On the remaining DMC-Hard tasks (dog-stand, dog-trot, dog-walk, humanoid-stand), DR.Q matches or slightly exceeds the best baselines β€” e.g., dog-stand: 972 for DR.Q vs. 981 for SimBaV2; dog-trot: 925 for DR.Q vs. 901 for FoG; dog-walk: 950 for DR.Q vs. 935 for SimBaV2; humanoid-stand: 938 for DR.Q vs. 931 for FoG.

Figure 7 shows learning curves: DR.Q's advantage is not just in final performance but in sample efficiency β€” on dog-run, DR.Q reaches 600 (which exceeds the final performance of all baselines except FoG) by approximately 300K steps; on humanoid-run, DR.Q's curve rises sharply around 250K steps while baselines improve more gradually or plateau lower.

HumanoidBench (Without Dexterous Hands) Results

DR.Q achieves an aggregate IQM (success-normalized) of 0.864 across the 14 HumanoidBench tasks without hands, compared to 0.846 for FoG, 0.799 for SimBaV2, 0.519 for MR.Q, and 0.734 for TDMPC2 (Table 9). The median is 0.823 for DR.Q vs. 0.781 for SimBaV2.

The performance distribution reveals that DR.Q's gains come primarily from a subset of tasks where it substantially exceeds all baselines. On h1-reach-v0 (a sparse-reward reaching task), DR.Q achieves 8101, more than doubling MR.Q (4902) and SimBaV2 (3850). On h1-sit-hard-v0, DR.Q achieves 843 vs. FoG (770) and SimBaV2 (679). On h1-sit-simple-v0, DR.Q achieves 931 vs. SimBaV2 (875). On h1-hurdle-v0, DR.Q achieves 344 vs. SimBaV2 (202) β€” a 70% improvement. On h1-run-v0, DR.Q achieves 820 vs. FoG (749) and SimBaV2 (415). On h1-pole-v0, DR.Q achieves 887 vs. SimBaV2 (791). On these tasks, DR.Q exhibits both faster initial learning and higher asymptotic performance (Figure 8).

However, DR.Q does not uniformly dominate. On h1-balance-simple-v0, SimBaV2 achieves 723 while DR.Q manages only 205 β€” a substantial gap. On h1-slide-v0, FoG achieves 674 while DR.Q reaches 355. On h1-balance-hard-v0, DR.Q (92) is between SimBaV2 (143) and MR.Q (69). These discrepancies suggest that the unified hyperparameters have environment-specific effects, though DR.Q's aggregate score still leads.

HumanoidBench (With Dexterous Hands) Results

The 14 HumanoidBench tasks with dexterous hands introduce substantially higher state and action dimensionality (observation dimensions up to 308, action dimensions up to 61) β€” the regime where the paper hypothesizes that the mutual information objective matters most due to task-irrelevant state dimensions. DR.Q achieves an aggregate IQM of 0.452, compared to 0.298 for SimBaV2, 0.286 for MR.Q, 0.254 for FoG, and 0.150 for TDMPC2 (Table 10). The mean normalized score is 0.534 for DR.Q vs. 0.417 for SimBaV2 β€” a 28% relative improvement over the strongest baseline on what the paper considers the most challenging domain.

The per-task breakdown shows DR.Q's largest margins on tasks requiring dynamic whole-body control. On h1hand-walk-v0, DR.Q achieves 512, far exceeding all baselines: TDMPC2 (234), MR.Q (95), SimBaV2 (64). On h1hand-stand-v0, DR.Q achieves 491 vs. MR.Q (300) and TDMPC2 (193). On h1hand-run-v0, DR.Q achieves 129 β€” a task where SimBaV2 achieves only 30 and MR.Q achieves 35. On h1hand-sit-hard-v0, DR.Q achieves 891 vs. SimBaV2 (724) and FoG (179). On h1hand-sit-simple-v0, DR.Q achieves 942 vs. SimBaV2 (927) and MR.Q (653). On h1hand-slide-v0, DR.Q achieves 285 vs. FoG (201) and SimBaV2 (136). On h1hand-stair-v0, DR.Q achieves 288 vs. SimBaV2 (120) and MR.Q (127).

However, on some tasks, DR.Q's performance is matched or exceeded by baselines. On h1hand-bookshelf-simple-v0, SimBaV2 achieves 838 while DR.Q manages 709. On h1hand-bookshelf-hard-v0, FoG achieves 577 and SimBaV2 achieves 496 while DR.Q reaches 349. On h1hand-crawl-v0, TDMPC2 achieves 897 while DR.Q manages only 526. These tasks involve fine manipulation (bookshelf, door) where the hand dexterity is the primary challenge, and DR.Q's representations β€” which the mutual information objective may bias toward locomotor-relevant features β€” may be less suited.

Figure 9 shows the learning curves: DR.Q exhibits rapid initial progress on h1hand-walk, h1hand-stand, h1hand-sit-hard, and h1hand-stair, while baselines (particularly MR.Q and SimBaV2) plateau substantially lower. On h1hand-reach-v0, DR.Q's curve rises throughout training while MR.Q plateaus around 4000.

Visual DMC Results

For the 12 visual control DMC tasks (pixel inputs, 84Γ—84 RGB, 3-frame stack), DR.Q achieves an aggregate IQM of 0.494 (normalized by raw return/1000), substantially above MR.Q (0.322), TDMPC2 (0.154), DrQ-v2 (0.241), DreamerV3 (0.168), and PPO (0.016) (Table 11). The median is 0.500 for DR.Q vs. 0.398 for MR.Q β€” a 25.6% relative improvement.

The most striking gains appear on tasks where MR.Q already improved over prior visual RL methods. On visual-dog-stand, DR.Q achieves 700, more than tripling MR.Q's 216 and TDMPC2's 117. On visual-dog-run, DR.Q achieves 118 vs. MR.Q's 60. On visual-dog-walk, DR.Q achieves 201 vs. MR.Q's 77. On visual-quadruped-run, DR.Q achieves 655 vs. MR.Q's 498. On visual-acrobot-swingup, DR.Q achieves 324 vs. MR.Q's 287. On visual-walker-run, DR.Q achieves 746 vs. MR.Q's 615.

However, on some tasks, DR.Q's advantage is modest or absent. On visual-reacher-hard, DR.Q achieves 954 vs. MR.Q's 965 β€” essentially tied. On visual-humanoid-run, all methods (DR.Q, MR.Q, TDMPC2) achieve a score of 1 β€” this task is essentially unsolved by all methods within the 1M frame budget, and the paper notes that DrQ-v2 requires 15M environment steps to achieve meaningful performance here.

Figure 10 shows the learning curves: DR.Q's sample efficiency on visual tasks is particularly notable β€” on visual-dog-stand, DR.Q reaches 500 by ~300K steps while MR.Q reaches only ~150; on visual-quadruped-run, DR.Q's curve rises steeply and continues improving while MR.Q begins to plateau.

Ablation Studies and Robustness Checks

InfoNCE loss ablation (Ξ»m = 0): Removing the InfoNCE loss generally degrades performance, with the largest drops occurring on high-dimensional HumanoidBench tasks. Figure 4 (top row, main paper) shows that on HalfCheetah-v4 (low-dimensional MuJoCo), removing InfoNCE has minimal effect (DR.Q and DR.Q w/o InfoNCE both reach ~14750), but on dog-run (high-dimensional DMC), the gap is visible (~721 vs. ~650), and on h1-pole-v0 and h1hand-walk-v0 (HumanoidBench), the gaps are substantial: on h1hand-walk-v0, DR.Q reaches ~500 while DR.Q w/o InfoNCE reaches ~250. The extended ablation in Figure 11 (Appendix E.1) confirms this pattern across 8 tasks: on Ant-v4 and dog-trot, the gaps are small; on humanoid-run, humanoid-walk, h1-run-v0, h1-sit-hard-v0, h1hand-pole-v0, and h1hand-sit-hard-v0, DR.Q w/o InfoNCE consistently underperforms DR.Q, sometimes substantially (e.g., h1hand-sit-hard-v0: DR.Q reaches ~890, w/o InfoNCE reaches ~700). The paper also notes that even without the InfoNCE loss, DR.Q w/o InfoNCE "is at least as competitive as MR.Q" β€” validating that the other components (faded PER, larger encoder) are not degrading performance, and the InfoNCE loss provides additional gains on top.

Faded PER ablation (sampling strategy variants): The paper compares DR.Q against two variants: DR.Q (only forget) β€” which uses only the forget mechanism with uniform TD-error weighting β€” and DR.Q (only LAP) β€” which uses only LAP prioritization without temporal decay. Figure 4 (bottom row, main paper) shows that removing either component degrades performance. On HalfCheetah-v4, DR.Q and both variants perform similarly. On dog-run, DR.Q (only LAP) drops to ~650 vs. DR.Q's ~721, while DR.Q (only forget) drops to ~680. On h1-pole-v0, DR.Q (only forget) drops sharply to ~400 (vs. DR.Q's ~887). On h1hand-walk-v0, all variants underperform DR.Q substantially. The extended ablation in Figure 12 (Appendix E.1) across 8 tasks reveals task-dependent sensitivity: on humanoid-run and humanoid-walk, DR.Q (only LAP) suffers severe degradation (from ~465 to ~200 on humanoid-run), highlighting the importance of the forget mechanism for these exploration-heavy tasks; on h1hand-walk-v0 and h1hand-sit-hard-v0, DR.Q (only forget) underperforms substantially. The pattern mirrors the complementary scaling axes insight: LAP matters more on tasks with sustained exploration needs, forget mechanism matters more on tasks with obsolete early experiences.

Latent dynamics consistency loss ablation: The paper also investigates removing the latent dynamics consistency loss term entirely (keeping only reward loss and InfoNCE loss), giving the variant DR.Q (w/o dyn loss). Figure 13 (Appendix E.1) shows results across 16 tasks. On simpler tasks (acrobot-swingup, fish-swim, hopper-hop), the effect is minor. On more complex tasks (h1hand-stair-v0, h1hand-pole-v0, h1hand-walk-v0, h1hand-sit-hard-v0), removing the dynamics loss causes significant degradation β€” e.g., h1hand-walk-v0 drops from ~500 to ~350; h1hand-pole-v0 drops from ~420 to ~300. The paper concludes that the dynamics loss term is "recommended to include it in the objective." This ablation validates that both the MSE dynamics loss and the InfoNCE loss are necessary β€” they provide complementary signals that neither alone can fully substitute.

MR.Q with scaled hyperparameters: To disentangle the effect of DR.Q's larger encoder architecture (hidden dim 750 vs. MR.Q's default) from the InfoNCE and faded PER contributions, the paper runs MR.Q with DR.Q's hyperparameters (larger hidden dimension, larger learning rate, weight decay). Figure 14 (Appendix E.2) shows results across 16 tasks. MR.Q (modified hyperparameters) generally improves over vanilla MR.Q β€” confirming that scaling the encoder helps β€” but still consistently underperforms DR.Q. On dog-run, DR.Q reaches ~721 while MR.Q (modified) reaches ~650; on humanoid-run, DR.Q reaches ~465 while MR.Q (modified) reaches ~350; on h1hand-walk-v0, DR.Q reaches ~512 while MR.Q (modified) reaches ~350. This demonstrates that DR.Q's gains are not solely attributable to network scaling.

Representation learning with noise injection: To test the hypothesis that the InfoNCE loss helps the encoder focus on task-relevant dimensions, the paper augments the state space with 50-dimensional Gaussian noise (mean 0, std 0.2) and compares DR.Q against MR.Q. Figure 15 (Appendix E.3) shows results on 8 tasks. The key finding: MR.Q's performance degrades under the extended state input (dashed vs. solid lines), while DR.Q's performance is "less affected." On hopper-stand, MR.Q drops from ~950 to ~900 with noise; DR.Q remains at ~950. On humanoid-stand, MR.Q drops from ~930 to ~900; DR.Q remains at ~940. On h1hand-walk-v0, the gap between DR.Q and MR.Q persists (~500 vs. ~350) even with noise. On h1hand-stand-v0, MR.Q drops from ~300 to ~250 with noise while DR.Q remains at ~490. The paper acknowledges that the 50-dimensional noise injection is modest relative to the 100-dim redundancy from dexterous hands, but the differential sensitivity between DR.Q and MR.Q supports the claim that the InfoNCE loss helps the encoder ignore task-irrelevant dimensions.

t-SNE visualization of learned representations: To qualitatively assess representation quality, the paper applies t-SNE (Maaten & Hinton, 2008) to the state-action representations z_sa of DR.Q and MR.Q after 200K training steps on 4 tasks: HalfCheetah-v4, humanoid-walk, h1-sit-hard-v0, and h1hand-stand-v0, using 5000 samples from the replay buffer (Figure 16, Appendix E.3). The visualizations show that MR.Q produces "separate, discontinuous clusters" β€” on HalfCheetah-v4 and humanoid-walk, the t-SNE plots show multiple distinct, well-separated clusters with void spaces between them. DR.Q, in contrast, produces "continuous and concentrated clusters" β€” the t-SNE plots show smoother, more connected manifolds with fewer disconnected islands. On h1hand-stand-v0, MR.Q's plot contains two large void areas (regions of the representation space with no samples), while DR.Q's plot is more uniformly covered. The paper interprets this as evidence that DR.Q learns "more structured, more informative and general representations."

Sampling strategy visualization: To illustrate how faded PER differs from standard LAP, the paper visualizes the sampling-relevant metric (TD error Γ— forget weight for DR.Q; TD error alone for MR.Q) for the first 100K transitions in the replay buffer after 200K training steps on 8 tasks (Figure 17, Appendix E.4). The visualizations show that under MR.Q (LAP only), old transitions with large TD errors persist with high sampling weight β€” in h1-run-v0, for example, transitions with time index 60K–100K (relatively old) still show TD errors of 2–6. Under DR.Q (faded PER), the product of TD error and forget weight is concentrated near time index 0 (newest transitions) β€” "no old experience attains a higher sampling probability than newer ones." This empirically validates Theorem 4.3(i) and confirms that faded PER successfully shifts the sampling distribution toward recent high-error transitions.

Critical Assessment

Does DR.Q genuinely improve over MR.Q and SimBaV2, or are the gains attributable to other factors?

The headline claim β€” that DR.Q "matches or surpasses recent strong baselines" β€” is well-supported by the aggregate results. DR.Q achieves the highest IQM on 4 of the 5 benchmark groupings (MuJoCo: highest IQM at 1.691 but within overlapping confidence intervals of SimBaV2 at 1.637; DMC-Easy: essentially tied; DMC-Hard: clear lead at 0.917; HumanoidBench w/o hand: highest at 0.864; HumanoidBench w/ hand: highest at 0.452; DMC-Visual: highest at 0.494). The DMC-Hard and HumanoidBench (w/ hand) improvements are the most robust β€” the confidence intervals show clear separation from the best baselines.

However, there is an important nuance: the gains are unevenly distributed across tasks within each benchmark. On Gym MuJoCo, DR.Q's aggregate lead is partly due to a single large outlier (Humanoid-v4: 11239 vs. SimBaV2's 10546) while it underperforms on Hopper-v4 and Walker2d-v4. On HumanoidBench (w/o hand), DR.Q's lead is driven by extraordinary performance on h1-reach-v0 (8101 vs. 3850 for SimBaV2) and h1-sit-hard-v0 (843 vs. 679), but it underperforms on h1-balance-simple-v0 (205 vs. 723). On HumanoidBench (w/ hand), the largest margins are on h1hand-walk-v0 (512 vs. 64 for SimBaV2) while underperforming on h1hand-bookshelf tasks. This task-level heterogeneity suggests that DR.Q's components provide benefits that are environment-specific rather than universally positive β€” the InfoNCE loss helps on tasks with redundant state dimensions (locomotion with dexterous hands) but may not help (or may even hurt) on tasks where most state dimensions are control-relevant (simpler MuJoCo tasks, fine manipulation tasks). The single-hyperparameter design means these task-specific effects are averaged over, and the aggregate wins are genuine, but a practitioner should not expect DR.Q to dominate uniformly across all tasks.

Do the ablations convincingly isolate the contributions of the InfoNCE loss and faded PER?

The ablation design has several strengths: (1) the InfoNCE ablation (Ξ»m = 0) isolates exactly one loss term, (2) the sampling strategy ablation compares the full faded PER against removing either PER or forget, and (3) the scaled MR.Q comparison controls for network capacity. These collectively support the claim that both the InfoNCE loss and faded PER contribute independently to performance.

However, there is a missing ablation: the paper does not report a variant that uses only InfoNCE + standard LAP (no forget mechanism), or only InfoNCE + pure forget (no LAP). The ablation in Figure 4 only considers DR.Q with neither InfoNCE nor faded PER (equivalent to scaled MR.Q), and DR.Q with InfoNCE but varying sampling strategies. A full factorial ablation (InfoNCE Γ— {LAP, forget, faded PER}) would strengthen the claim that the two components are complementary by showing the interaction effect. Without this, the complementarity argument β€” that InfoNCE and faded PER address different biases that manifest in different environments β€” rests primarily on observing that removing InfoNCE hurts most on high-dimensional tasks while removing a sampling component hurts most on exploration-heavy tasks. This is suggestive but correlational rather than causally demonstrated.

Additionally, the scaled MR.Q comparison (Figure 14) shows that network scaling (larger hidden dimension, weight decay) provides some of DR.Q's gains. MR.Q with DR.Q's hyperparameters generally outperforms vanilla MR.Q. The paper correctly notes that DR.Q still leads, but the magnitude of the scaling contribution vs. the InfoNCE contribution is not cleanly separated β€” one would need to compare MR.Q (scaled) + InfoNCE against MR.Q (scaled) without InfoNCE, holding network size constant.

Is the noise injection experiment sufficient to validate the information-theoretic claims?

The noise injection experiment (Figure 15) is designed to test the claim that the InfoNCE loss helps the encoder ignore task-irrelevant state dimensions by showing that DR.Q is more robust to noise injection than MR.Q. The experiment is clever and the results are directionally correct (DR.Q degrades less than MR.Q), but several aspects limit its strength:

  • The noise is injected into the state representation, not the raw state. The 50-dim noise vector is concatenated to the state before encoding. This means the encoder can learn to simply ignore those dimensions (by assigning them near-zero weights in the first layer). DR.Q's InfoNCE loss may or may not help with this β€” the experiment shows it does, but the mechanism is unclear. An alternative possibility is that DR.Q's larger encoder (hidden dim 750 vs. MR.Q's default) is simply better at learning to ignore irrelevant inputs, independent of the InfoNCE loss.
  • The noise magnitude is modest (std 0.2) compared to the typical scale of state features. A stronger test would sweep noise magnitudes and show that DR.Q's advantage grows as noise increases.
  • The experiment covers only 8 tasks β€” the set of tasks is not random but selected, and the paper does not specify the selection criterion.
  • No comparison to SimBaV2 or FoG on the noise-augmented tasks. If these baselines are also robust to noise (due to different mechanisms like normalization), then DR.Q's robustness is not unique.

Are there genuine weaknesses in the experimental design that limit the strength of the claims?

1. Single model architecture family. All experiments use the MR.Q architecture as the base, with modifications only to the encoder loss and sampling strategy. It is unknown whether the InfoNCE loss and faded PER would provide benefits when integrated into a different base algorithm (e.g., TDMPC2, SimBaV2, or a pure SAC-based method). The paper positions DR.Q as a general-purpose improvement to model-based representation learning, but the evidence only covers one architectural lineage.

2. No experiments on discrete-action domains. The paper explicitly excludes discrete-action tasks (e.g., Atari) from evaluation, citing cost. This is a significant gap because the primacy bias and representation informativeness problems should, in principle, be domain-agnostic. The claim that DR.Q provides "debiased model-based representations" would be substantially strengthened by showing benefits in at least one discrete-action domain.

3. Impact of individual hyperparameters is not systematically studied. The loss weights (Ξ»d=1.0, Ξ»r=0.1, Ξ»m=0.1), the forget rate (Ξ΅=0.0001), the forget threshold (Ξ΅_low=0.1), the temperature (Ο„=0.1), and the InfoNCE loss weight (Ξ»m=0.1) are fixed throughout all experiments. The paper provides no sensitivity analysis β€” no sweeps over Ξ»m, Ξ΅, or Ξ΅_low. This means we cannot assess whether the reported performance is near-optimal or whether substantial further gains are possible with better-tuned hyperparameters. The claim of "single set of hyperparameters" generality is a strength, but the optimality of that set is unverified.

4. Limited statistical comparisons against baselines. The paper reports confidence intervals for DR.Q and provides baseline numbers, but does not perform formal statistical tests (e.g., Mann-Whitney U, t-test with correction) comparing DR.Q against each baseline per task. On MuJoCo and DMC-Easy tasks where the aggregates are close, it is unclear whether DR.Q's leads are statistically significant. The aggregate IQM with overlapping confidence intervals on MuJoCo (DR.Q 1.691 [1.473, 1.879] vs. SimBaV2 1.637 [1.470, 1.791]) suggests the difference may not be reliable.

5. The exploration budget is confounded with representation learning quality. DR.Q uses a replay ratio of 1 while several baselines (FoG UTD=10, BRO UTD=2) use higher replay ratios. This means DR.Q achieves its sample efficiency with less computation per environment step. However, on tasks where DR.Q underperforms (Hopper-v4, h1-balance-simple-v0), it is unclear whether higher UTD would close the gap. The paper does not report a DR.Q variant with UTD > 1, which would help distinguish whether the representation quality or the gradient budget is the limiting factor.

6. The 1M step budget may not reveal asymptotic performance differences. On several tasks (DMC-Easy, some HumanoidBench tasks), learning curves have not converged at 1M steps. Methods that learn faster initially but plateau lower might appear superior at 1M steps but be inferior at 2M or 5M steps. DR.Q's curves generally show continued improvement at 1M steps (especially on DMC-Hard and HumanoidBench), but without longer training runs, we cannot assess whether its lead persists or converges.

7. No results on real-world (non-simulated) tasks. All evaluations are in simulation (MuJoCo, DMC, HumanoidBench). The paper's implicit claim is that DR.Q is suitable for practical continuous control, but the sim-to-real gap is unaddressed. Whether the InfoNCE objective and faded PER would behave similarly with noisy, partial real-world observations is unknown.

What experiments would strengthen the paper?

  • Sweep over Ξ»m and Ξ΅ to show that the chosen values are near-optimal and that performance is not hypersensitive to these hyperparameters. This would strengthen the "single hyperparameter set" claim by showing that the set is robust.
  • Run DR.Q on Atari (a subset of games) to test generalizability to discrete-action domains.
  • Run a full factorial ablation: (InfoNCE Γ— faded PER) β†’ {no InfoNCE + LAP, InfoNCE + LAP, no InfoNCE + faded PER, InfoNCE + faded PER}. This would isolate the interaction effect and directly test the complementarity hypothesis.
  • Run DR.Q on MR.Q's exact architecture (with only the InfoNCE loss and faded PER added, no hidden-dimension changes) to completely disentangle architectural scaling from algorithmic contributions.
  • Extend training to 2M–5M steps on a subset of tasks to assess asymptotic performance and whether leads persist.
  • Run DR.Q with UTD = 2 or 5 to test whether the replay ratio 1 setting is a bottleneck on tasks where DR.Q underperforms.
  • Include SimBaV2 and FoG in the noise injection experiment to show whether DR.Q's robustness to task-irrelevant dimensions is unique or shared by other strong methods with normalization techniques.
  • Evaluate on a distinct domain (e.g., Meta-World, Adroit hand manipulation, or a real robot benchmark) to test whether the benefits generalize beyond the three paper-standard benchmarks.

6. Limitations and Trade-offs

Assumption: Model-Based Representations Can Be Debiased Without Fundamental Architectural Changes

The constraint. DR.Q addresses two specific biases β€” the informativeness gap (via InfoNCE loss) and the primacy bias (via faded PER) β€” while preserving the core architectural decomposition inherited from MR.Q: separate state encoder, state-action encoder, linear MDP predictor, and downstream actor-critic networks. This decomposition assumes that the only problems with model-based representations are these two biases, and that adding corrective loss terms and adjusting the sampling distribution is sufficient to fix them.

The consequence. If there are structural limitations in the encoder architecture itself β€” the particular way state and action information is combined, the linearity constraint on the dynamics predictor, the fixed dimensionality of the representation space (512 dimensions for both $z_s$ and $z_{sa}$) β€” then DR.Q's debiasing approach can only partially compensate. The architecture imposes an upper bound on what representations can express, regardless of the training objective. On tasks where DR.Q underperforms strong baselines (Hopper-v4: DR.Q achieves 2504 vs. SimBaV2's 4054; h1-balance-simple-v0: DR.Q achieves 205 vs. SimBaV2's 723; h1-slide-v0: DR.Q achieves 355 vs. FoG's 674), the architecture may be fundamentally mismatched to the task dynamics, and neither the InfoNCE loss nor faded PER can overcome this.

What evidence exists. The paper provides indirect evidence through the scaled MR.Q comparison (Figure 14, Appendix E.2): MR.Q with DR.Q's larger encoder architecture (hidden dim 750) improves over vanilla MR.Q but still underperforms DR.Q in most environments β€” suggesting the encoder capacity matters but is not the sole determinant. However, the paper does not systematically vary the encoder architecture beyond this single scaling experiment. There is no ablation over representation dimension, no comparison with alternative encoder architectures (e.g., attention-based, graph-structured, or architectures that skip the linear dynamics constraint), and no experiment that tests whether DR.Q's representations saturate in expressiveness on tasks where it underperforms. The t-SNE visualizations (Figure 16) show DR.Q producing more structured representations than MR.Q, but they do not establish that these representations are optimal or near the expressiveness ceiling of the architecture β€” they only show improvement over a weaker baseline.

Mitigation status. The paper does not address this limitation. The architecture is treated as fixed and inherited from prior work, and the focus is entirely on the training objective and sampling strategy. Section 6 (Conclusion) suggests future work to "find better paradigms for learning model-based representations," which implicitly acknowledges that the current paradigm may not be optimal, but no concrete architectural alternatives are explored. A practitioner who finds DR.Q underperforming on their domain has no guidance from the paper about whether to increase representation dimension, modify the encoder structure, or relax the linear dynamics constraint.

Difficulty Estimation Is Not Attempted: No Mechanism for Detecting When Representations Are Inadequate

The constraint. DR.Q provides no mechanism for estimating whether its learned representations are sufficient for a given task, nor for detecting when they have failed. The system trains the encoder with a fixed objective and feeds the resulting representations to the actor-critic regardless of their quality. If the mutual information objective converges to a suboptimal local minimum (e.g., because the InfoNCE lower bound is loose for the chosen batch size and temperature), or if the dynamics consistency loss achieves low MSE by capturing irrelevant state dimensions (despite the mutual information objective), there is no signal that triggers corrective action β€” no adaptive adjustment of loss weights, no diagnostic metric that warns of insufficient representational quality, and no fallback to alternative representations.

The consequence. In domains where DR.Q's representations are inadequate, the downstream actor-critic inherits the deficiency with no recourse. The agent may plateau at a suboptimal policy and never recover, because the policy gradient is computed entirely in the representation space β€” if the representation does not distinguish between states that require different actions, the policy cannot learn to discriminate between them. This is not a hypothetical concern: on tasks like h1-balance-simple-v0, DR.Q achieves only 205 (vs. SimBaV2's 723), and on h1hand-bookshelf-simple-v0, DR.Q reaches 709 (vs. SimBaV2's 838). These failures are opaque β€” there is no diagnostic in the paper's framework that would alert a practitioner that the representations are the bottleneck rather than the policy optimization.

What evidence exists. The paper provides extensive ablation studies showing when representations are adequate (via performance comparisons), but no analysis of why they fail on specific tasks. The t-SNE visualizations (Figure 16) show qualitative differences between DR.Q and MR.Q representations, but they are generated after 200K steps β€” well into training β€” and do not serve as an online diagnostic. The noise injection experiment (Figure 15) shows DR.Q is more robust to irrelevant dimensions than MR.Q, but does not establish a threshold beyond which DR.Q's representations degrade. The paper also does not report training curves for the encoder losses (InfoNCE loss, dynamics MSE loss, reward loss) that might reveal whether poor downstream performance correlates with poor representation learning convergence.

Mitigation status. Not addressed at all. The paper treats representation quality as an emergent property of the training procedure and evaluates it only indirectly through downstream task performance. There is no suggestion of a future monitoring or diagnostic mechanism. For a practitioner, this means deploying DR.Q requires blind trust that the representations will be adequate, with no tool for verifying this trust or debugging failures.

The Faded PER Priority Computation Relies on the Quality of the Current TD Error Estimates

The constraint. Faded PER uses the current TD error $|\delta(i)|$ stored in the replay buffer to compute sampling probabilities (Equation 10). However, the TD error of a transition is computed at the time the transition was last sampled and used for a critic update, not its true informativeness under the current policy and value function. As the value function evolves, the TD error of any given transition changes, but the buffer only updates priorities when that specific transition is re-sampled. This creates a staleness problem: transitions that have not been sampled recently retain outdated TD error estimates, potentially causing the sampling distribution to lag behind the agent's current learning needs.

The consequence. The faded PER mechanism can systematically under-sample transitions that were initially assigned low priority (because their TD error was small when first added to the buffer) but have become highly informative as the policy and value function evolved. This is the inverse of the primacy bias: instead of over-sampling old high-error transitions, faded PER risks under-sampling old transitions whose importance has increased. The temporal decay $(1-\epsilon)^i$ compounds this problem β€” it reduces the sampling probability of all old transitions regardless of whether their TD error estimate is accurate or stale. A transition from 500K steps ago with a stale low TD error has both a low error-based priority (because the stored TD error is small) and a low temporal weight (because it is old), making it essentially invisible to the sampling process even if it contains a rare and valuable state transition that the current policy rarely visits.

What evidence exists. The paper provides no empirical analysis of the staleness problem. The sampling visualization (Figure 17) shows the TD error Γ— forget weight product for recent transitions but does not show how often TD error estimates are updated, what the distribution of "true" TD errors (under the current value function) would be for old transitions, or how frequently a transition with a stale low-priority estimate gets re-sampled and re-prioritized. Theorem 4.3 establishes bounds on expected sample counts, but these bounds depend on the current TD error, which is assumed known β€” the theorem does not address the staleness issue.

Mitigation status. The LAP variant used by DR.Q (Fujimoto et al., 2020) updates priorities every time a transition is sampled, which partially addresses staleness β€” transitions that are sampled get their priorities refreshed. But transitions that are not sampled (because their stored priority is low) never get updated, creating a self-reinforcing cycle. The paper does not discuss this issue, does not propose a mechanism for periodic priority refreshing (e.g., sweeping through the buffer and recomputing TD errors), and does not compare faded PER against alternative prioritization schemes that address staleness more aggressively. The flat minimum priority of 1 (from LAP) provides some protection against complete exclusion, but with a large buffer (1M transitions) and batch size 256, a transition with minimum priority has expected sample count $256 \times 1 / 10^6 \approx 0.000256$ per gradient step β€” it will be sampled approximately once every ~3900 gradient steps, which is infrequent enough that its priority estimate could be severely outdated.

Performance Is Uneven Across Tasks, and the Paper Provides No Guidance on When DR.Q Should Be Preferred

The constraint. DR.Q is evaluated as a "general-purpose" algorithm with a single set of hyperparameters, and the aggregate results show competitive or superior performance. However, the per-task breakdown reveals substantial variance: DR.Q dramatically outperforms baselines on some tasks (h1-reach-v0: DR.Q 8101 vs. 3850 for SimBaV2; h1hand-walk-v0: DR.Q 512 vs. 64 for SimBaV2; humanoid-run: DR.Q 465 vs. 194 for SimBaV2) while underperforming on others (Hopper-v4: DR.Q 2504 vs. 4054 for SimBaV2; h1-balance-simple-v0: DR.Q 205 vs. 723 for SimBaV2; h1-slide-v0: DR.Q 355 vs. 674 for FoG).

The consequence. A practitioner choosing whether to adopt DR.Q for a new continuous control task has no principled way to predict whether it will excel or underperform. The paper identifies some patterns β€” the mutual information objective helps most in high-dimensional environments with task-irrelevant state dimensions (HumanoidBench with dexterous hands), while simpler tasks may see marginal benefits β€” but these are post-hoc interpretations of results, not predictive criteria. The paper provides no characterization of which environment properties (state dimensionality, action dimensionality, reward sparsity, dynamics complexity, contact frequency, degree of partial observability) correlate with DR.Q's relative performance. Without such characterization, DR.Q's claim of being "general-purpose" is an empirical observation about a specific set of benchmarks, not a guarantee of performance on unseen domains.

What evidence exists. The task-level variance is documented explicitly in the per-task tables (Tables 6–11) and learning curves (Figures 3, 5–10). The paper acknowledges the Hopper-v4 weakness in the limitations section: "its performance on tasks like Hopper-v4 is inferior, which can be a side effect of adopting unified hyperparameters across all benchmarks" (Section 6). But this acknowledgment treats the failure as an isolated case rather than part of a broader pattern of task-dependent effectiveness. The paper does not aggregate performance by task characteristics (dimensionality, reward density, contact frequency) to reveal systematic trends.

Mitigation status. The paper does not attempt to characterize when DR.Q works or fails. The limitations section (Section 6) mentions the Hopper-v4 failure and the failure on visual-humanoid-run, but does not propose a diagnostic framework, a set of task properties that predict success, or even a recommendation for practitioners to run a small-scale evaluation before committing to DR.Q. The "single set of hyperparameters" claim, while demonstrating robustness, also means that if the default configuration is suboptimal for a given task, the paper provides no guidance on which hyperparameters to adjust.

Visual Control Performance Is Not Fully Demonstrated, and High-Dimensional Visual Tasks Remain Unsolved

The constraint. The paper evaluates DR.Q on 12 visual DMC tasks but explicitly acknowledges (Section 6) that it only runs for 1M environment frames (500K agent steps with action repeat 2). The paper states:

"DrQv2 requires 15M environment steps to achieve meaningful performance on visual-humanoid-run, while we only run DR.Q and baselines for 1M steps. The encoders may not capture good representations for downstream policy and critic learning within such a limited budget."

On visual-humanoid-run, all methods (DR.Q, MR.Q, TDMPC2) achieve a score of 1 β€” essentially zero progress. On other visual tasks, DR.Q shows strong improvements over MR.Q (e.g., visual-dog-stand: 700 vs. 216), but the absolute performance is still far below the proprioceptive versions of the same tasks (proprioceptive dog-stand: DR.Q achieves 972). This means DR.Q's visual representations, while better than MR.Q's within the 1M-frame budget, may still be substantially worse than what is achievable with state-based inputs, and it is unknown whether they would close the gap with more training.

The consequence. The paper's claim that DR.Q is effective for visual control is only partially supported. For a practitioner considering DR.Q for a visual control task (e.g., a real robot using camera inputs), the evidence is limited to 12 simulated tasks at a relatively short training horizon, with the hardest visual task completely unsolved. It is unknown whether DR.Q's representations would continue to improve over MR.Q's with extended training (2M–15M steps), or whether the InfoNCE loss provides diminishing returns as visual representations become more refined. The failure on visual-humanoid-run also raises concerns about DR.Q's ability to handle visual tasks with complex dynamics and high-dimensional action spaces β€” the combination of partial observability, high-dimensional visual inputs, and complex locomotive dynamics may exceed the capacity of the MR.Q-style encoder architecture regardless of the training objective.

What evidence exists. The visual DMC results (Table 11, Figure 10) show DR.Q outperforming MR.Q on 11 of 12 tasks within the 1M-frame budget, with particularly large margins on dog tasks (dog-stand: 700 vs. 216; dog-walk: 201 vs. 77). However, the learning curves in Figure 10 show that DR.Q's performance on visual tasks like dog-run (~118) and dog-trot (~113) remains well below the proprioceptive counterparts (~721 and ~925, respectively), and the curves on several visual tasks are still rising at 1M frames (visual-dog-stand, visual-quadruped-run), suggesting significant headroom remains. The paper does not report longer training runs for any visual task, and does not characterize the representation quality for visual inputs separately from state-based inputs.

Mitigation status. The paper acknowledges the budget limitation explicitly (Section 6), framing it as a constraint of the experimental setup rather than a fundamental limitation of the method. However, the acknowledgment does not include a commitment to longer training runs in future work, nor does it discuss whether the architectural choices (e.g., the state encoder architecture designed for proprioceptive inputs) are appropriate for visual inputs. The visual experiments use the same encoder architecture as the proprioceptive experiments (with the addition of convolutional layers for processing pixel inputs, as in MR.Q), but no ablation tests whether a visual-specific encoder design would improve performance. For a practitioner deploying DR.Q on visual tasks, the paper offers evidence that DR.Q is better than MR.Q within a limited budget, but no guarantee that this advantage persists at scale or that the absolute performance is sufficient for practical applications.

Scalability to Larger Networks and Longer Training Horizons Is Unexplored

The constraint. DR.Q uses a fixed encoder capacity (hidden dimension 750, representation dimension 512) and a fixed training horizon (1M environment steps for MuJoCo, 500K agent steps for DMC and HumanoidBench). The paper does not investigate how DR.Q's performance scales with increased network capacity (larger hidden dimensions, deeper networks, larger representation dimensions) or extended training (beyond 1M steps). This matters because several recent works (BRO, SimBaV2, FoG) have demonstrated that scaling network capacity can provide substantial performance gains in continuous control, and it is unknown whether DR.Q's debiasing mechanisms interact positively or negatively with scaling.

The consequence. If DR.Q's representations saturate in quality at a given network capacity β€” as the comparison between scaled MR.Q and DR.Q (Figure 14) hints (DR.Q outperforms scaled MR.Q, but not by enormous margins on all tasks) β€” then further scaling may provide diminishing returns, and DR.Q may not benefit from the "bigger is better" trend that has driven progress in other areas of deep RL. Conversely, if the InfoNCE loss and faded PER enable more efficient use of additional capacity (by focusing representational power on task-relevant features and avoiding overfitting to early experiences), then DR.Q might scale better than baselines. Neither hypothesis is tested. For a practitioner with access to large-scale compute resources, it is unclear whether to invest in scaling DR.Q's networks or whether alternative methods (SimBaV2 with its normalization scheme, BRO with its regularization) would benefit more from additional capacity.

What evidence exists. The paper provides one indirect scaling experiment: comparing MR.Q with DR.Q's larger encoder hyperparameters against vanilla MR.Q (Figure 14). This shows that scaling helps MR.Q, and that DR.Q (which already uses the scaled architecture) outperforms scaled MR.Q. But this is a single scaling step β€” from MR.Q's default capacity to DR.Q's fixed capacity β€” not a scaling curve. The paper does not sweep hidden dimensions (e.g., 256, 512, 750, 1024, 2048), representation dimensions, or encoder depth, and does not report how the gap between DR.Q and baselines changes with capacity. The training horizon is also fixed β€” no experiments extend beyond 1M steps, so asymptotic behavior and potential overfitting at longer horizons are uncharacterized.

Mitigation status. Not addressed. The paper frames DR.Q as achieving competitive performance with a replay ratio of 1 and "without altering network configurations" (Section 2), which implicitly positions scalability as out of scope. But the question of how DR.Q's gains interact with network scale is a natural one for practitioners deciding whether to adopt it as a base algorithm for large-scale RL, and the paper provides no evidence to guide that decision. The limitations section (Section 6) does not mention this scalability gap.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the conversation around model-based representation learning in RL by reframing what was previously treated as a geometric problem into an information-theoretic one. Before DR.Q, the dominant paradigm β€” spanning DeepMDP (Gelada et al., 2019), TD7 (Fujimoto et al., 2023), MR.Q (Fujimoto et al., 2025), and the representation-learning components of model-based systems like TDMPC2 (Hansen et al., 2024) β€” assumed that minimizing the Euclidean distance between predicted and actual next-state representations was both necessary and sufficient for learning useful latent dynamics. DR.Q's Theorem 4.1 proves this assumption is mathematically unfounded: MSE and mutual information have no consistent monotonic relationship, meaning an encoder can achieve low prediction error by focusing on task-irrelevant state dimensions while neglecting control-critical features. This is not an incremental observation β€” it identifies a structural blind spot in the dominant objective function used across an entire subfield.

The magnitude of this reframing is significant but bounded. It is not a paradigm shift in the Kuhnian sense β€” DR.Q does not propose abandoning the model-based representation framework or the encoder-actor-critic decomposition. Rather, it provides a corrective diagnosis that explains why prior methods sometimes underperform in high-dimensional environments (where irrelevant state dimensions provide "easy" MSE reduction) and a constructive prescription (mutual information maximization via InfoNCE loss) that fills the gap. The practical consequence is that future work on dynamics-based representation learning can no longer assume MSE is sufficient β€” the burden of proof shifts to authors to either include an information-theoretic objective or to demonstrate that their specific domain lacks the irrelevant-dimension problem.

The paper also resolves a latent tension in the literature between PER-based and forget-mechanism-based replay strategies. Prior work treated these as competing solutions to the replay sampling problem: PER (Schaul et al., 2015; Fujimoto et al., 2020) argues that TD-error magnitude determines informativeness; forget mechanisms (Novati & Koumoutsakos, 2019; Kang et al., 2025) argue that recency determines relevance to the current policy. Practitioners were left with an either-or choice. DR.Q shows that these are not competing hypotheses but complementary signals addressing orthogonal biases β€” PER corrects for uniform sampling's blindness to informativeness, and the forget mechanism corrects for PER's blindness to distributional staleness. The multiplicative combination in faded PER (Equation 10) provides a principled synthesis, validated by the ablation in Figure 4 (bottom row), where removing either component degrades performance β€” but on different tasks: removing LAP hurts most on humanoid-run (Figure 12, dropping from ~465 to ~200), while removing the forget mechanism hurts most on h1-pole-v0 (Figure 4, dropping from ~887 to ~400). This task-dependent sensitivity to different components of the replay strategy resolves the apparent contradiction between PER-only and forget-only approaches: neither is universally dominant because they address different problems that manifest to different degrees in different environments.

The paper also shifts research priorities within model-based RL toward verifier/encoder quality rather than planning sophistication. The finding that a simple InfoNCE loss β€” a contrastive objective with no planning, no explicit model rollouts beyond the 5-step encoder horizon, and no tree search β€” can yield 15.5% improvement over SimBaV2 on DMC-Hard tasks and 28% over SimBaV2 on HumanoidBench with dexterous hands (Figure 1) suggests that much of the remaining performance gap in continuous control RL is attributable to representation quality rather than policy optimization sophistication. This redirects attention away from ever-more-complex planning algorithms (MCTS, trajectory optimization) and toward the fundamental question of what makes a state representation "good" for control β€” a question that DR.Q begins to answer in information-theoretic terms but leaves substantially open.

Importantly, DR.Q also raises the bar for what constitutes a "general-purpose" RL algorithm. Table 4 (Appendix C.2) reveals that several baselines claiming single-hyperparameter generality β€” including SimBa, SimBaV2, and FoG β€” actually switch algorithmic configurations (single Q vs. clipped double Q, batch size, reset schedules) across domains. DR.Q genuinely uses the same hyperparameters and the same algorithmic choices across all 73 tasks. This is not just a statistical nicety; it is an empirical validation that the debiasing mechanisms address structural problems in representation learning (the informativeness gap) and sampling (the primacy bias) that are domain-agnostic, rather than compensating for domain-specific idiosyncrasies. Future work claiming generality must now contend with DR.Q as a baseline that achieves this standard with a simpler recipe β€” no parameter resets, no distributional critics beyond two-hot rewards, no layer normalization, no UTD scaling.

Follow-Up Research This Work Enables

Characterizing the failure modes of the InfoNCE + MSE combination across environment types. DR.Q demonstrates that the InfoNCE loss provides the largest gains on high-dimensional tasks with redundant state information (HumanoidBench with dexterous hands), but it underperforms SimBaV2 on several lower-dimensional tasks (Hopper-v4, h1-balance-simple-v0, h1-slide-v0). A systematic study that sweeps state dimensionality, the proportion of task-irrelevant dimensions (by controlled injection of noise dimensions with known relevance), and the dynamics complexity (measured by Lyapunov exponents or intrinsic dimension) across a range of custom-built environments would reveal when the InfoNCE loss helps, when it is neutral, and crucially, whether it ever hurts relative to MSE-only training. The hypothesis β€” that InfoNCE helps when the ratio of irrelevant-to-relevant state dimensions is high and hurts (by adding noise to the gradient signal) when most dimensions are control-relevant β€” is testable: build a parametric family of environments where the fraction of irrelevant dimensions can be smoothly varied, train DR.Q and an MSE-only baseline across this sweep, and plot the performance gap as a function of irrelevance fraction. The DR.Q paper provides the experimental template (same hyperparameters, multiple seeds, aggregate metrics) but cannot answer this because benchmark tasks do not come with labeled relevance annotations.

Combining DR.Q's representations with high-UTD training to break through the sample-efficiency ceiling. DR.Q achieves its performance with a replay ratio of 1, which is substantially lower than baselines like FoG (UTD=10), REDQ (UTD=20), and BRO (UTD=2). The paper's motivation for UTD=1 is that better representations should reduce the need for multiple gradient steps per transition β€” quality over quantity. However, this leaves open the question: could DR.Q's representations compound with high UTD? A direct experiment would run DR.Q with UTD ∈ {1, 2, 5, 10} on the tasks where DR.Q underperforms (Hopper-v4, h1-balance-simple-v0, h1-slide-v0) and on tasks where it already dominates (humanoid-run, h1hand-walk-v0, dog-run). The null hypothesis β€” that DR.Q's representations are already sufficient and high UTD provides no additional benefit β€” would validate the quality-over-quantity framing. The alternative β€” that high UTD amplifies DR.Q's advantages by providing more gradient steps to exploit the improved representations β€” would suggest that DR.Q + high UTD could set a new state-of-the-art. The DR.Q architecture makes this experiment straightforward since the encoder and actor-critic training are already separated.

Verifier-quality analysis: how does the mutual information objective affect value estimation accuracy? Lemma 4.2 establishes that maximizing I(Z_{sa}; Z_{s'}) reduces H(Z_{s'} | Z_{sa}), which tightens the value-error bounds from DeepMDP and MR.Q. However, the paper evaluates this only indirectly through downstream task performance. A direct measurement would instrument the training process to track: (a) the mutual information estimate (the InfoNCE loss value, which serves as a lower bound) during training, (b) the TD error of the critic (a proxy for value estimation accuracy), and (c) the policy improvement rate (the change in average Q-value over successive actor updates). Plotting these three quantities against each other over the course of training on a set of tasks with varying dimensionalities would test whether reductions in InfoNCE loss cause reductions in TD error (indicating the mutual information objective is mechanistically improving value estimation, not just correlating with better final performance) or whether the relationship is more complex. This analysis could be done using DR.Q's existing training logs without any new experiments β€” just instrumenting the code to log these quantities β€” and would provide the missing mechanistic link between the information-theoretic theory and the empirical gains.

Testing the generality of faded PER beyond model-based representation learning. Faded PER is proposed as a fix for sampling bias in the encoder training of DR.Q, but Theorem 4.3 establishes properties that are independent of the representation-learning context β€” it applies to any replay-based training where transitions have associated TD errors and the primacy bias is a concern. A natural experiment is to plug faded PER into a pure model-free algorithm (SAC, TD3, or DroQ) that uses standard PER, and measure whether the multiplicative combination of TD-error prioritization and temporal decay improves sample efficiency over PER alone, forget alone, and uniform sampling, on a set of standard continuous control tasks. The DR.Q paper already provides the theoretical justification (Theorem 4.3) and a practical recipe (Equation 10 with LAP modifications), making this a low-implementation-effort extension. A negative result β€” faded PER provides no benefit in pure model-free settings β€” would reveal that the primacy bias is specifically a representation-learning problem (because encoder representations overfit to early data even when the critic does not), which would refine our understanding of where the bias originates. A positive result would establish faded PER as a general-purpose replacement for standard PER across off-policy RL.

Scaling DR.Q's encoder capacity to test whether the debiasing mechanisms interact positively with network size. The paper provides only one indirect scaling comparison (MR.Q with DR.Q's larger encoder vs. vanilla MR.Q, Figure 14), which shows that scaling helps MR.Q but DR.Q still leads. A proper scaling study would sweep encoder hidden dimensions (256, 512, 750, 1024, 2048), representation dimensions (128, 256, 512, 1024), and β€” crucially β€” InfoNCE batch size (which determines the tightness of the mutual information lower bound) across a representative set of tasks (one low-dimensional, one medium, one high-dimensional). The key question: does DR.Q's advantage over MSE-only methods grow with network capacity (because the InfoNCE loss provides a better inductive bias that channel

s additional capacity toward task-relevant features) or shrink (because with enough capacity, the MSE loss alone becomes sufficient since the encoder can afford to represent both relevant and irrelevant dimensions)? The answer has direct practical implications: if the gap grows with scale, DR.Q becomes increasingly attractive for large-scale RL; if it shrinks, DR.Q's value is primarily in the low-to-medium-capacity regime. The DR.Q architecture is designed to handle this experiment β€” the only changes needed are the hidden dimension hyperparameter and a scaling of the InfoNCE batch size β€” and the paper's single-hyperparameter-set design means no per-task tuning is required.

Adaptive difficulty-aware loss weighting as a meta-extension of DR.Q. The paper fixes Ξ»m = 0.1 (InfoNCE weight) and Ξ΅ = 0.0001 (forget rate) across all tasks, but the ablation results (Figures 11, 12) reveal that the need for these components varies by task: InfoNCE matters more in high-dimensional environments, and the forget mechanism matters more when early experiences are poor. A natural extension is to make these weights adaptive β€” for example, increasing Ξ»m when the spread of the InfoNCE loss across the batch is small (indicating the representations have saturated in mutual information) or adjusting Ξ΅ based on the policy improvement rate (increasing Ξ΅ when the policy is changing rapidly, since old experiences become obsolete faster). This would convert DR.Q from a fixed-configuration algorithm into one that automatically tunes its debiasing strength to the current training phase and environment characteristics. The paper already provides the conceptual framework (bias diagnosis β†’ targeted correction) and the empirical evidence that different biases dominate in different situations; an adaptive version would close the loop by automating the diagnosis. The measurement infrastructure for such adaptation β€” tracking InfoNCE loss, policy entropy, TD-error distribution over buffer age β€” is already implicit in DR.Q's training loop and would require only logging and a simple meta-controller.

Practical Applications and Downstream Use Cases

High-dimensional robotic control with redundant sensing. DR.Q's strongest empirical results come from HumanoidBench tasks with dexterous hands, where observation dimensions reach 308 and many hand joint angles are irrelevant for locomotion tasks. On h1hand-walk-v0, DR.Q achieves 512 while the next-best baseline (TDMPC2) achieves 234 β€” more than 2Γ— improvement; on h1hand-stand-v0, DR.Q achieves 491 vs. SimBaV2's 103 β€” nearly 5Γ— improvement (Table 10). For robotics practitioners deploying learned policies on real humanoid or multi-fingered platforms, where sensor suites inevitably capture task-irrelevant information (all joint encoders report values regardless of task context, cameras capture background clutter), DR.Q's mutual information objective provides a principled mechanism to focus representational capacity on the sensory channels that actually predict the robot's future state. A team building a humanoid robot that needs to walk, run, stand, and manipulate objects could train a single DR.Q policy with the same hyperparameters across all behaviors, without manually engineering which sensor channels are relevant to each behavior β€” the InfoNCE loss automatically down-weights channels that don't help predict the next state. The 28% improvement over SimBaV2 on HumanoidBench with hands (mean normalized score 0.534 vs. 0.417) translates to substantially faster training and higher final performance without additional engineering effort.

Sample-efficient sim-to-real transfer with limited real-world interaction budgets. DR.Q's replay ratio of 1 means it achieves its sample efficiency with minimal computation per environment step β€” one gradient step for each of encoder, critic, and actor. In sim-to-real transfer scenarios where the real robot is the bottleneck (limited hours of operation, human supervision required for resets, wear and tear on hardware), the number of real-world interactions dominates the cost. DR.Q's strong sample efficiency on complex locomotion β€” reaching 721 on dog-run within 1M environment steps (Table 8), which the paper claims is unprecedented β€” suggests it could learn competent policies from fewer real-robot interactions than methods requiring higher UTD (which multiply gradient steps but don't reduce environment steps). For a lab with a quadrup ed or humanoid platform and a budget of, say, 50K real-world steps for fine-tuning a sim-trained policy, DR.Q's architecture would maximize the value extracted from each of those precious interactions: its representations would adapt to the sim-to-real distribution shift via the dynamics consistency and mutual information objectives (which don't require reward signals, only state transitions), and its faded PER would prevent early real-world experiences (which may be poor due to the initial policy mismatch) from dominating the replay buffer. The paper doesn't evaluate sim-to-real transfer directly, but the combination of high sample efficiency, representation learning from transitions (not just rewards), and robustness to irrelevant state dimensions (Figure 15) makes it a strong candidate for this application.

Batch inference and offline data processing for policy pre-training. Organizations with large datasets of logged robot interactions (from teleoperation, previous experiments, or heuristic controllers) could use DR.Q's encoder pre-training phase to learn high-quality state representations from this offline data before any online interaction. Because the encoder loss (Equation 9) uses only reward prediction, dynamics consistency (MSE), and mutual information (InfoNCE) β€” all of which can be computed from logged transitions without requiring on-policy actions or value function bootstrapping β€” the encoder can be pre-trained on arbitrary offline data without concerns about distributional shift in the value function. The downstream actor-critic would then start online training with representations that already capture the environment's dynamics structure, dramatically accelerating the early phase of learning where most baselines flounder with random exploration. The paper's result that DR.Q with faded PER avoids overfitting to early poor experiences (Figure 12: DR.Q only LAP degrades severely on humanoid-run because it overfits to early random data) directly supports this use case β€” pre-training on diverse offline data would give the encoder a strong initialization, and faded PER would ensure that the first online experiences don't overwrite this useful prior. This is particularly attractive for manipulation tasks where the state space changes little between tasks (same robot, different objects) β€” an encoder pre-trained on a large corpus of manipulation data could serve as a frozen or fine-tuned backbone for many downstream policies, amortizing the cost of representation learning across tasks.

When to Prefer This Method

The paper does not articulate an explicit decision rule comparing DR.Q against named alternatives. It positions DR.Q as a general-purpose improvement over MR.Q β€” adding an InfoNCE loss and faded PER to the same architectural template β€” and as a competitive alternative to SimBaV2, TDMPC2, and FoG in aggregate across the evaluated benchmarks. The paper's primary empirical claim is that DR.Q "matches or surpasses recent strong baselines" (Section 5.1, Figure 1) with a single set of hyperparameters, not that it dominates any specific alternative under well-defined conditions. The limitations section (Section 6) identifies tasks where DR.Q underperforms (Hopper-v4, visual-humanoid-run) but attributes these to the unified hyperparameter choice rather than to fundamental domain characteristics. No per-domain sensitivity analysis is provided that would support a conditional deployment recommendation. Rather than fabricate a decision matrix that the paper does not support, I note this absence: a practitioner choosing between DR.Q and SimBaV2 for a new continuous control task has no evidence-based guidance from this paper about which will perform better β€” the aggregate statistics favor DR.Q on most benchmark groupings, but the per-task variance is high, and no predictive model of when DR.Q excels or fails is developed.