ArXiv: 2006.04779
🎯 Pitch
Standard offline RL catastrophically overestimates Q-values when bootstrapping on actions never seen in the data, but CQL shows you can simply regularize the critic to be conservative rather than constrained, achieving 2–5× higher returns by learning a Q-function that lower-bounds the true policy value only in expectation, not pointwise.
1. Executive Summary
This paper introduces conservative Q-learning (CQL), an algorithmic framework for offline reinforcement learning that learns a Q-function whose expected value under the learned policy lower-bounds the true policy value, thereby addressing the overestimation problem caused by distributional shift between the dataset and the learned policy when bootstrapping from out-of-distribution actions. The authors evaluate CQL across continuous-control tasks from the D4RL benchmark (MuJoCo gym, Adroit, AntMaze, and Franka Kitchen domains) and discrete-control Atari games from Arcade Learning Environment, demonstrating that a simple Q-value regularizer—minimizing Q-values under the current policy while maximizing them under the behavior policy—can be added to standard actor-critic (SAC) and Q-learning (QR-DQN) implementations in fewer than 20 lines of code. On complex, multi-modal datasets where prior methods struggle, CQL attains 2–5× higher final return than the best-performing baselines (e.g., outperforming BEAR and BRAC by over 40 points on walker2d-medium-expert and hopper-random-expert), establishing that a conservative value estimate that lower-bounds only the expected policy value—rather than imposing pointwise lower bounds or explicit policy constraints—suffices to enable stable offline learning across a wide range of domains, while proving through theoretical analysis that CQL produces a high-confidence safe policy improvement guarantee over the behavior policy.
2. Context and Motivation
The Core Problem: Offline RL Breaks Standard RL Algorithms
The paper addresses a deceptively simple question: how can we learn effective policies from previously collected, static datasets without any further environment interaction? This is the central challenge of offline reinforcement learning (also called batch RL), and it represents a fundamental departure from how RL has traditionally been conceived.
Standard RL operates as an active learning process: an agent takes actions in an environment, observes outcomes, and iteratively improves. This works in simulation or controlled settings, but it imposes severe constraints on real-world applicability. The paper identifies specific scenarios where online interaction is prohibitively expensive or dangerous: robotics (where exploration can damage hardware), healthcare (where experimenting with treatments on patients is unethical), recommendation systems (where serving poor recommendations loses users), and autonomous driving (where exploration means real accidents). In all these domains, massive datasets exist—collected from human operators, existing controllers, or logged interactions—but standard RL algorithms cannot effectively learn from them.
The technical reason for this failure is distributional shift in the action space during Q-function bootstrapping. When a standard off-policy algorithm like Q-learning or actor-critic is trained offline, the Bellman backup uses target values computed with actions from the learned policy , but the Q-function is trained only on actions that appear in the dataset (from the behavior policy ). Since is explicitly optimized to maximize predicted Q-values, it naturally gravitates toward actions that the Q-function mistakenly rates as high-value—precisely because those actions have never been seen in the data and their true values are unknown. In online RL, the agent can correct such errors by actually trying those actions and observing the consequences. In offline RL, there is no such corrective mechanism. The errors compound through bootstrapping, leading to erroneously optimistic value estimates and catastrophic policy degradation—what practitioners observe as Q-values diverging to infinity during training.
This is not a minor implementation detail. It is a structural failure mode of off-policy algorithms when decoupled from online data collection. As the paper notes in Section 2:
"Offline RL algorithms based on this basic recipe suffer from action distribution shift during training, because the target values for Bellman backups in policy evaluation use actions sampled from the learned policy, , but the Q-function is trained only on actions sampled from the behavior policy that produced the dataset , ."
Why This Matters: The Gap Between RL's Promise and RL's Practice
The practical importance of solving offline RL is difficult to overstate. The paper frames this in terms of a fundamental asymmetry between supervised learning and reinforcement learning (Section 1). Supervised learning has achieved transformative success—ImageNet classification, BERT-scale language models, recommendation systems—precisely because it can leverage enormous, static datasets collected once and reused indefinitely. The ImageNet dataset was collected once; every subsequent advance in vision has built on it without requiring new data collection. RL, by contrast, has traditionally required every training run to generate its own data through environment interaction. This makes RL vastly more expensive per unit of learned capability.
Offline RL promises to bridge this gap: if we can make RL algorithms work reliably from static datasets, the same data-leveraging paradigm that enabled the deep learning revolution becomes available for sequential decision-making. A factory could collect operator telemetry data for months and then train a policy without ever risking production downtime. A hospital could use historical treatment records to learn better clinical decision policies. A dialogue system could improve from existing conversation logs. The datasets exist—the bottleneck is algorithmic.
The paper also identifies a more subtle theoretical significance. The offline RL problem forces us to confront a question that online RL can largely ignore: how should an agent reason about actions it has never taken? This is fundamentally a question about generalization under uncertainty, and it connects to deep issues in causal inference, robust optimization, and the relationship between data collection policies and learned policies. Solving offline RL requires developing principled answers to these questions, which may yield insights beyond RL itself.
Prior Approaches and Where They Fall Short
The paper identifies three broad families of prior approaches, each with specific limitations that motivate CQL.
Behavior-regularized policy methods. The most common approach in prior offline RL work is to constrain the learned policy to remain "close" to the behavior policy that generated the data. Methods like BEAR (Kumar et al., 2019) use maximum mean discrepancy (MMD) to measure this distance; BRAC (Wu et al., 2019) explores KL-divergence and Wasserstein distance constraints; Siegel et al. (2020) and Jaques et al. (2019) use KL penalties. The shared intuition is: if we prevent from selecting actions far from , then the Q-function's target values will be computed from actions that actually appear in the data, and the overestimation problem is avoided.
The paper identifies a critical practical weakness of these methods (Section 5):
"Most of these methods require a separately estimated model to the behavior policy, , and are thus limited by their ability to accurately estimate the unknown behavior policy."
This estimation is particularly problematic when the dataset is generated from multiple behavior policies (e.g., a mixture of expert demonstrations, suboptimal controllers, and random exploration)—which is precisely the realistic setting where offline RL would be most valuable. The behavior policy in such cases may be highly complex and multi-modal, making accurate density estimation challenging. Furthermore, these methods are fundamentally limited by the quality of the behavior policy constraint: if is too restrictive, the learned policy cannot substantially improve over behavioral cloning; if it is too loose, OOD actions still creep in.
Uncertainty-based methods. Another line of work attempts to estimate the epistemic uncertainty of Q-value predictions and then optimize a lower-confidence bound (or "pessimistic" estimate) rather than the raw Q-value. Methods in this category include bootstrapped DQN ensembles (Osband et al., 2016), which use multiple independently-initialized Q-networks to estimate prediction variance, and REM (Agarwal et al., 2019), which uses random convex combinations of ensemble members. The intuition is appealing: if we know which Q-value predictions are unreliable (because they correspond to state-action pairs far from the data), we can avoid being over-optimistic about them.
The paper argues that this approach has proven insufficient in practice (Section 5, Appendix E):
"These methods have not been generally performant in offline RL due to the high-fidelity requirements of uncertainty estimates in offline RL."
The core issue is that uncertainty estimation techniques developed for exploration in online RL—where loose, uncalibrated estimates that merely indicate relative novelty are sufficient to drive effective exploration—are inadequate for offline RL. In offline RL, the uncertainty estimates must be tight and calibrated to avoid both overestimation (too optimistic about OOD actions) and excessive conservatism (underestimating in-distribution actions, preventing any improvement over behavioral cloning). Most practical uncertainty estimation techniques with neural networks produce estimates that are far from calibrated, leading to either continued overestimation or crippling pessimism.
Robust MDP formulations. A third theoretical perspective frames offline RL as learning under an uncertain MDP, where the transition dynamics and rewards are only partially known (Iyengar, 2005; Petrik et al., 2016; Tamar et al., 2014). These methods construct an uncertainty set around the estimated MDP parameters and optimize the worst-case performance. While theoretically appealing, the paper notes that:
"Robust MDPs have been a popular theoretical abstraction for offline RL, but tend to be highly conservative in policy improvement."
This conservatism arises because robust MDP methods typically protect against the worst-case transition dynamics at all state-action pairs, even those that are well-supported by data. This pointwise pessimism can prevent the policy from exploiting genuinely good actions about which we have sufficient evidence, leading to policies that barely improve over the behavior policy—or worse.
A critical insight the paper elevates: the distinction between pointwise and policy-level lower bounds. The paper identifies a subtle but crucial distinction that prior work generally missed. Most prior methods that aim for conservative value estimates attempt to produce a pointwise lower bound—i.e., for every state-action pair. But this is unnecessarily restrictive. For policy improvement, we only care about the expected value under the policy: . Requiring pointwise lower bounds forces over-conservatism at in-distribution actions (where the Q-function is actually reliable) in order to achieve conservatism at OOD actions. The paper's explicit goal is to achieve the latter bound—policy-level conservatism—which is both theoretically sufficient and empirically much less restrictive.
How This Paper Positions Itself
CQL positions itself as a unified solution that combines the strengths of prior approaches while avoiding their weaknesses. Rather than constraining the policy (which requires behavior policy estimation), estimating uncertainty (which demands calibration), or protecting against worst-case MDP parameters (which induces pointwise conservatism), CQL directly regularizes the Q-function itself during training.
The key insight is remarkably simple in retrospect: add a penalty term to the standard Bellman error objective that pushes down Q-values for actions under the current policy while pushing up Q-values for actions under the dataset distribution. This creates a "gap-expanding" effect (Theorem 3.4): the difference in Q-values between in-distribution and out-of-distribution actions is artificially widened, making OOD actions systematically less attractive to the policy. The learned policy is thus implicitly constrained to favor actions supported by the data—without ever explicitly estimating or building uncertainty sets.
The paper explicitly positions this as a conceptual advance over pointwise lower-bound methods (Section 3.1):
"If we can instead learn a conservative estimate of the value function, which provides a lower bound on the true values, this overestimation problem could be addressed. In fact, because policy evaluation and improvement typically only use the value of the policy, we can learn a less conservative lower bound Q-function, such that only the expected value of Q-function under the policy is lower-bounded, as opposed to a point-wise lower bound."
This distinction—between pointwise and policy-level lower bounds—is the paper's primary theoretical innovation. It allows CQL to be less conservative than methods that require underestimation at every state-action pair, while still providing formal guarantees (Theorem 3.2) that the expected value is a lower bound.
The paper also explicitly positions itself against the practical complexity of prior methods. CQL can be implemented in fewer than 20 lines of code on top of existing SAC or DQN implementations (Section 4). It does not require:
- Estimating the behavior policy (unlike BEAR, BRAC, and other policy-constraint methods)
- Training ensemble models for uncertainty (unlike bootstrapped DQN or REM, though ensembles can still be used for the base Q-function)
- Solving robust optimization problems (unlike robust MDP approaches)
- Any auxiliary models beyond the Q-function and policy already present in standard actor-critic algorithms
This simplicity is not merely an implementation convenience—it eliminates entire categories of estimation error. When a policy constraint method fails because is poorly estimated on a multi-modal dataset, that failure mode simply does not exist for CQL. The regularizer operates directly on the Q-function's predictions, using only samples from the dataset.
Finally, the paper provides a theoretical unification of these ideas (Section 3.2, Theorem 3.5). It shows that CQL is equivalent to optimizing a penalized RL objective in the empirical MDP: maximize the return of in the dataset-derived MDP , minus a penalty proportional to . This penalty is a form of divergence that penalizes policies that deviate from the behavior policy in a state-conditional way. The connection to safe policy improvement (Theorem 3.6) provides the formal guarantee that the resulting policy cannot be much worse than , with the degradation bounded by a term that decays as dataset size increases—providing exactly the kind of high-confidence safety guarantee that makes offline RL deployable in practice and connecting CQL to the SPIBB (Safe Policy Improvement with Baseline Bootstrapping) framework of Laroche et al. (2017).
3. Technical Approach
This is primarily a theoretical and algorithmic paper whose core idea is that offline RL can be made stable and effective by learning a Q-function that is conservative — specifically, one whose expected value under the learned policy provides a lower bound on the true policy value — achieved through a simple Q-value regularizer that penalizes overestimation on out-of-distribution actions while preserving accurate values for in-distribution actions, without requiring explicit policy constraints or uncertainty estimation.
3.1 Reader Orientation
The paper builds a system for learning a decision-making policy from a fixed, previously collected dataset, where the system cannot interact with the environment to try new actions and observe their consequences. The core problem is that standard RL algorithms become over-optimistic: they assign unrealistically high values to actions that never appear in the dataset, causing the learned policy to favor these untested (and likely poor) actions. The solution takes the form of a modified Q-learning objective that adds a regularizer — a penalty term — which systematically pushes down Q-values for actions the learned policy wants to take (especially those far from the data) while pushing up Q-values for actions that actually appear in the dataset. This creates an "artificial gap" between safe, in-distribution actions and risky, out-of-distribution actions, so that the policy naturally prefers actions supported by evidence.
3.2 Big-Picture Architecture (Diagram in Words)
The CQL system modifies a standard off-policy actor-critic or Q-learning algorithm by replacing the standard Q-function training objective with a new objective. The architecture has three major components:
-
Q-function (with conservative regularizer) — a neural network that estimates the expected return of state-action pairs. It is trained not only to minimize Bellman error (as in standard RL) but also to push down Q-values under the current policy's action distribution while pulling up Q-values under the dataset's action distribution. This is the core of CQL.
-
Policy network (actor-critic variant only) — a neural network that outputs actions given states, trained to maximize the conservative Q-function's predictions. Because the Q-function is artificially pessimistic about out-of-distribution actions, this policy is implicitly constrained to stay near the data distribution without any explicit behavior policy constraint.
-
Automatic Lagrange multiplier (optional, for continuous control) — a scalar α that balances the Bellman error term against the conservative regularizer. It is automatically adjusted via dual gradient descent to maintain a target level of conservatism rather than requiring manual tuning.
Information flows as follows: a batch of transitions (state, action, reward, next state) is sampled from the offline dataset → the current policy is used to sample hypothetical actions at both current and next states → the Q-function computes predictions for all these actions → the CQL objective is computed (Bellman error plus the conservative regularizer) → the Q-function parameters are updated via gradient descent → (for actor-critic) the policy is updated to maximize the conservative Q-values → (for Lagrange variant) α is updated to maintain the target conservatism level → the cycle repeats for many gradient steps.
3.3 Roadmap for the Deep Dive
- First, the conservative off-policy evaluation objective (Equations 1 and 2), which establishes the theoretical machinery for learning a lower-bound Q-function — since this is the foundation that the full RL algorithm is built upon.
- Second, the full CQL family of RL algorithms (Equation 3 and variants), which extends the evaluation objective into a complete policy learning procedure — since understanding the discrete choices (CQL(H) vs CQL(ρ)) and the role of the regularizer R(μ) is essential for practical use.
- Third, the gap-expanding property (Theorem 3.4) and why it matters — since this is the mechanism by which CQL implicitly constrains the policy without needing an explicit behavior policy model.
- Fourth, the theoretical guarantees: lower-bound results (Theorems 3.1–3.3), the penalized RL equivalence (Theorem 3.5), and the safe policy improvement guarantee (Theorem 3.6) — since these provide the formal justification for why CQL is a principled approach.
- Fifth, the practical algorithm and implementation details (Section 4), including the specific CQL(H) and CQL(ρ) objectives, automatic α tuning, and integration with SAC and QR-DQN — since the gap between theory and working code requires careful design choices.
3.4 Detailed, Sentence-Based Technical Breakdown
Conservative Off-Policy Evaluation: Learning a Lower-Bound Q-Function
The paper begins not with a complete RL algorithm, but with a simpler sub-problem: off-policy evaluation. Given a fixed policy π and a static dataset D generated by behavior policy πβ, can we estimate the policy's value V^π(s) without overestimating it? The answer is the foundation for CQL.
The starting observation is that standard Bellman error minimization:
where $\hat{B}^\pi$ is the empirical Bellman operator using sampled transitions, $\hat{Q}^k$ is the Q-function from the previous iteration, and $D$ is the offline dataset — produces Q-values that can be erroneously high for actions not well-represented in D. This is because the empirical Bellman backup at state-action pairs with few samples can be optimistically biased (it sees a lucky high-reward transition but not the full distribution of possible outcomes).
CQL's first key insight: add a penalty that pushes down Q-values everywhere. The simplest conservative objective is:
where $\alpha > 0$ is a tradeoff coefficient controlling conservatism strength, and $\mu(a|s)$ is a distribution over actions at each state that we choose — the Q-function is penalized for having high values under μ.
What it computes: this objective simultaneously minimizes two terms. The first term $\alpha \cdot \mathbb{E}_{\mu}[Q(s,a)]$ penalizes large Q-values for actions drawn from μ — it is a "push down" force. The second term is the standard Bellman error that makes Q-values consistent with observed rewards and next-state values — it is a "fit the data" force. The Q-function learned is the compromise: accurate on in-distribution transitions (where Bellman error dominates) but low on actions from μ (where the penalty dominates).
Why this form: by choosing μ to cover actions that the learned policy might take, we systematically suppress Q-values for potentially problematic actions. Theorem 3.1 proves that for any μ whose support is contained in the support of the empirical behavior policy , the learned Q-function satisfies a pointwise lower bound: for all s,a in the dataset, provided α is sufficiently large to overcome sampling error. The penalty term $-\alpha \frac{\mu(a|s)}{\hat{\pi}_\beta(a|s)}$ appears in the fixed-point equation (derived by setting the gradient to zero in the tabular setting), guaranteeing systematic underestimation.
But pointwise lower bounds are too conservative. The paper identifies a crucial weakness of Equation 1: it forces underestimation at every state-action pair, including those where the Q-function is perfectly reliable (e.g., actions well-represented in the data). This unnecessary conservatism can prevent the policy from exploiting good actions. The paper's second key insight: we only need the expected value under the policy to be a lower bound, not the Q-value at every individual action.
The tightened objective introduces an additional maximization term:
where $\hat{\pi}_\beta(a|s)$ is the empirical behavior policy computed from the dataset counts at state s, and the new term subtracts the expected Q-value under the behavior policy.
What it computes: the net penalty is now $\mathbb{E}_{\mu}[Q] - \mathbb{E}_{\hat{\pi}_\beta}[Q]$ — the difference between expected Q-value under our chosen distribution μ and under the behavior policy. If μ assigns high probability to actions that the behavior policy rarely takes, this penalty is large (suppressing Q-values on those OOD actions). But if an action is common under both μ and the behavior policy, the penalty largely cancels, allowing the Q-function to be accurate for in-distribution actions. The Q-function is pulled down where μ differs from the data distribution and pulled up where it matches.
Why this form matters: Theorem 3.2 proves that when μ = π (the target policy), the expected value satisfies $\hat{V}^\pi(s) = \mathbb{E}_{\pi(a|s)}[\hat{Q}^\pi(s,a)] \leq V^\pi(s)$ — a policy-level lower bound rather than a pointwise one. The penalty term that appears in the value fixed-point equation is $\sum_a \pi(a|s)\left(\frac{\pi(a|s)}{\hat{\pi}_\beta(a|s)} - 1\right) \geq 0$, which is always non-negative (proved by rewriting as $\sum_a \frac{(\pi(a|s) - \hat{\pi}_\beta(a|s))^2}{\hat{\pi}_\beta(a|s)}$). This quantity is zero only when π exactly matches the behavior policy — any deviation induces underestimation proportional to the χ²-like divergence between π and π̂_β. This is the mathematical basis for CQL's conservatism: the more the learned policy diverges from the data distribution, the more the value estimate is pushed down.
Crucially, pointwise lower bounds are NOT guaranteed by Equation 2. The proof of Theorem 3.2 explicitly notes that the maximization term under π̂_β can cause Q-values at individual actions to be overestimated — specifically, actions common under π̂_β may have $\hat{Q}^\pi(s,a) > Q^\pi(s,a)$. But this is acceptable because (a) these are precisely the actions we trust, and (b) the aggregate value under the policy is still a lower bound. This is the central theoretical distinction that makes CQL less conservative than methods requiring pointwise bounds.
The necessity of maximizing under π̂_β. A natural question: why must the maximization distribution be the behavior policy specifically? Theorem D.3 (Appendix D.2) proves that for the expected value lower bound to hold for all possible choices of the target policy π, the maximization distribution must be π̂_β. If any other distribution ν is used, there exists some π for which the penalty term becomes negative and the lower-bound guarantee fails. This is proved via a concave-convex max-min optimization: finding ν that maximizes the minimum (over π) of the penalty $\sum_a \pi(a|s)\frac{\pi(a|s) - \nu(a|s)}{\hat{\pi}_\beta(a|s)}$. The optimal ν is exactly $\hat{\pi}_\beta$.
From Conservative Evaluation to Conservative Q-Learning: The CQL(R) Family
With the evaluation objective established, the paper builds toward a complete offline RL algorithm. The key challenge: in RL, the policy π is not fixed — it is updated based on the Q-function. So the distribution μ used in the penalty should track the improving policy.
The CQL(R) family formalizes this as a joint optimization:
where $\mathcal{R}(\mu)$ is a regularizer on μ that controls which distribution the inner maximization selects, $\pi_k$ (or $B^*$ for Q-learning) determines whether policy evaluation or optimality is used in the Bellman backup, and the inner max over μ with the minus sign in front means we are finding the worst-case (most overestimated) action distribution and then minimizing Q-values under it.
What it computes: the inner maximization over μ finds the action distribution that would produce the largest Q-values (adjusted by the regularizer R) — this is the adversarial distribution that the Q-function might be tempted to overestimate. The outer minimization over Q then pushes Q-values down specifically on those actions. The $-\mathbb{E}_{\hat{\pi}_\beta}[Q]$ term ensures that Q-values on data-supported actions are not penalized (they are implicitly pulled up relative to the μ distribution). The result is a Q-function that is pessimistic precisely where needed — on actions that differ from the data.
Why this form: it turns conservative Q-learning into a game: μ tries to find actions where the Q-function might overestimate, and Q defends by lowering its predictions on those actions. The regularizer R(μ) controls how aggressively μ searches for overestimated actions. Different choices of R yield different practical algorithms.
CQL(H) — the entropy-regularized variant. When $\mathcal{R}(\mu) = -D_{KL}(\mu, \text{Unif}) = H(\mu)$ is chosen as the negative entropy (equivalently, KL divergence to the uniform distribution), the inner maximization over μ can be solved in closed form. For an optimization problem of the form:
the optimal solution is $\mu^*(a|s) = \frac{1}{Z} \exp(f(a))$, where Z is the normalizing constant — the softmax distribution over f. Substituting $f(a) = Q(s,a)$ gives $\mu(a|s) \propto \exp(Q(s,a))$. Plugging this back into the CQL(R) objective yields:
What it computes: the first term is log-sum-exp of Q-values across all actions — this is a smooth approximation of $\max_a Q(s,a)$ (the "soft maximum"). It penalizes the Q-function for having any action with a large value. The second term is the average Q-value under the behavior policy — this rewards the Q-function for giving high values to in-distribution actions. The net penalty is high when the maximum Q-value exceeds the average in-distribution Q-value.
Why this form is practical: for discrete action spaces (Atari), log-sum-exp can be computed exactly. For continuous action spaces, it is estimated via importance sampling: sample N actions from both a uniform distribution and the current policy, then compute $\log\left(\frac{1}{2N}\sum_{a_i \sim \text{Unif}} \frac{\exp(Q(s,a_i))}{\text{Unif}(a_i)} + \frac{1}{2N}\sum_{a_i \sim \pi} \frac{\exp(Q(s,a_i))}{\pi(a_i|s)}\right)$. The paper uses N=10 for continuous control experiments. The uniform sampling ensures coverage of the action space; the policy sampling focuses computation where the Q-function is likely to be larger.
CQL(ρ) — the policy-conditioned variant. When $\mathcal{R}(\mu) = -D_{KL}(\mu, \rho)$ for some prior distribution ρ(a|s), the optimal μ becomes $\mu(a|s) \propto \rho(a|s) \cdot \exp(Q(s,a))$. If ρ is chosen as the previous policy $\hat{\pi}_{k-1}$, then the μ distribution concentrates on actions that are both high-value under the current Q-function and likely under the previous policy. This is more stable in high-dimensional action spaces (like the 24-DoF Adroit hand) because the importance sampling estimate of log-sum-exp can have high variance when sampling from broad distributions. By focusing the adversarial distribution μ on actions near the current policy, CQL(ρ) provides a more targeted form of conservatism that avoids the variance issues of CQL(H).
Automatic tuning of α (Lagrange version). Instead of manually choosing α, the paper introduces a dual gradient descent procedure that maintains a constraint on the expected conservatism:
where $\tau$ is a target threshold for the expected difference between log-sum-exp Q and in-distribution Q.
What it computes: if the average gap $\log\sum\exp(Q) - \mathbb{E}_{\hat{\pi}_\beta}[Q]$ exceeds τ, the Lagrange multiplier α grows (because the constraint $\text{gap} \leq \tau$ is violated), increasing the penalty strength. If the gap is less than τ, α shrinks, relaxing conservatism. This automates the tradeoff: the user sets τ (how conservative they want to be) and the algorithm adjusts α to achieve it. For D4RL Gym MuJoCo domains, τ=10.0 is used; for Franka Kitchen and Adroit domains, τ=5.0 provides stronger conservatism on the more challenging human-demonstration datasets.
The Gap-Expanding Property: Why CQL Implicitly Constrains the Policy
Theorem 3.4 establishes a property that the paper calls "gap-expanding": at each iteration, CQL increases the difference between the expected Q-value under the behavior policy and the expected Q-value under the adversarial distribution μ, relative to what that difference would be with standard (non-conservative) Q-learning:
where $\hat{Q}^k$ is the CQL Q-function, $Q^k$ is the Q-function from standard Bellman backups, $\pi_\beta$ is the behavior policy, and $\mu_k$ is the distribution chosen by the inner maximization at iteration k.
What this means operationally: if function approximation error causes some out-of-distribution action to have an erroneously high Q-value (making it attractive to the policy), CQL's gap-expanding property makes that OOD action look less attractive relative to in-distribution actions than it would under standard Q-learning. The Q-function learned by CQL systematically favors actions that appear in the data.
Why this is important: it means CQL provides robustness to Q-function approximation error without explicitly constraining the policy. The policy $\pi_k(a|s) \propto \exp(\hat{Q}^k(s,a))$ is naturally biased toward the data distribution because the Q-values themselves have been reshaped to create an artificial advantage for in-distribution actions. This is a more fundamental solution than adding a separate policy constraint term: rather than fighting the Q-function (which wants to maximize values and may pull the policy toward OOD actions), CQL makes the Q-function itself prefer in-distribution actions.
The gap scales with α. The proof of Theorem 3.4 shows that the additional gap introduced at iteration k is $-\alpha_k \hat{\Delta}_k$, where $\hat{\Delta}_k = \sum_a \frac{(\mu_k(a|s) - \pi_\beta(a|s))^2}{\pi_\beta(a|s)} \geq 0$ is a measure of how much μk diverges from the behavior policy. Larger α means a larger gap, making CQL more conservative. This is the mechanism by which α controls conservatism: it directly determines how much "extra advantage" in-distribution actions receive.
Theoretical Guarantees: Lower Bounds, Penalized RL, and Safe Improvement
The paper provides three categories of theoretical results, moving from evaluation to learning to deployment guarantees.
Theorem 3.1 (Pointwise lower bound for Equation 1):
where $(I - \gamma P^\pi)^{-1}$ is the matrix encoding the expected discounted state-action occupancies under policy π, $\frac{\mu}{\hat{\pi}_\beta}$ is the vector of density ratios $\mu(a|s)/\hat{\pi}_\beta(a|s)$, and the last term captures sampling error (bounded by $C_{r,T,\delta}R_{\max}/((1-\gamma)\sqrt{|\mathcal{D}(s,a)|})$ with probability ≥1-δ).
What it computes: the underestimation at each (s,a) consists of a "deliberate" component (proportional to α times the discounted sum of future density ratios) minus any "accidental" overestimation from finite-sample bias in the empirical Bellman operator. By choosing α large enough to dominate the maximum sampling error, we guarantee $\hat{Q}^\pi \leq Q^\pi$ pointwise.
Theorem 3.2 (Policy-level lower bound for Equation 2, the tightened version):
where $\mathbb{E}_\pi\left[\frac{\pi}{\hat{\pi}_\beta} - 1\right]$ is the state-conditional χ²-like divergence $\sum_a \pi(a|s)(\frac{\pi(a|s)}{\hat{\pi}_\beta(a|s)} - 1) \geq 0$ — this is non-negative and is zero only when π = π̂_β.
What it computes: the value underestimation is proportional to the discounted sum of divergences between π and π̂_β at visited states. If the learned policy stays close to the behavior policy, the underestimate is small; if it deviates substantially, the value estimate is strongly penalized. This is tight in the sense that when π = π̂_β, the deliberate underestimation vanishes (leaving only sampling error, which also vanishes with infinite data).
Theorem 3.3 (CQL with policy optimization still lower-bounds): Let $\pi_{\hat{Q}^k}(a|s) \propto \exp(\hat{Q}^k(s,a))$ be the soft-optimal policy for the current Q-function. If the actual policy update is slow — $D_{TV}(\hat{\pi}_{k+1}, \pi_{\hat{Q}^k}) \leq \varepsilon$ — then the policy value under $\hat{Q}^k$ lower-bounds the true value as long as:
What this means in practice: the conservatism induced by CQL in the Q-function (left side) must exceed the potential overestimation caused by the policy not exactly matching the soft-optimal policy (right side). If ε is small (policy changes slowly), a modest amount of conservatism suffices. This is why the paper uses a smaller policy learning rate (3e-5) than the Q-function learning rate (3e-4 or 1e-4) — it ensures the policy moves slowly enough that the lower-bound property holds. In the limit of infinitesimally small policy updates (ε → 0), any α > 0 guarantees a lower bound.
Theorem 3.5 (CQL optimizes a penalized empirical RL objective): The optimal policy $\pi^*(a|s) = \arg\max_\pi \mathbb{E}_{\rho(s)}[\hat{V}^\pi(s)]$ obtained from CQL is equivalent to:
where $J(\pi, \hat{M})$ is the expected return of π in the empirical MDP M̂ constructed from the dataset (transitions and rewards as observed in D), $d^\pi_{\hat{M}}(s)$ is the discounted state visitation distribution under π in M̂, and $D_{CQL}(\pi, \hat{\pi}_\beta)(s) = \sum_a \pi(a|s)\left(\frac{\pi(a|s)}{\hat{\pi}_\beta(a|s)} - 1\right)$ is the state-conditional divergence.
What it computes: CQL implicitly solves a constrained optimization: maximize the empirical return, minus a penalty for deviating from the behavior policy. The penalty is state-dependent and weighted by how often the state is visited — states visited more frequently incur more penalty for deviating. This is the dual of explicit policy constraint methods: rather than hard-constraining $D(\pi, \pi_\beta) \leq \epsilon$, CQL adds a soft penalty $-\alpha D_{CQL}(\pi, \pi_\beta)$ to the objective.
Why this perspective matters: it reveals that CQL is not ad-hoc — it is the natural Lagrangian of a constrained optimization problem, where α is the Lagrange multiplier. This connects CQL to the broader literature on regularized MDPs and provides a principled way to think about the choice of α.
Theorem 3.6 (Safe policy improvement guarantee): The CQL-optimal policy $\pi^*$ satisfies, with probability ≥ 1-δ:
where $C_{r,\delta}$ and $C_{T,\delta}$ are concentration constants for rewards and transitions, $|\mathcal{D}(s)|$ is the number of times state s appears in the dataset, and the empirical improvement term is non-negative because π* optimizes the penalized objective in M̂ (and π̂_β incurs no penalty since $D_{CQL}(\hat{\pi}_\beta, \hat{\pi}_\beta) = 0$).
What this guarantees: the actual performance of π* in the true MDP M is at most worse than the behavior policy by an amount that depends on (a) sampling error — larger datasets reduce this penalty (inverse square root of state counts), and (b) the divergence between π* and π̂_β — the second term penalizes policies that deviate far from the data. Crucially, if the sampling error is small (large dataset) and the policy doesn't stray too far from the behavior policy, we are guaranteed improvement. This is a high-confidence safe policy improvement result: we can deploy π* with a formal guarantee that it won't be much worse than the status quo policy π̂_β.
Connection to prior safe improvement results: The paper explicitly compares this to Theorems 1 and 2 of Laroche et al. (2017) (SPIBB). Both bounds have a quadratic dependence on the horizon (from the $(1-\gamma)^{-2}$ term) and an inverse square-root dependence on state counts. The difference is that CQL's penalty depends on $D_{CQL}$ (a χ²-type divergence) rather than the ∞-norm constraint used in SPIBB, which the paper argues makes CQL less conservative.
Practical Algorithm and Implementation
Algorithm 1 (pseudocode): The CQL algorithm modifies standard deep RL in two places. First, the Q-function update uses the CQL objective (Equation 4 for CQL(H)) instead of pure Bellman error (Line 3). Second, the policy update (for actor-critic) is unchanged from standard maximum-entropy RL: update the policy to maximize $\mathbb{E}_{a \sim \pi}[Q_\theta(s,a) - \log \pi(a|s)]$ (Line 4). The tradeoff factor α is either fixed (for discrete-control Atari: α=1.0 for 10% data, α=4.0 for 1% data, α=0.5 for the first-20% condition in Figure 1) or automatically tuned via the Lagrange method (for continuous-control D4RL: τ=10.0 for MuJoCo gym, τ=5.0 for Adroit and Kitchen).
Integration with SAC (continuous control): The base implementation is soft actor-critic. The Q-function is trained with the CQL(H) objective. For the log-sum-exp computation in continuous action spaces, 10 actions are sampled from both Uniform(a) and the current policy π(a|s), and importance weighting is used as described in Appendix F. The Q-function learning rate is 3e-4 (SAC default). The policy learning rate is reduced to 3e-5 (slower updates per Theorem 3.3). Twin Q-networks, soft target updates, and entropy regularization from SAC are retained. Training runs for 1M gradient steps on all D4RL domains.
Integration with QR-DQN (discrete control): For Atari, the base implementation is quantile regression DQN. The CQL(H) objective is used with the exact log-sum-exp over the discrete action space (computed via tf.reduce_logsumexp). The α value is fixed per experiment condition. All hyperparameters from the QR-DQN implementation of Agarwal et al. (2019) are kept identical except for the added CQL regularizer.
Key hyperparameter choices and their justifications:
- Policy learning rate (3e-5 vs. standard 3e-4): Theorem 3.3 requires
$D_{TV}(\hat{\pi}_{k+1}, \pi_{\hat{Q}^k}) \leq \varepsilon$for the lower-bound guarantee. A 10× smaller learning rate ensures the policy changes slowly enough relative to Q-function updates. Empirically, 3e-5 was found to "almost uniformly attain good performance" on continuous control tasks. - Lagrange threshold τ: τ=10.0 on MuJoCo gym tasks (less conservative, since these datasets are larger and the tasks are simpler) versus τ=5.0 on Adroit and Kitchen (more conservative, since these have human demonstrations and complex action spaces). τ=2.0 led to α growing to "the order of millions" and "highly underestimated Q-functions," while τ=10.0 on Adroit "was unable to prevent overestimation" — confirming the importance of task-appropriate conservatism.
- Number of action samples for log-sum-exp (10): Using both uniform and on-policy samples balances coverage of the action space with computational efficiency.
- No behavior policy estimation needed: Unlike BEAR, BRAC, and other policy-constraint methods, CQL's regularizer uses only samples from the dataset (for the
$\mathbb{E}_{\hat{\pi}_\beta}[Q]$term) without separately modeling π̂_β. This eliminates the brittleness that arises when the estimated behavior policy is inaccurate on multi-modal datasets.
What happens during training: At each gradient step, a batch of transitions (s,a,r,s') is sampled from D. For continuous control: actions are sampled from the current policy at both s (to compute the current-policy contribution to log-sum-exp) and s' (to compute the Bellman target). The Q-function loss is the sum of Bellman error (half MSE) and the CQL regularizer (α times log-sum-exp minus average Q on the batch actions). For the Lagrange variant, α is then updated via dual gradient ascent: $\alpha \leftarrow \alpha + \eta_\alpha(\text{CQL penalty} - \tau)$, with α clipped to be non-negative. The policy is updated via the standard SAC policy gradient. This cycle repeats for 1M steps, with no interaction with the environment.
4. Key Insights and Innovations
Innovation 1: Policy-Level Lower Bounds as a New Design Principle for Offline RL
The most fundamental conceptual shift in this paper is the distinction between pointwise and policy-level lower bounds on the Q-function. Prior offline RL methods that aimed for conservatism — whether through uncertainty estimation (Osband et al., 2016; Agarwal et al., 2019), robust MDP formulations (Iyengar, 2005; Petrik et al., 2016), or penalty-based approaches (Kumar et al., 2019) — implicitly sought a uniform guarantee: $\hat{Q}(s,a) \leq Q(s,a)$ for every state-action pair. This seems natural: if we underestimate everywhere, we certainly won't overestimate anywhere. But the paper identifies this as a categorical mistake in designing conservative algorithms.
The error is that policy improvement only ever uses the expected value under the policy, $\mathbb{E}_{\pi(a|s)}[\hat{Q}(s,a)]$, never the Q-value at individual actions in isolation. Requiring pointwise lower bounds forces over-conservatism at actions the behavior policy actually takes — where the Q-function is perfectly reliable — simply to achieve conservatism at actions it doesn't take. It's like requiring every individual measurement in a scientific study to be conservative, rather than just the final conclusion. The paper's insight: you only need the aggregate to be a lower bound.
This is not a minor refinement. It represents a fundamental reconceptualization of what "conservatism" means in offline RL. Theorem 3.2 formalizes this: the penalty term that appears in the policy value is $\sum_a \frac{(\pi(a|s) - \hat{\pi}_\beta(a|s))^2}{\hat{\pi}_\beta(a|s)} \geq 0$, which is zero when π matches the behavior policy and grows with deviation. The Q-function can be overconfident at individual in-distribution actions (Theorem 3.2 explicitly does not guarantee pointwise lower bounds), and this is not just acceptable but desirable — it means the policy can confidently exploit actions the data supports while still being pessimistic about actions it doesn't.
The paper validates this empirically through the comparison in Table 4: CQL(H) achieves negative differences between predicted and actual policy values (around -43, -11, and -7 on three hopper datasets), confirming it lower-bounds the true value, while policy-constraint methods like BEAR show positive differences (+66, +1399, +4.3), meaning they overestimate. Critically, the variant of CQL using Equation 1 (the pointwise lower-bound formulation) produces much more negative differences (-151, -23, -157), demonstrating that the pointwise approach is unnecessarily conservative. The policy-level formulation (Equation 2) is tighter: it still guarantees a lower bound on value (preventing catastrophic overestimation) without the crippling pessimism that makes pointwise methods barely outperform behavioral cloning. This conceptual advance — figuring out what quantity actually needs to be bounded — is what enables CQL to achieve 2–5× performance gains over prior methods on complex datasets while using a simpler algorithm.
Innovation 2: Understanding Offline RL as a Gap-Expansion Problem Rather Than a Policy-Constraint Problem
The dominant paradigm in offline RL before CQL was policy constraint: explicitly force the learned policy to stay close to the behavior policy, typically by some divergence measure like KL, MMD, or Wasserstein distance (Kumar et al., 2019; Wu et al., 2019; Siegel et al., 2020). The reasoning was straightforward: if the policy can't go to regions where the Q-function is unreliable, the overestimation problem is avoided.
CQL proposes a fundamentally different mechanism. Rather than constraining the policy directly, it reshapes the Q-function so that in-distribution actions are systematically preferred to out-of-distribution actions — a property the paper calls "gap-expanding" (Theorem 3.4). The Q-function itself is modified to create an artificial advantage for actions that appear in the data. When the policy then selects actions that maximize this reshaped Q-function, it is implicitly constrained to stay near the data distribution — not because it's forced to, but because the Q-values it's optimizing have been redesigned to make OOD actions look worse.
This is a significant conceptual shift because it attacks the root cause rather than a symptom. Policy constraints treat the Q-function's tendency to overestimate OOD actions as a given, and try to prevent the policy from exploiting that tendency. CQL instead asks: why not fix the Q-function itself? If the Q-function systematically prefers in-distribution actions, the policy doesn't need to be constrained. This is more elegant because it eliminates an entire source of estimation error — the behavior policy estimator required by policy-constraint methods — and because the gap-expanding property directly counteracts function approximation error (the mechanism that creates spuriously high Q-values at OOD actions in the first place).
The empirical evidence for this is in Appendix B, Figure 2. The quantity $\hat{\Delta}_k = \mathbb{E}_{s,a \sim D}[\max_{a'}\hat{Q}^k(s,a') - \hat{Q}^k(s,a)]$ measures the "advantage" of OOD actions over in-distribution actions under the learned Q-function. For the policy-constraint method BEAR on hopper-expert-v0, $\hat{\Delta}_k$ is positive and grows during training (reaching values of 30–50), meaning the Q-function thinks OOD actions are better than the expert's actions — exactly the pathology that offline RL must avoid. For CQL, $\hat{\Delta}_k$ is reliably negative (-2 to -10), meaning the Q-function correctly ranks in-distribution actions above OOD ones. The result is that BEAR's policy performance eventually deteriorates (the "unlearning" phenomenon), while CQL's does not. This diagnostic — measuring the gap, not just the final return — reveals a deeper truth about what offline RL algorithms need to achieve: not just high performance, but a Q-function that correctly encodes which actions are trustworthy, even under function approximation error.
Innovation 3: A Unified Framework Connecting Conservative Q-Learning to Safe Policy Improvement with Explicit Dependence on a χ²-Type Divergence
The paper's theoretical analysis does more than prove correctness — it reveals what CQL is actually optimizing. Theorem 3.5 shows that the policy obtained from CQL is the solution to:
This is illuminating in several ways that go beyond the algorithm itself. First, it shows that CQL is optimizing a well-defined penalized objective, not an ad-hoc regularizer. The penalty term is a χ²-type divergence between the learned policy and the behavior policy, weighted by state visitation frequency. This connects CQL to the principled framework of regularized MDPs and gives a clear interpretation: α controls how much we penalize deviation from the data distribution.
Second, the form of the penalty — $\sum_a \frac{\pi(a|s)^2}{\hat{\pi}_\beta(a|s)} - 1$ — is specific. It's not KL divergence (which would be $\sum_a \pi(a|s) \log \frac{\pi(a|s)}{\hat{\pi}_\beta(a|s)}$), not total variation, not MMD. This matters because different divergences have different properties: χ²-divergence penalizes large density ratios more aggressively than KL, which means CQL is particularly averse to the policy putting high probability on actions that were unlikely under the behavior policy. This is appropriate for offline RL: an action that barely appeared in the data is highly uncertain, and we should be very cautious about adopting it, no matter how promising the Q-function makes it look.
Third, the safe policy improvement result (Theorem 3.6) provides a deployment guarantee that explicitly decomposes the performance bound into sampling error and empirical improvement:
The guarantee is: you won't be much worse than the status quo (behavior policy), and if the dataset is large enough and your policy improvement in the empirical MDP is genuine, you'll be better. This is a high-confidence safe improvement result that directly parallels the SPIBB framework (Laroche et al., 2017), but with a dependence on $D_{CQL}$ rather than the ∞-norm constraint used in SPIBB. The practical implication is that CQL provides formal safety assurances — crucial for healthcare, autonomous systems, and any domain where deploying a worse policy than the current one is unacceptable — while being substantially less conservative than prior safe improvement methods in the same theoretical framework.
The significance here is not that CQL has theorems (many offline RL papers do), but that the theorems reveal what structure matters. The divergence that CQL implicitly minimizes, the way sampling error enters the bounds, and the decomposition into safety and improvement terms all provide a conceptual vocabulary for thinking about offline RL that extends beyond this specific algorithm.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. The experiments span three major benchmark suites: (1) the D4RL benchmark (Fu et al., 2020) for continuous control, consisting of MuJoCo gym tasks (HalfCheetah, Hopper, Walker2d), Adroit dexterous manipulation tasks (pen, hammer, door, relocate), AntMaze navigation tasks (umaze, medium, large), and Franka Kitchen tasks; (2) the Arcade Learning Environment (Bellemare et al., 2013) for discrete control with image observations, using the DQN-replay dataset released by Agarwal et al. (2019) at 1%, 10%, and 20% subsampling levels; and (3) the paper uses the standard D4RL train/test splits with normalized return scoring as specified by Fu et al. (2020). The D4RL MuJoCo datasets include several composition types: single-policy datasets ("random", "medium", "expert"), mixed datasets combining multiple policies ("medium-expert", "random-expert"), and "mixed" datasets from diverse behavior policies. The Adroit tasks use human demonstrations ("human") and behavior-cloned policies ("cloned"). The AntMaze tasks provide suboptimal trajectory data requiring the agent to compose fragments into goal-reaching policies. The Kitchen tasks provide human teleoperation data with sparse 0-1 completion rewards. For Atari, the datasets are generated from the replay buffer of an online DQN agent, subsampled to 1%, 10%, or the first 20% of samples.
-
Base model(s). For continuous control, CQL is built on top of soft actor-critic (SAC) (Haarnoja et al., 2017), using the default architecture of twin Q-networks with 256-unit hidden layers and a separate policy network with Gaussian action distributions and entropy regularization. For discrete control (Atari), CQL is built on top of quantile regression DQN (QR-DQN) (Dabney et al., 2018), using the convolutional architecture standard for Atari benchmarks. The choice to build on SAC and QR-DQN rather than developing new architectures is deliberate: it demonstrates that CQL's improvements come from the regularizer itself, not from network design, and it means CQL can be adopted by adding fewer than 20 lines of code to existing implementations. No pre-training or transfer from other domains is used — all methods are trained from scratch on each dataset.
-
Metrics. The primary metric is normalized average return, computed as
(score − random_score) / (expert_score − random_score) × 100for D4RL tasks, where random and expert scores are benchmark-provided reference values (following Fu et al., 2020). This yields a percentage where 0 corresponds to a random policy and 100 corresponds to an expert policy. For Atari, the metric is raw game score, following the convention of Agarwal et al. (2019) and the ALE benchmark. All D4RL results are averaged over 4 random seeds with the smooth, undiscounted episodic return reported. For value estimation verification (Table 4), the paper reports the difference between predicted and actual policy values by evaluating the learned policy's true Monte Carlo return against the Q-function's expected value estimate. -
Baselines. The paper compares against a comprehensive set of prior offline RL methods: BEAR (Kumar et al., 2019), a policy-constraint method using maximum mean discrepancy (MMD) between the learned policy and behavior policy; BRAC-p and BRAC-v (Wu et al., 2019), behavior-regularized actor-critic with policy (KL-divergence) and value penalties respectively; SAC (Haarnoja et al., 2017) adapted to the offline setting (no environment interaction, training only on the fixed dataset), representing what happens when a standard off-policy algorithm is used without offline-specific modifications; BC (behavioral cloning), a simple supervised imitation baseline that predicts the action taken in the dataset given the state; and for Atari, QR-DQN (Dabney et al., 2018) without offline modifications and REM (Agarwal et al., 2019), a random ensemble mixture method designed for offline RL. The results for BEAR, BRAC, SAC, and BC on D4RL gym tasks are taken directly from the numbers reported by Fu et al. (2020), ensuring standardized comparison. For Atari, QR-DQN and REM results are reproduced using the authors' official codebase.
-
Generation budget / compute accounting. The paper measures computation in gradient steps. For all D4RL domains, training runs for 1,000,000 gradient steps. For Atari experiments in setting (1) (first 20% of DQN data), performance is plotted as a function of training iterations (Figure 1); for setting (2) (1% and 10% data), results are reported after 5× the number of gradient steps used by Agarwal et al. (2019), following their convention. There is no "generation budget" in the sense of number of environment interactions or sampled trajectories, since all training uses only the fixed offline dataset. The fair comparison across methods is ensured by giving all algorithms access to the same dataset with the same number of gradient steps, and (for policy-constraint methods that estimate the behavior policy) the same amount of computation for auxiliary models. CQL requires no additional forward passes beyond what SAC or QR-DQN already use, except for importance-sampling the log-sum-exp term in continuous action spaces (10 extra action samples per state per update), which is a negligible overhead.
-
Cross-validation / statistical protocol. All D4RL continuous control results are averaged over 4 random seeds, with the normalized, smooth average undiscounted return reported as per the D4RL benchmark conventions. The standard deviations are not explicitly reported in the main paper tables, which is a limitation. Hyperparameter selection (Section 4, Appendix F) was performed based on predicted Q-values on the dataset — specifically, the authors monitored the magnitude and stability of Q-values to choose the Lagrange threshold τ and learning rates, without using online policy evaluation. The key hyperparameter selection criteria were: (1) Q-values should remain bounded (not diverging to
>1e6or dropping to<-1e6), (2) for the Lagrange variant, α should stabilize rather than growing unboundedly, and (3) the policy should exhibit a consistent, stable improvement trend. The authors explicitly note: "none of these hyperparameter selections required any notion of online evaluation, since these choices were made based on the predicted Q-values on dataset state-action pairs" (Appendix F). For the Atari experiments, hyperparameters (primarily α) were chosen uniformly per dataset size condition and not tuned per-game, mitigating overfitting concerns. No separate validation split of the offline data was used for hyperparameter tuning, which is a potential concern for reproducibility in new domains — the paper implicitly relies on Q-value diagnostics that may not transfer.
Main Quantitative Results
Performance on D4RL Gym MuJoCo Tasks
Overall pattern: CQL roughly matches or modestly exceeds prior methods on single-policy datasets, but dramatically outperforms them on datasets that combine multiple behavior policies. Table 1 reports the normalized returns for all methods across 18 dataset-task combinations.
Single-policy datasets (random, medium, expert). On these relatively simple datasets where all data comes from a single behavior policy, CQL performs comparably to the best prior methods:
halfcheetah-random: CQL(H) achieves 35.4 vs. SAC's 30.5, BEAR's 25.5, BRAC-p's 23.5, BRAC-v's 28.1. A moderate improvement.hopper-expert: CQL(H) achieves 109.9 vs. BEAR's 110.3, BC's 109.0. Essentially tied with prior best.walker2d-expert: CQL(H) achieves 153.9 vs. BC's 125.7, BEAR's 106.1 — a notable improvement of ~28 points over the prior best method.halfcheetah-medium: CQL(H) achieves 44.4 vs. BRAC-v's 45.5, BRAC-p's 44.0 — essentially tied.walker2d-medium: CQL(H) achieves 79.2 vs. BRAC-v's 81.3 — slightly below.hopper-medium: CQL(H) achieves 58.0 vs. BEAR's 47.6, BRAC-p's 31.2 — a meaningful improvement of ~10 points.
The key takeaway from single-policy datasets: CQL is competitive with the best prior methods but not dramatically better. This is expected because single-policy datasets present a relatively easy offline RL problem — the distribution shift between the data and any policy that stays close to the behavior policy is minimal. Both policy-constraint methods (which explicitly restrict the policy) and CQL (which implicitly reshapes the Q-function) suffice to prevent catastrophic overestimation.
Multi-policy and mixed datasets (medium-expert, random-expert, mixed). On these more complex datasets — where the data comes from a mixture of multiple behavior policies — CQL dramatically outperforms prior methods:
halfcheetah-medium-expert: CQL(H) achieves 62.4 vs. BEAR's 51.7, BRAC-v's 45.3, BC's 35.8. A 10.7-point improvement over the best prior method.walker2d-medium-expert: CQL(H) achieves 98.7 vs. BEAR's 10.8, BRAC-v's 0.9, BC's 11.3. This is a ~88-point improvement — nearly an order of magnitude — over the best prior offline RL method. Even behavioral cloning (11.3) substantially outperforms BEAR (10.8) on this dataset.hopper-medium-expert: CQL(H) achieves 111.0 vs. BC's 111.9 — CQL essentially matches behavioral cloning, while BEAR (4.0), BRAC-p (1.1), and BRAC-v (0.8) completely fail.halfcheetah-random-expert: CQL(H) achieves 92.5 vs. SAC's 53.0, BRAC-p's 30.2, BEAR's 24.6. A 39.5-point improvement.walker2d-random-expert: CQL(H) achieves 91.1 vs. BEAR's 1.9, BRAC-v's 2.7, BC's 0.7. An improvement of ~88 points over the best prior method.hopper-random-expert: CQL(H) achieves 110.5 vs. BRAC-v's 11.1, BC's 10.1, BEAR's 10.1. An improvement of ~99 points.hopper-mixed: CQL(H) achieves 48.6 vs. BEAR's 25.3, BC's 11.8. An improvement of 23.3 points.
What these numbers reveal: The multi-policy datasets are precisely where behavior policy estimation becomes difficult — the data distribution is multi-modal, generated by a mixture of expert, medium, and sometimes random policies. Policy-constraint methods (BEAR, BRAC) rely on accurately estimating the behavior policy density to constrain the learned policy. When the behavior policy is multi-modal, this estimation is challenging, and the constraint becomes either too loose (allowing OOD actions that cause overestimation) or too tight (preventing any improvement over BC). CQL avoids this entirely: it never estimates the behavior policy explicitly. The regularizer $\mathbb{E}_{a \sim \hat{\pi}_\beta}[Q(s,a)]$ uses only the empirical average of Q-values on the actions that actually appear in the batch — no density model is needed. This makes CQL robust to complex, multi-modal data distributions in a way that prior methods are not.
The walker2d-medium-expert result deserves particular attention. BEAR achieves 10.8 — worse than behavioral cloning (11.3) and essentially failing to learn anything useful. CQL achieves 98.7, approaching expert-level performance (~100+ on many D4RL tasks). This is not a marginal improvement — it's the difference between a method that completely fails and one that effectively solves the task. This single result is strong evidence that the core failure mode of prior methods is not just "insufficient conservatism" but specifically the brittleness introduced by requiring explicit behavior policy estimation.
Performance on Adroit Dexterous Manipulation Tasks
Setting: The Adroit tasks (Table 2) involve controlling a 24-DoF robotic hand to manipulate objects (pen, hammer, door, relocate). The datasets consist of limited human demonstrations, making them substantially more challenging than the MuJoCo gym tasks in terms of both high action dimensionality and narrow data coverage. Prior offline RL methods "generally struggle to learn meaningful behaviors on these tasks, and the strongest baseline is BC" (Section 6).
Human demonstration datasets (pen-human, hammer-human, door-human, relocate-human):
pen-human: CQL(ρ) achieves 55.8 vs. CQL(H)'s 37.5, BC's 34.4, SAC's 6.3, BEAR's -1.0. CQL(ρ) is the only method that substantially outperforms behavioral cloning (by ~21 points).hammer-human: CQL(H) achieves 4.4 vs. CQL(ρ)'s 2.1, BC's 1.5. A modest improvement, but notably all methods score near zero on this extremely challenging task.door-human: CQL(H) achieves 9.9 vs. CQL(ρ)'s 9.1, BC's 0.5, SAC's 3.9. Both CQL variants substantially outperform BC.relocate-human: Both CQL variants score near zero (0.20, 0.35), as do all other methods. This task appears to be beyond the capability of current offline RL given the limited human data.
Behavior-cloned demonstration datasets (pen-cloned, hammer-cloned, door-cloned, relocate-cloned):
pen-cloned: CQL(ρ) achieves 40.3 vs. CQL(H)'s 39.2, BC's 56.9. Notably, BC outperforms all offline RL methods here — the cloned policy data provides good coverage of expert-like behavior, and the conservatism of CQL prevents it from fully exploiting this.hammer-cloned: CQL(ρ) achieves 5.7 vs. CQL(H)'s 2.1, BC's 0.8. CQL(ρ) is the only method to make meaningful progress.door-cloned: CQL(ρ) achieves 3.5 vs. CQL(H)'s 0.4, BC's -0.1. Again, CQL(ρ) is the only method showing non-trivial improvement.relocate-cloned: All methods score ~0.
Key insight: CQL(ρ) vs. CQL(H) on high-dimensional action spaces. The paper notes: "CQL(ρ) with ρ = π̂_{k-1} (the previous policy) outperforms CQL(H) on a number of these tasks, due to the higher action dimensionality resulting in higher variance for the CQL(H) importance weights." The CQL(H) variant requires estimating $\log\sum_a \exp(Q(s,a))$ via importance sampling in continuous action spaces. In a 24-dimensional action space, sampling 10 actions from a uniform distribution provides extremely sparse coverage, leading to high-variance estimates of the log-sum-exp. CQL(ρ) concentrates its adversarial distribution μ near the previous policy, providing lower-variance estimates and more stable training. The paper reports that CQL(ρ) "trains more stably, with no sudden fluctuations in policy performance over the course of training, on the higher-dimensional Adroit tasks." This is a practically important finding: the theoretically elegant CQL(H) variant breaks down in high-dimensional action spaces due to the curse of dimensionality in importance sampling, and the more heuristic CQL(ρ) becomes necessary. This is not a failure of the CQL framework — both are valid instances of CQL(R) — but it highlights that practical implementation choices matter substantially and that the regularizer R(μ) should be adapted to the action space dimensionality.
Performance on AntMaze Navigation Tasks
Setting: The AntMaze tasks (Table 2) require controlling an 8-DoF "Ant" quadruped robot to navigate through mazes of increasing size and complexity to reach a goal location. The datasets consist of suboptimal trajectories that do not necessarily reach the goal, requiring the agent to compose segments of different trajectories (stitching) to find a path to the goal. The reward is sparse (1 for reaching the goal, 0 otherwise), and the agent must learn long-horizon planning from fragmented, suboptimal data.
Results:
antmaze-umaze(simplest): CQL variants achieve 73.5–74.0, comparable to BEAR (73.0), BRAC-v (70.0), and BC (65.0). All reasonable methods make progress on the easiest maze.antmaze-umaze-diverse: CQL(H) achieves 84.0 vs. BEAR's 61.0, BRAC-v's 70.0. A meaningful improvement.antmaze-medium-play: CQL(H) achieves 61.2 vs. all other methods scoring 0.0 (BEAR, BRAC-p, BRAC-v, BC, SAC). This is the most dramatic result in the AntMaze suite — CQL is the only method that achieves non-zero return. CQL(ρ) achieves only 4.6, suggesting that the broader adversarial distribution in CQL(H) is important for exploration in the maze.antmaze-medium-diverse: CQL(H) achieves 53.7 vs. BEAR's 8.0, with all other methods at 0.0.antmaze-large-play: CQL(H) achieves 15.8, CQL(ρ) achieves 3.2, all other methods 0.0.antmaze-large-diverse: CQL(H) achieves 14.9, CQL(ρ) achieves 2.3, all other methods 0.0.
What these results demonstrate: The AntMaze tasks are a "stitching" problem — the agent must combine fragments of different suboptimal trajectories into a policy that reaches the goal. Policy-constraint methods fail catastrophically because they are too constrained: by keeping the learned policy close to the behavior policy, they prevent the kind of compositional generalization needed to stitch trajectories. BEAR and BRAC essentially reproduce the behavior policy, which in the medium and large mazes never reaches the goal, so they get zero return. CQL succeeds because its conservatism is value-based rather than policy-based: it doesn't constrain the policy directly, but instead ensures that the Q-function doesn't overestimate the value of untested actions. This allows the policy to deviate from the behavior policy (to compose trajectories) while still being protected from the catastrophic overestimation that would cause standard SAC to fail (SAC also gets 0.0 on all medium and large mazes). The gap between CQL(H) (61.2 on antmaze-medium-play) and CQL(ρ) (4.6) is instructive: the more aggressive adversarial distribution in CQL(H) (which uses a uniform-based μ that can explore broadly) enables better stitching, while CQL(ρ)'s policy-conditioned μ is more conservative and prevents the necessary exploration in Q-space.
Performance on Franka Kitchen Tasks
Setting: The Kitchen tasks (Table 2) require controlling a 9-DoF Franka robot arm to manipulate multiple objects (microwave, kettle, burner, etc.) to reach a desired configuration. The data comes from human teleoperation and provides only sparse 0-1 completion rewards. The challenge is long-horizon sequencing: the robot must complete multiple subtasks in order, using highly multi-modal human demonstration data.
Results:
kitchen-complete: CQL(H) achieves 43.8 vs. BC's 33.8, SAC's 15.0, BEAR/BRAC 0.0.kitchen-partial: CQL(ρ) achieves 50.1, CQL(H) achieves 49.8, vs. BC's 33.8, BEAR's 13.1, BRAC 0.0.kitchen-undirected: CQL(ρ) achieves 52.4, CQL(H) achieves 51.0, vs. BC's 47.5, BEAR's 47.2, SAC's 2.5.
Key finding: CQL is the only method that consistently outperforms behavioral cloning on all three Kitchen tasks. On kitchen-complete, the gap is ~10 points (43.8 vs. 33.8). On kitchen-partial, CQL achieves ~50 vs. BC's 33.8 — a ~16 point improvement. The uniform failure of BEAR and BRAC (0.0 on kitchen-complete) again points to the brittleness of policy constraints with multi-modal human data. The fact that BC achieves 33.8 on kitchen-complete while BEAR gets 0.0 suggests that the policy constraint in BEAR is so tight that it cannot even reproduce the behavior policy's performance — likely because the constraint is enforced on a per-update basis and the compounding effect of small errors during training leads to policy degradation.
Offline RL on Atari Games (Discrete Control, Image Observations)
Setting (1): First 20% of DQN replay data (Figure 1). The dataset consists of all transitions observed by an online DQN agent during its first 20% of environment interactions — a relatively abundant but early-stage dataset. CQL is compared to QR-DQN (no offline modifications) and REM (random ensemble mixture, designed for offline RL).
Results:
- Pong: CQL learns stably, achieving scores comparable to or slightly better than QR-DQN and REM throughout training, ultimately reaching ~15–20 (near-optimal for Pong).
- Breakout: CQL achieves ~80–100, comparable to QR-DQN and REM.
- Qbert: CQL significantly outperforms, reaching ~4000–5000 vs. QR-DQN and REM at ~1000–2000.
- Seaquest: All methods show similar performance (~1000–1750), though CQL appears slightly less stable late in training. The paper notes: "its performance does not degrade as steeply as QR-DQN on Seaquest" — however, the plot shows CQL scores declining somewhat while REM remains more stable. This is one of the few cases where CQL does not clearly outperform the baselines.
Setting (2): 1% and 10% subsampled data (Table 3). These are extremely data-limited regimes, testing whether conservative Q-learning can prevent overfitting and overestimation when data is scarce.
1% data (top of Table 3):
- Pong: CQL achieves 19.3 vs. QR-DQN's -13.8, REM's -6.9. Both baselines get negative scores (worse than random), while CQL achieves positive performance.
- Breakout: CQL achieves 61.1 vs. QR-DQN's 7.9, REM's 11.0. A ~50-point improvement.
- Q*bert: CQL achieves 14,012.0 vs. QR-DQN's 383.6, REM's 343.4. This is a ~36× improvement over the best baseline.
- Seaquest: CQL achieves 779.4 vs. QR-DQN's 672.9, REM's 499.8. A modest improvement.
- Asterix: CQL achieves 592.4 vs. REM's 386.5, QR-DQN's 166.3. A ~200-point improvement.
10% data (bottom of Table 3):
- Pong: CQL achieves 18.5 vs. QR-DQN's 15.1, REM's 8.9.
- Breakout: CQL achieves 269.3 vs. QR-DQN's 151.2, REM's 86.7. A ~118-point improvement.
- Q*bert: CQL achieves 13,855.6 vs. REM's 8,624.3, QR-DQN's 7,091.3. A more modest but still substantial improvement.
- Seaquest: CQL achieves 3,674.1 vs. REM's 3,936.6, QR-DQN's 2,984.8 — slightly below REM.
- Asterix: CQL achieves 156.3 vs. QR-DQN's 189.2, REM's 75.1 — slightly below QR-DQN.
Key observations from Atari: CQL's advantage is most pronounced in the extremely data-scarce regime (1%), where overestimation and overfitting are most severe. At 10% data, CQL still generally outperforms but the gaps are smaller. At 20% data (Figure 1), CQL is competitive but not consistently better. This pattern makes sense: as data increases, the overestimation problem becomes less acute because the dataset covers more of the state-action space. CQL's conservative regularizer provides the most value precisely when standard methods would be most over-optimistic — when data is scarce.
The paper uses fixed α values per data condition (α=4.0 for 1%, α=1.0 for 10%, α=0.5 for 20%), which naturally provides stronger conservatism for smaller datasets. This is consistent with the theoretical results (Theorems 3.1, 3.2): the required α to guarantee a lower bound scales inversely with $\sqrt{|\mathcal{D}(s,a)|}$, so smaller datasets need larger α. The paper's hyperparameter choices respect this relationship, whether intentionally or through tuning.
Verification of Conservative Value Estimates
Table 4 reports the difference $\mathbb{E}_{s \sim \mathcal{D}}[\hat{V}^k(s)] - V^{\pi_k}_{\text{true}}$ — the predicted policy value minus the actual Monte Carlo return — for CQL(H), CQL with Equation 1 (pointwise lower bound), ensembles of various sizes, and BEAR on three hopper datasets.
Results:
- CQL(H): Differences are -43.20 (
hopper-medium-expert), -10.93 (hopper-mixed), -7.48 (hopper-medium). All negative, confirming lower bounds. - CQL (Eqn. 1 — pointwise lower bound): Differences are -151.36, -22.87, -156.70. Much more negative, confirming pointwise lower bounds are unnecessarily conservative.
- Ensemble (2 Q-functions): Differences are 3.71e6, 15.00e6, 26.03e12 — astronomically positive values indicating massive overestimation.
- Ensemble (4 Q-functions): 2.93e6, 59.93e3, 437.57e6 — still massive overestimation.
- Ensemble (10 Q-functions): 0.32e6, 8.92e3, 1.12e12 — some reduction but still extreme overestimation.
- Ensemble (20 Q-functions): 24.05e3, 2.47e3, 885e3 — improved but still positive (overestimated).
- BEAR: 65.93, 1399.46, 4.32 — positive for two tasks, indicating overestimation persists despite policy constraints.
Critical finding: Even ensembles of 20 Q-functions — a computationally expensive uncertainty estimation technique — still overestimate values by orders of magnitude (e.g., 885,000 on hopper-medium). This empirically validates the paper's claim that "uncertainty-based methods are not sufficient to prevent against OOD actions in and of themselves" (Appendix E). The estimates are not just slightly over-optimistic — they are catastrophically wrong. CQL's regularizer, in contrast, reliably produces conservative estimates with magnitudes that are reasonable (single to double digits), and CQL(H) achieves tighter bounds than the pointwise formulation.
Ablation Studies and Robustness Checks
CQL(H) vs. CQL(ρ) on Gym MuJoCo tasks (Table 5): On three MuJoCo tasks where action dimensionality is moderate, CQL(H) outperforms CQL(ρ): halfcheetah-medium-expert (7234.5 vs. 3995.6), walker2d-mixed (1227.2 vs. 812.7), hopper-medium (1866.1 vs. 1166.1). This confirms that when the importance sampling estimate for log-sum-exp is sufficiently accurate (low-to-moderate action dimensionality), the broader adversarial distribution of CQL(H) provides better conservatism. However, on the high-dimensional Adroit tasks (Table 2), CQL(ρ) outperforms CQL(H) on 5 of 8 tasks, demonstrating that the variance of importance sampling in CQL(H) becomes prohibitive.
CQL(H) with vs. without the dataset maximization term (Table 6): This ablation tests whether the $-\mathbb{E}_{a \sim \hat{\pi}_\beta}[Q(s,a)]$ term in Equation 2 is necessary, or whether the simpler Equation 1 (only minimizing Q-values) suffices. On hopper-medium-expert, the difference is small (3628.4 vs. 3610.3), but on hopper-mixed (1563.2 vs. 864.6) and hopper-medium (1866.1 vs. 1028.4), omitting the maximization term substantially degrades performance. This empirically validates Theorem 3.2's claim that Equation 1 (the pointwise lower bound) is unnecessarily conservative — when the dataset is generated from a single policy (hopper-medium), the extra conservatism of Equation 1 prevents the policy from exploiting in-distribution actions, cutting performance nearly in half.
CQL(H) with Lagrange (automatic α) vs. fixed α (Table 7): On MuJoCo gym tasks, the Lagrange version (with τ=10.0) and fixed α=5.0 perform comparably (hopper-medium-expert: 3628.4 vs. 3589.4; walker2d-random-expert: 4183.0 vs. 3934.5). However, on the harder AntMaze tasks, the Lagrange version dramatically outperforms: antmaze-medium-diverse (0.53 vs. 0.21), antmaze-large-play (0.15 vs. 0.02), antmaze-large-diverse (0.14 vs. 0.05). The automatic tuning is particularly valuable when the appropriate conservatism level varies across tasks — the fixed α=5.0 that works well on MuJoCo gym is clearly insufficient for AntMaze (where returns are near zero, possibly because α=5.0 is too high and prevents any learning, or too low and doesn't prevent overestimation — the paper doesn't clarify which). The Lagrange method adapts α during training to maintain the specifed τ constraint.
Effect of policy learning rate (Appendix F): The paper evaluated policy learning rates of 3e-5, 1e-4, and 3e-4. The rate 3e-5 was chosen as default because it "almost uniformly attain[s] good performance." The rate 1e-4 performed "better on some experiments (such as hopper-medium-v0 and antmaze-medium-play-v0), but it performed badly with the real-human demonstration datasets, such as the Adroit tasks." This aligns with Theorem 3.3's requirement that the policy changes slowly — on the more challenging Adroit tasks with narrow human data, faster policy updates likely violate the $D_{TV}(\hat{\pi}_{k+1}, \pi_{\hat{Q}^k}) \leq \varepsilon$ condition, causing the lower-bound guarantee to break.
Lagrange threshold τ (Appendix F): Three values were tested: τ=2.0, 5.0, 10.0. τ=2.0 "led to a huge increase in the value of α (sometimes up to the order of millions), and as a result, highly underestimated Q-functions on all domains (sometimes up to the order of -1e6)." τ=10.0 was "unable to prevent overestimation in Q-values in a number of cases and Q-values diverged to highly positive values (>1e+6)" on Kitchen and Adroit domains. τ=5.0 provided a good balance for the human-demonstration datasets, while τ=10.0 sufficed for the more abundant MuJoCo gym datasets. This highlights a practical sensitivity: the choice of τ is task-dependent, and the diagnostics (monitoring Q-value magnitudes) are essential for selection, but may not generalize to new domains without similar monitoring.
Gap-expanding property verification (Appendix B, Figure 2): The paper empirically validates Theorem 3.4 by tracking $\hat{\Delta}_k = \mathbb{E}_{s,a \sim D}[\max_{a'}\hat{Q}^k(s,a') - \hat{Q}^k(s,a)]$ during training. On hopper-expert-v0, CQL maintains negative $\hat{\Delta}_k$ (~-5 to -10), while BEAR shows positive $\hat{\Delta}_k$ that grows from ~5 to 30 during training. The positive values for BEAR mean the Q-function rates some OOD actions as better than the expert actions in the dataset — a clear failure mode. The corresponding policy performance (Figure 2, right panels) shows BEAR's return eventually declining (the "unlearning" effect), while CQL's remains stable. On -1 to -3), while BEAR's is positive (~1–4). This is the most direct empirical evidence that CQL's gap-expanding property (Theorem 3.4) operates in practice and distinguishes it from policy-constraint methods.hopper-medium-v0, the pattern is similar but less extreme: CQL's $\hat{\Delta}_k$ remains slightly negative (
Number of gradient steps (Appendix F): CQL uses 1M gradient steps for all D4RL domains. The paper notes that this is a heuristic choice — "due to a lack of a proper validation error metric for offline Q-learning methods, deciding the number of gradient steps dynamically has been an open problem in offline RL." Different prior methods use different numbers of steps, making exact comparison challenging. The Atari experiments follow the convention from Agarwal et al. (2019) of training for 5× the steps used in their paper.
Choice of Bellman backup (Appendix F): The paper mentions an alternative to the actor-critic formulation: using an "approximate max-backup" where, instead of taking the expectation under the policy at the next state, 10 actions are sampled from the current policy and the maximum of their Q-values is used as the target: $r(s,a) + \max_{a_1,\dots,a_{10} \sim \pi(a'|s')} Q(s',a')$. This was found to perform better on Franka Kitchen and AntMaze tasks, suggesting that the standard expected backup can be too conservative (taking the expectation over potentially poor actions) in goal-oriented sparse-reward tasks where identifying the single best next action is more important than averaging.
Critical Assessment
The central claim of the paper is that CQL — a simple Q-value regularizer — enables stable offline RL across a wide range of domains, substantially outperforming prior methods especially on complex, multi-modal datasets, while providing theoretical guarantees of conservative value estimation and safe policy improvement. The experiments provide strong support for this claim, but several qualifications and gaps merit attention.
Claim: CQL substantially outperforms prior methods on complex datasets (2–5× higher return). The evidence for this is compelling and consistent. On walker2d-medium-expert, CQL achieves 98.7 vs. BEAR's 10.8 — a 9× improvement. On hopper-random-expert, CQL achieves 110.5 vs. BRAC-v's 11.1 — a 10× improvement. On antmaze-medium-play, CQL achieves 61.2 vs. 0.0 for all baselines — an effectively infinite relative improvement. On Atari Q*bert with 1% data, CQL achieves 14,012 vs. REM's 343 — a 40× improvement. These are not marginal gains; they represent qualitative differences where baselines completely fail and CQL succeeds. The claim of 2–5× improvement is, if anything, conservative relative to the reported numbers for the hardest datasets.
However, the "2–5×" framing masks important heterogeneity. On single-policy datasets (halfcheetah-medium, walker2d-medium, hopper-expert), CQL's improvement is negligible (0–10%). On the simplest tasks, it sometimes slightly underperforms the best prior method (walker2d-medium: CQL 79.2 vs. BRAC-v 81.3). The dramatic gains are concentrated in two specific settings: (1) multi-policy datasets where behavior policy estimation fails for policy-constraint methods, and (2) extremely data-scarce regimes (Atari 1%). This is not a weakness — it's precisely where CQL's design advantages (no behavior policy estimation, value-based rather than policy-based conservatism) matter most — but the "2–5×" headline number should be understood as an average that obscures this task-dependence.
Claim: CQL learns conservative Q-functions that lower-bound the true value. Table 4 provides direct evidence: CQL(H) underestimates by -7 to -43 on the tested datasets, while ensembles of 2–20 Q-functions overestimate by thousands to trillions. This is convincing but limited in scope — it is shown for only 3 datasets (all hopper variants) and only for the final policy. The paper does not show how the conservatism evolves during training, whether the degree of underestimation is well-calibrated to the actual performance gap, or whether the lower-bound property holds across the full diversity of D4RL tasks. The Atari experiments have no corresponding value-estimation verification. Given that the lower-bound property is the paper's primary theoretical contribution, more extensive empirical validation of this property would strengthen the argument.
Claim: CQL enables safe policy improvement over the behavior policy. Theorem 3.6 provides a theoretical guarantee, but the paper does not empirically verify it in a direct way. The safe improvement bound depends on the specific form of $D_{CQL}(\pi, \hat{\pi}_\beta)$ and the dataset size — the paper never reports whether CQL policies actually satisfy the bound with realistic values of the concentration constants, or whether the bound is tight enough to be practically meaningful (as opposed to being vacuous for typical dataset sizes). This is a gap between the theory and the empirical validation: the experiments show that CQL works, but they don't show that it works for the reasons the theory predicts (the specific χ²-divergence penalty, the role of sampling error, etc.).
Missing baselines and comparisons. Several comparisons that would strengthen the paper are absent:
- No comparison to other Q-function regularization methods. The paper positions CQL against policy-constraint methods (BEAR, BRAC) and uncertainty-based methods (ensembles, REM), but doesn't compare against simpler Q-function penalties like the one in Kumar et al. (2019) or the value penalty in BRAC-v (which is included in Table 1, but not analyzed as a Q-function regularization approach distinct from policy constraints).
- No comparison to recently contemporary methods that also avoid behavior policy estimation. Algorithms like AWAC (Nair et al., 2020) and CRR (Wang et al., 2020) use advantage-weighted regression, which implicitly constrains the policy without explicit density estimation — they would be natural comparisons for evaluating whether CQL's Q-function regularization is superior or just one of several viable approaches to the same problem.
- No comparison to CQL variants that use the actual behavior policy (oracle). It would be informative to see CQL's performance when using the true πβ(a|s) in the regularizer (for the
$\mathbb{E}_{\hat{\pi}_\beta}[Q]$term) rather than the empirical batch average — this would isolate whether the batch-average approximation introduces meaningful error. - No comparison on the full D4RL benchmark. The paper omits several D4RL tasks (e.g., the maze2d domains, the flow domains, some of the Franka Kitchen variants). While covering all would be excessive, the selection of tasks appears somewhat favorable to CQL (focusing on the domains where policy-constraint methods are known to struggle, per Fu et al., 2020).
Dataset and evaluation limitations:
- Single benchmark suite for continuous control (D4RL). While D4RL is the standard benchmark and covers substantial diversity, all tasks share MuJoCo physics and a relatively similar state/action structure. Performance on D4RL does not guarantee performance on real-world offline RL problems with different characteristics (e.g., partial observability, stochastic dynamics not captured in simulation, non-Markovian dependencies from the data collection process). The paper's claim that CQL is "a promising choice for a wide range of real-world offline RL problems" (Section 7) is aspirational rather than demonstrated.
- No standard deviations reported in main tables. The D4RL results (Tables 1, 2) report averages over 4 seeds without error bars or standard deviations. For the most dramatic results (e.g.,
walker2d-medium-expert: CQL 98.7 vs. BEAR 10.8), even large variance would not change the qualitative conclusion, but for closer comparisons (e.g.,hopper-medium-expert: CQL 111.0 vs. BC 111.9), standard deviations are essential to assess whether differences are statistically meaningful. In Appendix G, Table 5 reports standard deviations for select tasks, but not comprehensively. - Atari results evaluated at a single training point. The Atari results in Table 3 report performance after a fixed number of gradient steps, following Agarwal et al. (2019)'s convention. Figure 1 shows training curves for the 20% data condition, but no such curves are provided for the 1% and 10% conditions. It's possible that the baselines catch up or CQL degrades with additional training, but this isn't shown.
- Small number of Atari games. Only 5 games are evaluated, and the selection (Pong, Breakout, Qbert, Seaquest, Asterix) is inherited from Agarwal et al. (2019). The results on Qbert are dramatic (~36× improvement at 1%), but on Seaquest, CQL does not consistently outperform the baselines. Without results on more games, it's difficult to assess whether CQL's strong performance generalizes across the ALE benchmark or is concentrated in a subset of games.
Hyperparameter sensitivity and reproducibility concerns:
- Lagrange threshold τ varies by domain. τ=10.0 for MuJoCo gym, τ=5.0 for Adroit and Kitchen. The paper provides heuristics for choosing τ (monitor Q-values, ensure they don't diverge), but these heuristics require running the algorithm and observing the Q-values — they are diagnostics, not a priori selection criteria. In a new domain, a practitioner would need to run multiple τ values and select based on Q-value stability, which amounts to a form of hyperparameter search using a proxy metric that may not correlate perfectly with policy performance.
- α for Atari varies by data quantity. α=0.5 for 20% data, α=1.0 for 10%, α=4.0 for 1%. The paper doesn't report how these were chosen (tuning? grid search? following the theoretical scaling
$\alpha \propto 1/\sqrt{|\mathcal{D}|}$?) or how sensitive performance is to α. If α must be carefully tuned per dataset size, this limits CQL's applicability in settings where dataset size varies across state-action pairs (as it always does in practice). - Policy learning rate of 3e-5 is crucial but task-dependent. The paper notes that 1e-4 works better on some tasks but worse on Adroit. This suggests that the performance is sensitive to the relative learning rates of Q-function and policy, and the theoretically-motivated "slow policy updates" condition (Theorem 3.3) requires empirical tuning to satisfy.
What the experiments do not test:
- Generalization to stochastic environments. All D4RL tasks use deterministic MuJoCo dynamics — the stochasticity comes only from the behavior policy and initial state distribution. The theoretical analysis accounts for stochastic transitions via the concentration bounds, but the empirical evaluation cannot verify whether CQL's conservatism is well-calibrated for environments where actions have genuinely uncertain outcomes. Over-conservatism in stochastic environments could prevent the policy from taking actions that have high variance but high mean return.
- Scaling with dataset size. The Atari experiments test 1%, 10%, and 20% subsampling, but the D4RL experiments use the standard benchmark datasets without subsampling. It would be informative to see CQL's performance on subsampled D4RL datasets (e.g., 10% of
halfcheetah-medium-expert) to test whether the relative advantage over baselines grows as data decreases, as the Atari results would predict. - Performance with different Q-network architectures. CQL uses the standard SAC architecture (256-unit MLPs). The sensitivity to network size, depth, and activation functions is unexplored. Given that function approximation error is the mechanism that creates overestimation in the first place, the interaction between CQL's regularizer and network capacity is important: a larger network might overfit more (requiring stronger conservatism) or might represent the Q-function more accurately (reducing the need for conservatism).
- Ablation of the number of action samples for log-sum-exp. The paper uses N=10 for continuous control. The sensitivity to N is untested — does CQL(H) performance degrade at N=5? Does it improve at N=50? In the 24-DoF Adroit setting, N=10 from a uniform distribution provides extremely sparse coverage, and the choice to use CQL(ρ) instead of CQL(H) is motivated by this variance. A more systematic study of how the importance sampling accuracy affects performance would inform practitioners about when to prefer CQL(ρ) over CQL(H).
Overall assessment: The experiments convincingly demonstrate that CQL outperforms prior methods on the standard benchmarks, particularly in the hard cases (multi-policy datasets, data-scarce regimes) that matter most for real-world applicability. The breadth of evaluation — continuous control (gym, Adroit, AntMaze, Kitchen) and discrete control (Atari), with 19 continuous-control task-dataset combinations and 5 Atari games — is substantial for the field at the time. The empirical verification of conservative value estimates (Table 4) and the gap-expanding property (Figure 2) provides direct evidence that the proposed mechanism operates as theorized, which is more than many RL papers achieve.
However, the experiments are better at demonstrating that CQL works than at characterizing precisely when and why. The paper would be strengthened by: (1) systematic sensitivity analysis of the key hyperparameters (α, τ, policy learning rate) across domains, with recommendations that don't require per-task tuning; (2) explicit verification of the safe improvement bound on at least one domain to bridge the theory-practice gap; (3) comparison to more recent methods that also avoid behavior policy estimation, to clarify whether the Q-function regularizer approach is superior or merely one of several effective paradigms; (4) results on subsampled D4RL datasets to test the data-scarcity findings beyond Atari; and (5) standard deviation reporting for all main results. The dramatic performance gaps on the hardest tasks are robust enough that these limitations don't undermine the central claim, but they limit the paper's ability to provide prescriptive guidance for practitioners deploying CQL in new domains.
6. Limitations and Trade-offs
6.1 Hyperparameter Sensitivity Requires Task-Specific Tuning with No A Priori Selection Protocol
The assumption or constraint. CQL introduces several critical hyperparameters—the tradeoff coefficient α (or the Lagrange threshold τ), the policy learning rate relative to the Q-function learning rate, and the specific variant (CQL(H) vs. CQL(ρ))—whose optimal values vary substantially across domains. The paper acknowledges this implicitly through the choices reported in Appendix F:
"We evaluated our method on varying number of gradient steps... deciding the number of gradient steps dynamically has been an open problem in offline RL."
The Lagrange threshold τ requires markedly different values across domains: τ=10.0 for MuJoCo gym tasks, τ=5.0 for Adroit and Kitchen tasks. The fixed α for Atari varies with data quantity: α=0.5 for 20% data, α=1.0 for 10%, α=4.0 for 1%. The policy learning rate of 3e-5 is justified by Theorem 3.3, but the paper notes that 1e-4 "performed better on some experiments" while failing on Adroit. The selection of CQL(H) versus CQL(ρ) depends on action-space dimensionality, with CQL(ρ) required for the 24-DoF Adroit hand but CQL(H) preferred for lower-dimensional MuJoCo tasks.
The consequence. A practitioner deploying CQL in a new domain has no principled way to select these hyperparameters without running multiple training runs. The paper's recommended diagnostics—monitoring Q-values for stability and ensuring they don't diverge (Appendix F)—require training CQL first and observing the resulting Q-value magnitudes. If α is too high, Q-values collapse to highly negative values (e.g., -1e6 with τ=2.0); if α is too low, Q-values diverge to highly positive values (e.g., >1e6 with τ=10.0 on Adroit). Both failure modes are catastrophic for policy performance, and the window of acceptable values appears narrow. The theoretical relationship $\alpha \propto 1/\sqrt{|\mathcal{D}(s,a)|}$ provides directional guidance (more data needs less conservatism), but does not give a numerical value without knowing the concentration constants $C_{r,\delta}$ and $C_{T,\delta}$, which are task-dependent and impractical to estimate.
This sensitivity is not merely a tuning inconvenience—it undermines the paper's claim that CQL can be deployed without online evaluation. If a practitioner must run multiple α values and select based on Q-value behavior (which itself may not perfectly correlate with policy performance), this constitutes a form of hyperparameter search that requires proxy metrics whose reliability is unverified outside the tested domains. The paper's statement that "none of these hyperparameter selections required any notion of online evaluation" (Appendix F) is true only in the narrow sense that the final evaluation doesn't use environment interaction, but it obscures the fact that hyperparameter selection itself requires running the algorithm and interpreting diagnostic signals that may not generalize.
What evidence exists in the paper. Appendix F, Table 7, and the Lagrange threshold discussion provide direct evidence of sensitivity. Table 7 shows that on AntMaze, the Lagrange version with τ=10.0 achieves normalized returns of 0.53 (medium-diverse), 0.15 (large-play), and 0.14 (large-diverse), while fixed α=5.0 achieves only 0.21, 0.02, and 0.05—differences of 2.5× to 7.5×. On MuJoCo gym tasks, fixed α=5.0 and Lagrange with τ=10.0 perform comparably (hopper-medium-expert: 3589.4 vs. 3628.4), but this similarity does not hold on harder tasks. The Atari experiments in Table 3 show that α must increase as data decreases (from 0.5 to 4.0), but the paper does not report how performance varies if α is misspecified for a given data regime.
Mitigation status. The Lagrange dual gradient descent method partially automates α selection by replacing the manual α with a target τ that the algorithm adjusts α to maintain. However, this merely shifts the hyperparameter from α to τ—the practitioner must still choose τ appropriately for each domain, and the paper provides only heuristic guidance (τ=10.0 for "abundant" data, τ=5.0 for "scarce" data). No method for selecting τ a priori is proposed. The paper suggests future work on "simple and effective early stopping methods, analogous to validation error in supervised learning" (Section 7), but does not address the broader hyperparameter selection problem. The sensitivity of CQL(ρ) vs. CQL(H) choice to action dimensionality is documented but not resolved—a practitioner with a moderately high-dimensional action space (e.g., 10-DoF) has no clear guidance on which variant to use.
6.2 CQL(H) Breaks Down in High-Dimensional Action Spaces Due to Importance Sampling Variance
The assumption or constraint. CQL(H)—the entropy-regularized variant that is the primary algorithm evaluated on most benchmarks—requires computing $\log\sum_a \exp(Q(s,a))$, the log-sum-exp of Q-values over the continuous action space. Since exact computation is intractable in continuous spaces, the paper uses importance sampling with N=10 actions drawn from both a uniform distribution and the current policy (Appendix F). This estimator has variance that grows with action-space dimensionality, because 10 samples from a uniform distribution over a 24-dimensional space provide vanishingly sparse coverage.
The paper explicitly acknowledges this limitation in Section 6 when discussing the Adroit results:
"CQL(ρ) with ρ = π̂_{k-1} (the previous policy) outperforms CQL(H) on a number of these tasks, due to the higher action dimensionality resulting in higher variance for the CQL(H) importance weights."
The consequence. In high-dimensional action spaces such as the 24-DoF Adroit hand, CQL(H) provides a poor estimate of the log-sum-exp penalty. The paper reports that CQL(ρ) "trains more stably, with no sudden fluctuations in policy performance over the course of training" on Adroit tasks (Section 6), implying that CQL(H) suffers from training instability. This is not merely a performance gap—it's a qualitative failure mode where the theoretically-motivated CQL(H) variant becomes unreliable.
More subtly, even in moderate-dimensional action spaces like the 8-DoF Ant or 6-8 DoF MuJoCo tasks, the importance sampling estimator may introduce bias that interacts with the Q-function learning dynamics in complex ways. The paper uses N=10 for all continuous control experiments and does not ablate this choice. At N=10 in 24 dimensions, the probability that any of the uniform samples falls in a region of high Q-value is extremely small, meaning the log-sum-exp estimate is dominated by the on-policy samples. This effectively turns CQL(H) into a noisy approximation of CQL(ρ), but without the stability benefits of the explicit KL-regularized formulation.
What evidence exists in the paper. Table 2 shows that on 8 Adroit tasks, CQL(ρ) outperforms CQL(H) on 5: pen-human (55.8 vs. 37.5), hammer-cloned (5.7 vs. 2.1), door-cloned (3.5 vs. 0.4), with pen-cloned (40.3 vs. 39.2) and relocate-cloned (tie near zero) being comparable. Only on hammer-human (2.1 vs. 4.4) and door-human (9.1 vs. 9.9) does CQL(H) edging out CQL(ρ). On the Kitchen tasks, which have a 9-DoF action space, the two variants perform comparably (CQL(H): 43.8, 49.8, 51.0; CQL(ρ): 31.3, 50.1, 52.4 on the three tasks). The paper's statement that CQL(ρ) "trains more stably" is based on qualitative observation of training curves, not a quantitative stability metric, and no training curves for Adroit are shown.
Mitigation status. The paper provides CQL(ρ) as an alternative for high-dimensional action spaces, effectively acknowledging that CQL(H) is not universally applicable. However, the transition between the two is not principled—at what action dimensionality does CQL(H) break down? The paper does not provide guidance. Moreover, CQL(ρ) introduces its own hyperparameter (the choice of prior ρ), and the paper explores only ρ = π̂_{k-1} (the previous policy). Other choices of ρ—a uniform distribution, a mixture of the previous policy and uniform, a slowly-updated target policy—might provide better tradeoffs between coverage and variance, but are unexplored. The importance sampling approach itself could potentially be improved by techniques from the Monte Carlo literature (e.g., adaptive importance sampling, multiple importance sampling with additional proposal distributions, or quasi-Monte Carlo methods), but these are not discussed.
6.3 No Early Stopping or Model Selection Criterion Exists for Offline Q-Learning, Making Deployment Unreliable
The assumption or constraint. CQL, like all offline Q-learning methods at the time, trains for a fixed number of gradient steps (1M for D4RL, 5× the baseline steps for Atari) without any validation procedure to determine when to stop. The paper explicitly identifies this as an unresolved problem:
"Due to a lack of a proper validation error metric for offline Q-learning methods, deciding the number of gradient steps dynamically has been an open problem in offline RL." (Appendix F)
All standard supervised learning pipelines use a held-out validation set to detect overfitting and perform early stopping and model selection. Offline RL has no analogous procedure because the policy's performance cannot be evaluated without environment interaction, and Q-value metrics (training loss, Q-value magnitudes) do not reliably correlate with policy quality—as the paper demonstrates in Table 4, Q-values can diverge to trillions (for ensemble methods) or drop to highly negative values (for overly conservative α) without a clear relationship to actual policy return.
The consequence. In a practical deployment, the lack of early stopping means:
- The practitioner cannot detect when the policy has begun to overfit the dataset and should stop training. The "unlearning" effect documented in Appendix B (Figure 2a) for policy-constraint methods—where performance rises then deteriorates—could also affect CQL with suboptimal hyperparameters, and there is no way to detect it without online evaluation.
- Model selection (choosing between multiple training runs with different hyperparameters or random seeds) cannot be done using offline data alone. The paper selects the best α, τ, and learning rate based on Q-value diagnostics that may not transfer to new domains.
- The number of gradient steps (1M for D4RL) is a heuristic tuned to the benchmark and may not be appropriate for datasets of different sizes, tasks with different horizon lengths, or different network architectures.
This is not a limitation specific to CQL—it affects all offline RL methods—but it is particularly consequential for CQL because CQL's primary selling point is applicability to real-world settings where online evaluation is impossible. If the method cannot be validated without the very environment interaction it's designed to avoid, its practical deployability is fundamentally limited.
What evidence exists in the paper. The paper does not systematically study the effect of training duration on CQL performance. The D4RL results report final performance after 1M steps without showing learning curves (except for the gap-expansion analysis in Figure 2, which shows BEAR performance but only for a single task). The Atari results in Figure 1 show performance over training iterations for the 20% data condition, revealing that CQL's performance on Seaquest declines somewhat late in training (though "not as steeply as QR-DQN"). No such curves are shown for the 1% and 10% conditions in Table 3, where the number of gradient steps (5× the baseline) is taken from Agarwal et al. (2019) without validation. The paper cannot answer whether CQL performance on antmaze-medium-play (61.2 at 1M steps) would be higher or lower at 500K or 2M steps.
Mitigation status. The paper does not attempt to address this limitation beyond flagging it as future work: "another important challenge for future work is to devise simple and effective early stopping methods, analogous to validation error in supervised learning." No concrete direction is proposed, and the experiments do not investigate whether any offline proxy (e.g., the value of the CQL penalty term, the gap between Q-values under π and π̂_β, or the policy's entropy) correlates with true performance and could serve as a validation signal.
6.4 Theoretical Lower-Bound Guarantee Requires α to Be Impossibly Large in Practice, Creating a Theory-Practice Gap
The assumption or constraint. Theorems 3.1, 3.2, and D.1 provide conditions on α that guarantee the learned Q-function lower-bounds the true Q-function (or the policy value). These conditions take the form:
for Theorem 3.1 (pointwise lower bound), and a similar form for Theorem 3.2 with the χ²-type divergence term. The problem is that $C_{r,T,\delta}$ is a concentration constant that depends on the variance of rewards and transition dynamics, and $\min_{s,a} |\mathcal{D}(s,a)|$ is the minimum number of samples for any state-action pair—which is zero or near-zero for most states in any practical offline dataset (since continuous state spaces ensure most states have at most one observation). The paper acknowledges this implicitly in Section 3.1: the entries of $1/\sqrt{|\mathcal{D}|}$ for state-action pairs with zero counts are "a very large but finite value $\delta \geq 2R_{\max}/(1-\gamma)$."
The consequence. The theoretically-required α to guarantee a lower bound is enormous—potentially millions or higher—because it must compensate for the worst-case sampling error at the most poorly-sampled state-action pair. However, the paper's empirical results use α values that are orders of magnitude smaller: α=5.0 for MuJoCo tasks (Appendix F), α=0.5–4.0 for Atari (Appendix F). When the paper tested τ=2.0 (which would drive α to large values), it resulted in "highly underestimated Q-functions on all domains (sometimes up to the order of -1e6)" and presumably poor policy performance (though the paper doesn't report the policy returns for this condition).
This means the practical algorithm operates in a regime where the theoretical lower-bound guarantee does NOT hold. CQL works empirically not because α satisfies the conditions of Theorems 3.1–3.2, but because moderate values of α provide enough conservatism to prevent the catastrophic overestimation that plagues standard Q-learning, while not being so large as to cripple learning. The theory provides a qualitative understanding (more data means less conservatism is needed; larger α means more underestimation) but does not give a usable formula for setting α. The safe policy improvement guarantee in Theorem 3.6 similarly depends on constants ($C_{r,\delta}$, $C_{T,\delta}$) that are not estimated and a bound that may be vacuous for realistic dataset sizes.
This is a significant theory-practice gap: the paper's primary theoretical contribution (provable lower bounds and safe improvement) does not actually apply to the algorithm configuration that achieves the reported empirical results. The theory describes a limiting case (infinite data or extremely large α) that the practical algorithm cannot achieve without becoming uselessly conservative.
What evidence exists in the paper. The paper does not directly measure or discuss this gap. The Lagrange threshold experiment in Appendix F provides indirect evidence: τ=2.0 caused α to grow to "the order of millions" with Q-values dropping to -1e6, confirming that the theoretically-sufficient α leads to catastrophic underestimation. Yet the paper's theoretical analysis never calculates what α would actually be required for a specific dataset or checks whether the lower-bound property holds empirically at the α values used (Table 4 shows the lower bound holds for CQL(H) at the operational α, but doesn't report whether it would hold with any α or whether the bound is tight). The gap between theoretical conditions and practical hyperparameters is never explicitly acknowledged.
Mitigation status. The paper does not address this gap. The theoretical analysis (Theorems 3.1–3.2 and D.1–D.2) is presented as supporting evidence for the algorithm, without discussion of whether the stated conditions are satisfied in the experiments. The extension to neural networks via NTK (Theorem D.2) further widens rather than closes the gap, as NTK assumptions require infinite-width networks and infinitesimal learning rates that are not met in practice. The paper does not suggest how to estimate the concentration constants, how to handle the continuous-state case where $|\mathcal{D}(s,a)| \approx 0$ for most (s,a), or how to derive a more practical condition on α. This is a missed opportunity: a theory that predicts when the algorithm works (and when it fails) would be far more valuable than a theory that requires conditions that cannot be met.
6.5 CQL Does Nothing for the Hardest Problems Where the Base Policy Lacks Competence—It Cannot Create Capability That Isn't Already in the Data
The assumption or constraint. CQL is fundamentally a method for preventing overestimation on out-of-distribution actions—it ensures that the policy doesn't erroneously prefer actions that look good under function approximation error but are actually poor. However, it does not and cannot create information where none exists: if the dataset contains no trajectories that achieve high return (because the behavior policy is uniformly poor, or because the task requires capabilities absent from the data), CQL cannot synthesize effective behavior from nothing.
This limitation is structural rather than accidental. The regularization term $\mathbb{E}_{a \sim \hat{\pi}_\beta}[Q(s,a)]$ pulls Q-values up for in-distribution actions and the gap-expanding property (Theorem 3.4) makes OOD actions less attractive. But if all in-distribution actions are poor (as in the relocate-human task, where even human demonstrators struggle), then the best the algorithm can do is stay close to the behavior policy—which means achieving behavior-policy-level performance or slightly better. CQL cannot discover entirely novel strategies that require sequences of actions never demonstrated in the data.
The consequence. On tasks where the dataset is insufficient to solve the problem, CQL will not solve it. The paper provides clear evidence of this across multiple benchmarks:
- On
relocate-humanandrelocate-cloned(Table 2), all methods, including both CQL variants, score near zero. The human demonstrations are too limited or too poor to enable learning. - On
antmaze-large-playandantmaze-large-diverse(Table 2), CQL(H) achieves only 15.8 and 14.9 respectively, while the maximum possible return (goal-reaching) is 100. The performance, while better than the 0.0 achieved by all baselines, is far from solving the task. - On the hardest Atari games with 1% data (Table 3), CQL achieves 19.3 on Pong and 61.1 on Breakout—far from the near-optimal scores achievable with more data or online training.
This is not a bug—it's an inherent constraint of offline RL. But it has crucial practical implications: offline RL with CQL is appropriate when the dataset contains demonstrations of effective behavior that the algorithm can stitch together or refine, not when the dataset is uniformly poor. The paper's safe policy improvement guarantee (Theorem 3.6) formalizes this: the guarantee is relative to the behavior policy, not relative to some absolute performance standard. If the behavior policy is terrible, CQL ensures the learned policy won't be much worse, but it doesn't promise it will be much better.
What evidence exists in the paper. The evidence is distributed across all experimental sections. Table 2 shows that on relocate-human, CQL(H) achieves 0.20 and CQL(ρ) achieves 0.35—functionally zero. The AntMaze large-* tasks show CQL achieving only 14.9–15.8, with the performance degradation from umaze (73.5–74.0) to medium (53.7–61.2) to large (14.9–15.8) tracking the increasing mismatch between the available data and what's needed to solve the task. The Atari results show substantial but not complete performance recovery from 1% data. No experiment specifically characterizes the relationship between dataset quality and CQL's improvement over BC—for instance, how much expert data must be present in a mixed dataset for CQL to substantially outperform BC? This is a practical question the paper leaves unanswered.
Mitigation status. The paper does not address this limitation explicitly, though it is implicitly acknowledged by the safe policy improvement framework: the guarantee is improvement over $\hat{\pi}_\beta$, not attainment of any absolute performance level. The paper does not provide guidance on assessing whether a given dataset is "good enough" for CQL to be effective, nor does it propose methods for combining CQL with techniques that can extrapolate beyond the data (e.g., model-based planning, data augmentation, or incorporation of prior knowledge).
6.6 The Experiments Do Not Verify that the Safe Policy Improvement Bound Is Practically Meaningful
The assumption or constraint. Theorem 3.6 provides a formal safe policy improvement guarantee that decomposes the performance of the CQL-optimal policy relative to the behavior policy into a sampling error penalty (which decreases with dataset size) and an empirical improvement term (which is non-negative). This bound is one of the paper's major theoretical contributions and is positioned as providing the deployment safety assurances that make offline RL viable for real-world applications.
However, the bound depends on concentration constants $C_{r,\delta}$ and $C_{T,\delta}$ for the reward and transition functions, the discount factor γ, the maximum reward R_max, and the state-conditional divergence $D_{CQL}(\pi^*, \hat{\pi}_\beta)(s)$. None of these quantities are estimated or reported in the experiments, and the bound is never evaluated numerically for any of the benchmark tasks.
The consequence. Without empirical evaluation, it is unknown whether Theorem 3.6 provides a useful guarantee or a vacuous one. In many RL theory papers, bounds of this form—involving concentration constants, inverse powers of (1-γ), and worst-case quantities like R_max—produce numerical values that are far larger than the maximum possible return, meaning the guarantee that J(π*, M) ≥ J(π̂_β, M) - ζ is trivially satisfied (ζ > R_max/(1-γ)) but provides no useful information. The paper does not address this.
A practitioner who wants to deploy CQL with a safety guarantee needs to know: for a dataset of size N, with discount factor γ=0.99, what is the actual bound on potential performance degradation? If ζ = 1000 on a task where max return is 100, the guarantee is meaningless—the policy could be catastrophically worse than the behavior policy and still satisfy the bound. The paper provides no evidence that ζ takes practically reasonable values for realistic dataset sizes.
Furthermore, the bound depends on $D_{CQL}(\pi^*, \hat{\pi}_\beta)$, the divergence between the learned policy and the behavior policy. This quantity is an output of the algorithm—it is determined by the optimization, not set by the practitioner. For the bound to be useful in practice, one would need to compute $D_{CQL}$ after training and verify that the ζ value is acceptable. The paper does not report $D_{CQL}$ for any experiment, nor does it demonstrate that typical values produce useful bounds.
What evidence exists in the paper. None. The safe policy improvement guarantee is presented purely theoretically. The experimental section verifies that CQL learns lower-bound Q-functions (Table 4) and achieves strong performance, but does not connect these results to Theorem 3.6. There is no computation of $D_{CQL}(\pi^*, \hat{\pi}_\beta)$, no estimation of the concentration constants, no bounding of the sampling error term, and no comparison of the bound's predictions (e.g., that performance degradation should scale as $1/\sqrt{|\mathcal{D}|}$) to empirical scaling behavior. The paper's statement that "CQL provides a ζ-safe policy improvement over π̂_β" (Theorem 3.6) is a theoretical claim whose practical significance is completely unvalidated.
Mitigation status. The paper does not acknowledge this gap. The theoretical analysis in Theorem 3.6 is presented as a self-contained contribution, and the experiments are presented as validating the practical performance of CQL. The connection between the two—whether the theoretical guarantee actually applies to the empirical results—is not discussed. Future work on empirically evaluating the tightness of such bounds, or on deriving more practical bounds that use estimable quantities rather than concentration constants, would be needed to bridge this gap and make the safe improvement guarantee actionable for practitioners.
7. Implications and Future Directions
How This Work Changes the Landscape
This paper fundamentally shifts the conceptual framing of offline RL from a policy-constraint problem to a value-regularization problem. Before CQL, the dominant paradigm treated the core difficulty of offline RL—distributional shift causing overestimated Q-values on out-of-distribution actions—as something that must be addressed by explicitly constraining the learned policy to stay near the behavior policy. Methods like BEAR, BRAC, and others all converged on variations of "restrict π to be close to π̂_β." CQL reframes the solution entirely: don't constrain the policy—fix the Q-function instead. If the Q-function systematically prefers in-distribution actions, the policy will naturally stay near the data without any explicit constraint.
This is not merely a different implementation of the same idea. It represents a genuinely different causal model of why offline RL fails and what must be done to fix it. Policy-constraint methods implicitly assume that the Q-function's tendency to overestimate OOD actions is an unavoidable consequence of function approximation, and the best we can do is prevent the policy from exploiting it. CQL challenges that assumption: the overestimation is caused by the interaction between Bellman backups and function approximation on limited data, and it can be directly counteracted by modifying the Q-learning objective itself. The gap-expanding property (Theorem 3.4) formalizes this: CQL's Q-function doesn't just avoid overestimating OOD actions—it actively makes them look worse than in-distribution actions, by an amount proportional to the regularizer strength α.
The shift has both theoretical and practical consequences:
Theoretical reframing. Prior theoretical analyses of offline RL typically bounded performance in terms of concentrability coefficients or distribution ratios that capture the divergence between π and π_β (e.g., Chen and Jiang, 2019; Xie and Jiang, 2020). These analyses assumed the algorithm would constrain the policy. CQL's analysis (Theorems 3.5 and 3.6) shows that the same χ²-type divergence can be achieved by regularizing the Q-function rather than constraining the policy—the divergence appears in the implicit objective of the Q-learning procedure, not in an explicit policy constraint. This suggests that the theoretical study of offline RL should focus on properties of the Q-function update operator (gap expansion, conservative fixed points) rather than on policy divergence bounds.
Practical reframing. CQL eliminates the need for behavior policy estimation, which the paper identifies as a critical failure point in prior methods. On walker2d-medium-expert, where the data comes from a mixture of medium and expert policies, BEAR achieves 10.8—worse than behavioral cloning at 11.3—while CQL achieves 98.7. This is not a marginal improvement from better tuning; it's a qualitative difference between a method that succeeds and one that catastrophically fails. The mechanism is clear: when the behavior policy is multi-modal (a mixture of distinct policies), density estimation is hard, and constraining π to match a poorly estimated π̂_β is either too loose (allowing overestimation) or too tight (preventing improvement). CQL sidesteps this entirely by using only the empirical average Q-value on batch actions for regularization, with no density model.
Reconciling prior contradictions. The paper resolves a tension in prior offline RL work: uncertainty-based methods (ensembles, bootstrap DQN, REM) sometimes worked and sometimes didn't, with no clear explanation of when or why. Table 4 provides the answer: even ensembles of 20 Q-functions produce astronomically overestimated values (e.g., 885,000 on hopper-medium), because uncertainty estimates from finite ensembles are not calibrated to prevent overestimation. CQL's regularizer, by directly penalizing large Q-values for OOD actions, achieves conservative estimates (e.g., -7.48 on hopper-medium) that are orders of magnitude more accurate. This doesn't mean ensembles are useless—CQL itself uses twin Q-networks—but the paper clarifies that ensembles alone address only part of the problem. Uncertainty estimation asks "how confident are we in this Q-value?" while CQL asks "should this action be preferred?", and these are different questions.
Which research directions become more attractive. The paper's success with a simple regularizer suggests that designing better Q-function penalties is a more promising direction than designing more sophisticated policy constraints. Rather than asking "what distance metric should we use to constrain the policy?", the relevant question becomes "what penalty on the Q-function produces the right degree of conservatism?" This opens up connections to distributionally robust optimization (the CQL(var) variant in Appendix A suggests variance-based penalties), adversarial training (the min-max formulation in Section 3.2 already has an adversarial flavor), and calibrated uncertainty estimation (where the uncertainty is used to weight the conservatism rather than directly selecting actions).
The paper also makes stitching—composing fragments of suboptimal trajectories into better behavior—a central evaluation criterion. The AntMaze results show CQL achieving 61.2 on medium-play where all baselines get 0.0. This isn't just "better sample efficiency" or "more stable training"; it's evidence that conservative Q-learning enables a form of compositional generalization that policy-constraint methods prevent. Future offline RL methods should be evaluated on their ability to stitch, not just on their ability to reproduce or slightly improve upon the behavior policy.
Which directions become less attractive. The paper implicitly makes behavior-policy estimation a less attractive research investment. CQL's strongest results are precisely where behavior policy estimation is hardest (multi-policy datasets, Adroit human demonstrations), and it achieves these results without estimating the behavior policy at all. This suggests that effort spent on better behavior policy density models—generative models, normalizing flows, variational inference over behavior policies—may be misplaced, because the problem can be circumvented entirely by regularizing the Q-function instead. Similarly, pure uncertainty-based methods without explicit conservatism (like REM, which relies on randomized ensembles to approximate a lower-confidence bound) are shown to be insufficient.
The paper also narrows the role of robust MDP theory in practical offline RL. Robust MDPs provide worst-case guarantees by considering all MDPs within an uncertainty set, but CQL's policy-level (rather than pointwise) lower bound achieves effective conservatism without the crippling pessimism of worst-case formulations. The theoretical connection to robust optimization (the CQL(var) variant) suggests a more nuanced role: robust optimization principles inform the regularizer's form, but the regularizer is applied to Q-values, not to MDP parameters.
Follow-Up Research This Work Enables
Characterizing when CQL fails due to excessive conservatism and when it fails due to insufficient conservatism, on the same task, as α varies. The paper reports that α=∞ (τ=2.0 in the Lagrange formulation) causes "highly underestimated Q-functions" and performance collapse, while α too small (τ=10.0 on Adroit) fails to prevent overestimation. But we don't know the shape of the performance-vs-α curve for any task. Is there a broad plateau where performance is stable, or a narrow peak? Does the acceptable α range narrow as the dataset becomes more complex? A systematic study sweeping α on 3–4 D4RL tasks (one single-policy, one multi-policy, one Adroit, one AntMaze) with performance, Q-value magnitude, and policy divergence from π̂_β all tracked, would give practitioners a diagnostic toolkit: "if your Q-values are in range X and your policy divergence is Y, your α is probably acceptable." The paper provides this direction implicitly through the Lagrange threshold experiments (Appendix F), but never maps the α-to-performance landscape.
Can CQL be combined with model-based offline RL to handle the hardest tasks where purely model-free methods plateau? On antmaze-large-play, CQL achieves only 15.8 (out of 100). This is far better than 0.0 (all baselines), but far from solving the task. Model-based methods (e.g., MOPO, MOReL) can plan through learned dynamics to compose trajectories, but suffer from model exploitation where the planner finds dynamics-adversarial regions. CQL's conservative Q-function could serve as the terminal value function for model-based planning, preventing the planner from being over-optimistic about model predictions far from the data. A concrete experiment: train a dynamics model on the AntMaze datasets, use CQL to learn a conservative value function, and run model-predictive control with CQL's Q-function as the terminal cost. Compare to pure CQL (15.8 on large-play) and pure MOPO (likely fails due to model exploitation in the maze). A positive result would show that CQL's regularizer generalizes beyond actor-critic to provide calibrated conservatism for planning.
Does CQL's gap-expanding property prevent the policy from discovering novel compositions of actions that are individually in-distribution but collectively out-of-distribution? The AntMaze results suggest not—CQL achieves 61.2 on medium-play by composing trajectory fragments, which is precisely this kind of novelty. But there's a tension: Theorem 3.4 says CQL expands the gap between in-distribution and OOD Q-values. If the composed trajectory visits a state where the composed action is technically OOD (because no single trajectory in the data took that exact action at that exact state), the gap expansion could penalize it. The experiment: on AntMaze, measure the average Q-value advantage of the composed action relative to the nearest in-distribution action, as a function of trajectory length, for CQL vs. BEAR. If CQL still assigns higher Q-values to composed actions than BEAR does (despite gap expansion), it would demonstrate that CQL's regularizer is selective—penalizing actions that are far from any data-supported action, not those that are novel but near data-supported actions. This would clarify the mechanism of stitching and inform the design of regularizers that encourage composition.
Can the importance sampling issue in CQL(H) be resolved by using more sophisticated Monte Carlo methods, enabling the theoretically cleaner variant to scale to high-dimensional action spaces? The transition from CQL(H) to CQL(ρ) on Adroit is a practical hack: CQL(H)'s log-sum-exp estimator has high variance in 24 dimensions, so the more stable but less principled CQL(ρ) is used. Techniques from the Monte Carlo literature—adaptive multiple importance sampling, randomized quasi-Monte Carlo, or normalizing flows that learn a proposal distribution for the log-sum-exp integral—could dramatically reduce the variance. A concrete experiment: replace the uniform + on-policy mixture sampler in CQL(H) with a normalizing flow trained to minimize the variance of the importance-weighted log-sum-exp estimate, updated jointly with the Q-function (or in alternating steps). Test on Adroit tasks where CQL(H) currently underperforms CQL(ρ) (e.g., pen-human, hammer-cloned). If the flow-based CQL(H) matches or exceeds CQL(ρ), it would eliminate the need for the policy-conditioned variant and provide a principled, variance-adaptive estimator that works across action dimensionalities.
Does CQL's safe policy improvement bound (Theorem 3.6) produce practically meaningful guarantees for realistic offline RL deployments, or is it vacuous? The bound depends on concentration constants that are never estimated, and the numerical values it would produce for typical D4RL tasks are unknown. A concrete project: on a single D4RL task (e.g., hopper-medium), estimate or bound $C_{r,\delta}$ and $C_{T,\delta}$ from the dataset (using bootstrap or Hoeffding-style concentration), estimate $D_{CQL}(\pi^*, \hat{\pi}_\beta)(s)$ from the learned policy and empirical behavior counts, and compute the bound on $J(\pi^*, M) - J(\hat{\pi}_\beta, M)$. Compare this bound to the empirically observed improvement (or degradation) when deploying CQL policies. If the bound is orders of magnitude larger than the maximum possible return (e.g., ζ = 10,000 on a task with max return 100), the guarantee is theoretically correct but practically useless—and the field needs tighter analyses. If the bound is within a factor of 2–5 of the empirical difference, the safe improvement claim is genuinely actionable and represents a rare bridge between offline RL theory and practice. A negative result would motivate work on tighter concentration inequalities or data-dependent bounds that use empirical Bernstein rather than Hoeffding.
Can CQL's regularizer be adapted to prevent overfitting rather than just overestimation, providing a unified perspective on the failure modes of offline Q-learning? The paper identifies overestimation as the core problem, but Table 4 reveals another pathology: ensemble methods overestimate by orders of magnitude more than the true value (trillions vs. hundreds), which goes beyond "optimism" into a regime of complete Q-function breakdown. This looks like overfitting: the Q-function memorizes the dataset's rewards and extrapolates wildly. CQL's regularizer $-\mathbb{E}_{\hat{\pi}_\beta}[Q]$ can be interpreted as a data-dependent prior that prevents memorization. A concrete experiment: train CQL with varying α on datasets with injected noise (random rewards on a fraction of transitions) and measure generalization error (value prediction accuracy on held-out transitions). Compare to ensembles and policy-constraint methods. If CQL's regularizer reduces overfitting independently of its effect on overestimation, this would suggest a broader role for value regularization in offline RL—not just conservatism, but also generalization. This connects to the paper's call for "simple and effective early stopping methods": if the CQL regularizer provides a natural overfitting signal (when the penalty term grows, the Q-function may be memorizing), it could serve as a validation proxy.
Practical Applications and Downstream Use Cases
Industrial robotics with demonstration data. A factory deploys a robot arm for a new assembly task. Rather than programming the policy manually or running online RL (risking hardware damage, production downtime, and thousands of real-world trials), operators collect demonstration data by teleoperating the robot through the task for several hours—yielding a few hundred trajectories. These demonstrations are suboptimal (human operators are not perfect) and cover only a fraction of the state space. CQL can learn from this data offline: on the Franka Kitchen tasks (Table 2), CQL achieves 43.8–52.4% success rate from human teleoperation data, while the best prior method (BEAR) gets 0.0–47.2% and behavioral cloning gets 33.8–47.5%. The concrete benefit: CQL extracts more value from limited demonstration data than imitation learning alone, without requiring online interaction. The safe policy improvement guarantee (Theorem 3.6) provides formal assurance that the deployed policy will not be substantially worse than the human operator—critical in industrial settings where a policy that breaks equipment is unacceptable. The practical workflow: deploy CQL, verify that the learned policy achieves acceptable performance in the first few real-world trials, and iterate by collecting more demonstrations on the failure cases.
Healthcare treatment optimization from electronic health records. A hospital has years of electronic health records documenting which treatments were administered to which patients and what outcomes occurred. The behavior policy is the collective practice of hundreds of clinicians, producing a complex, multi-modal data distribution. Online experimentation (testing new treatment policies on patients) is unethical, making offline RL the only viable approach. CQL's primary advantage—no behavior policy estimation required on multi-modal data—is directly relevant here. The behavior policy is a mixture of many clinicians' practices; modeling this density accurately is infeasible. CQL instead regularizes the Q-function to be conservative about treatment strategies that deviate from observed practice. The paper's results on multi-policy D4RL datasets (Table 1: CQL achieves 92.5 on halfcheetah-random-expert vs. BEAR's 24.6) suggest CQL can effectively learn from such heterogeneous data. The value-based conservatism is particularly appropriate for healthcare: the algorithm will only deviate from standard practice when the data provides strong evidence that an alternative is better, which is exactly the burden of proof required for clinical decision-making.
Autonomous driving from logged fleet data. A fleet of autonomous vehicles generates petabytes of driving data—camera feeds, LIDAR, control actions—from a mixture of human drivers and existing autonomy stacks. Most of the data is "uneventful" (normal highway driving), with rare safety-critical events. Online RL is obviously impossible for safety-critical scenarios. CQL can learn driving policies offline from this data. The sparse nature of critical events means the Q-function must be conservative about actions with little evidential support (e.g., an unusual evasive maneuver), which is precisely what CQL's regularizer enforces. The gap-expanding property (Theorem 3.4) ensures that the policy will prefer standard, well-demonstrated maneuvers over speculative alternatives, unless the data strongly supports the alternative—a desirable property for safety-critical systems. The paper does not test on driving domains, so this application requires validation, but the structural parallels to the AntMaze tasks (composing fragments of many trajectories into goal-directed behavior in a high-dimensional continuous space) are promising. The concrete integration: train CQL on the fleet data, use the learned Q-function to score candidate trajectories from a motion planner, and deploy with the safe improvement guarantee providing a bound on the worst-case performance relative to the fleet's current behavior.
When to Prefer This Method
The paper positions CQL as a general-purpose offline RL algorithm, demonstrating strong performance across a diverse range of benchmarks. It does not propose explicit tradeoffs against named alternatives as part of a decision framework (e.g., "prefer CQL over BEAR when X; prefer BEAR over CQL when Y"). Rather, the paper's positioning is that CQL is a simpler, more robust replacement for prior methods across settings. The comparisons in the experiments consistently show CQL matching or exceeding the best prior method on nearly every task, and the cases where it falls short (e.g., slightly below BRAC-v on walker2d-medium, slightly below BC on pen-cloned) are presented as exceptions to a generally dominant pattern rather than as evidence for a principled tradeoff. The paper does not articulate conditions under which a practitioner should choose BEAR, BRAC, or REM over CQL. Consequently, a prescriptive "prefer X when..." matrix would impose structure that the paper itself does not provide—it would be my interpretation, not the authors' articulated framework. The closest the paper comes to a conditional recommendation is the CQL(H) vs. CQL(ρ) choice within the CQL family (Section 6, Appendix F): use CQL(H) for low-to-moderate action dimensionalities (≤ ~10 DoF, as in MuJoCo and Kitchen), and CQL(ρ) for high-dimensional action spaces (24-DoF Adroit) where the importance sampling variance in CQL(H) becomes prohibitive.
