Asynchronous Methods for Deep Reinforcement Learning

ArXiv: 1602.01783

🎯 Pitch

Simply running multiple agents in parallel on a single CPU, without any experience replay, can stabilize deep reinforcement learning across four different algorithms—and the resulting asynchronous actor-critic method (A3C) beats GPU-accelerated DQN on Atari while training in half the time.

---

1. Executive Summary

This paper introduces a lightweight framework for deep reinforcement learning that replaces experience replay with asynchronous gradient descent across multiple parallel actor-learners running on a single multi-core CPU. The authors present asynchronous variants of four standard RL algorithms — one-step Q-learning, one-step Sarsa, n-step Q-learning, and advantage actor-critic — and demonstrate that parallel actor-learners exert a stabilizing effect on training, enabling all four methods to successfully train neural network controllers on the Atari 2600 domain. The best-performing method, Asynchronous Advantage Actor-Critic (A3C), surpasses the state-of-the-art on 57 Atari games while training in half the time of GPU-based methods on a single CPU, achieving a mean human-normalized score of 496.8% with a feedforward network and 623.0% with an LSTM after four days of training. The framework also succeeds on continuous motor control tasks using MuJoCo and on a new 3D maze navigation task from visual input, establishing that asynchronous parallel actor-learners can substitute for experience replay as a stabilization mechanism across value-based and policy-based, on-policy and off-policy, and discrete and continuous domains — though this stabilization effect is most pronounced when using shared RMSProp optimization with per-thread exploration diversity.

2. Context and Motivation

The Core Problem: Deep RL Was Stuck on Experience Replay

When this paper was written in 2016, deep reinforcement learning had just achieved its breakthrough moment. The DQN algorithm (Mnih et al., 2015) had demonstrated that a single neural network architecture could learn to play 49 Atari 2600 games directly from pixels, achieving human-level performance on many of them. This was a landmark result — it showed that deep neural networks could serve as function approximators for reinforcement learning in high-dimensional state spaces, something that had been attempted and largely failed for decades.

But there was a catch, and it was a big one: DQN only worked because of experience replay. The algorithm stored every transition $(s_t, a_t, r_t, s_{t+1})$ the agent experienced into a large replay buffer, and then trained the Q-network by sampling random minibatches from this buffer. This mattered for two interconnected reasons that every deep RL practitioner at the time understood viscerally:

First, the non-stationarity problem. In online RL, an agent generates data sequentially by interacting with an environment using its current policy. The distribution of states, actions, and rewards it encounters is constantly shifting as the policy improves. Neural networks trained on such non-stationary data streams are notoriously unstable — the optimization landscape keeps changing under the optimizer's feet, and gradient descent can diverge or collapse. Experience replay breaks this temporal structure by shuffling data from many different time steps and policy iterations together, creating something closer to the i.i.d. data assumption that supervised deep learning relies on.

Second, the correlation problem. Consecutive samples from an RL agent are highly correlated — the state at time $t+1$ is almost identical to the state at time $t$ (one frame apart in Atari, for example), and the actions and rewards are causally linked. Training on highly correlated sequences causes the network to overfit to the recent trajectory and makes gradient updates highly variable, since each update pushes the network in a direction specific to the current local region of state space rather than improving performance globally. Random sampling from a large replay buffer decorrelates these updates.

The authors put it concisely in the introduction:

"the sequence of observed data encountered by an online RL agent is non-stationary, and online RL updates are strongly correlated. By storing the agent's data in an experience replay memory, the data can be batched or randomly sampled from different time-steps. Aggregating over memory in this way reduces non-stationarity and decorrelates updates, but at the same time limits the methods to off-policy reinforcement learning algorithms."

That last clause — "limits the methods to off-policy reinforcement learning algorithms" — is the crux of the problem this paper sets out to solve.

The Hidden Cost of Experience Replay: It Blocks On-Policy Methods

Experience replay is not a free lunch. It comes with three specific drawbacks that the paper identifies:

1. Memory and computation overhead. Maintaining a replay buffer of, say, one million transitions (as DQN did) consumes substantial RAM. Each training step requires sampling a minibatch from this buffer and computing gradients — operations that are independent of the agent's actual interaction with the environment. In DQN, the agent takes 4 environment steps (with action repeat), then performs one minibatch update from the replay buffer. This means a significant fraction of the system's computational resources are spent on replay sampling and gradient computation rather than on collecting new experience. As the paper notes:

"experience replay has several drawbacks: it uses more memory and computation per real interaction; and it requires off-policy learning algorithms that can update from data generated by an older policy."

2. The off-policy straitjacket. This is the deeper, more fundamental limitation. Experience replay inherently mixes data from different policies — the buffer contains transitions generated by the agent's policy from many iterations ago alongside transitions from recent policies. Any algorithm that learns from this buffer must be an off-policy algorithm: it must be able to learn from data generated by a behavior policy that differs from the current target policy being optimized.

Q-learning happens to be off-policy (it learns about the optimal policy regardless of which policy generated the data), which is why DQN works with experience replay. But many powerful RL algorithms are on-policy — they require data generated by the current policy to compute valid updates. These include:

• Sarsa (State-Action-Reward-State-Action), which updates Q-values using the action actually taken under the current policy rather than the maximum Q-value. Sarsa's update target is $r + \gamma Q(s', a'; \theta)$ where $a'$ is the action the agent actually took, not the optimal action. This makes it on-policy: the target depends on the behavior policy's action at the next step.

• Actor-critic methods, which maintain a separate policy network (the actor) and value network (the critic). The policy gradient update $\nabla_\theta \log \pi(a_t|s_t; \theta) (R_t - V(s_t))$ is an on-policy gradient — it needs to be computed using actions sampled from the current policy $\pi_\theta$. Using data from an old policy introduces bias that can destabilize or prevent learning.

• n-step methods, which use multi-step returns like $r_t + \gamma r_{t+1} + \cdots + \gamma^{n-1} r_{t+n-1} + \gamma^n \max_a Q(s_{t+n}, a)$. These propagate rewards faster through the value function, but the n-step return is only correct if the intermediate actions come from the policy being evaluated — making them on-policy in the general case.

Because experience replay was considered essential for stabilizing deep RL training, and experience replay forced the use of off-policy algorithms, the entire field was effectively locked out of using on-policy methods with deep neural networks. This was a serious constraint. On-policy methods have well-known advantages: they tend to be more stable in theory (since the update target matches what the agent actually experiences), they can naturally incorporate exploration through stochastic policies (via entropy regularization or learned variance), and they extend gracefully to continuous action spaces where Q-learning requires a separate optimization over actions at each step.

3. Slower reward propagation with one-step methods. Even within the off-policy family, standard Q-learning only uses one-step lookahead: the update target is $r + \gamma \max_{a'} Q(s', a'; \theta)$. When the agent receives a reward, only the single state-action pair that immediately preceded it gets a direct value update. The values of earlier state-action pairs are affected only indirectly through the Bellman backup propagated one step at a time through future updates. This makes learning slow, especially in games with sparse or delayed rewards — a common situation in RL. The paper explains:

"One drawback of using one-step methods is that obtaining a reward $r$ only directly affects the value of the state action pair $s, a$ that led to the reward. The values of other state action pairs are affected only indirectly through the updated value $Q(s, a)$. This can make the learning process slow since many updates are required to propagate a reward to the relevant preceding states and actions."

Multi-step (n-step) returns can accelerate this propagation by directly updating $n$ state-action pairs per reward. But n-step methods, as noted above, are on-policy when used with the actions actually taken, which again conflicts with experience replay.

What Prior Solutions Existed, and Why They Didn't Fully Solve the Problem

By 2016, several approaches had been developed to stabilize deep RL, all of them working within the experience replay paradigm:

DQN and its variants (Mnih et al., 2013; 2015). The original solution combined experience replay with a target network — a separate, slowly-updated copy of the Q-network used to compute the target values $r + \gamma \max_{a'} Q(s', a'; \theta^-)$. The target network parameters $\theta^-$ are updated to match the main network $\theta$ only every $C$ steps (typically thousands of frames), which reduces the harmful feedback loop where the Q-network chases a moving target that itself depends on the Q-network's parameters. This was effective but still purely off-policy and relied on one-step updates.

Double DQN (Van Hasselt et al., 2015). This addressed the overestimation bias in Q-learning — the fact that $\max_{a'} Q(s', a'; \theta)$ systematically overestimates the true value because the max operator picks the action with the highest estimated Q-value, which tends to be the one where the estimation error is most positive. Double DQN decouples action selection from action evaluation: it uses the online network to select the action $a^* = \arg\max_{a'} Q(s', a'; \theta)$ and the target network to evaluate it, computing $Q(s', a^*; \theta^-)$. This reduces but does not eliminate the bias, and still operates entirely within the replay buffer + off-policy framework.

Dueling DQN (Wang et al., 2015). This architectural modification splits the Q-network into two streams: one that estimates the state value $V(s)$ and one that estimates the advantage $A(s, a)$ for each action. These are combined as $Q(s, a) = V(s) + A(s, a) - \frac{1}{|\mathcal{A}|}\sum_{a'} A(s, a')$ to recover the Q-values. The dueling architecture learns more efficiently because it can generalize value estimates across actions without needing to experience every action in every state. Again, this is a modification within the experience replay paradigm.

Prioritized Experience Replay (Schaul et al., 2015). Rather than sampling uniformly from the replay buffer, this method samples transitions with probability proportional to their TD error — the magnitude of the difference between the current Q-value and the target. Transitions that the network is currently "wrong" about (high TD error) are replayed more often, improving sample efficiency. This is an enhancement to experience replay, not an alternative.

Gorila (Nair et al., 2015). The closest predecessor to this paper. Gorila distributed DQN across a massive system: 100 separate actor-learner processes, each with its own copy of the environment and its own replay buffer, plus 30 parameter server instances. Actors generated experience and computed gradients from their local replay buffers, then asynchronously sent gradients to the parameter servers. The parameter servers updated a central model and periodically pushed the updated parameters back to the actors. By using 130 machines, Gorila significantly outperformed DQN — reaching DQN's final score up to 20 times faster on many games. But it still relied on experience replay within each actor, still used off-policy Q-learning exclusively, and required an enormous distributed infrastructure.

The missing piece in all these approaches. Every solution to the stability problem — DQN, Double DQN, Dueling DQN, Prioritized Replay, Gorila — was a variation on the same theme: find a better way to use experience replay. None of them questioned whether experience replay was necessary in the first place. None of them enabled on-policy methods like Sarsa or actor-critic to work with deep neural networks. And all of them required substantial computational resources — GPUs for training (8-10 days on an Nvidia K40 GPU) or, in the case of Gorila, a cluster of over 100 machines.

The Gap: A Stabilization Mechanism That Doesn't Require Experience Replay

The specific gap this paper identifies is deceptively simple: can we achieve the stabilizing effect of experience replay — decorrelated, stationary-ish training data — without actually using a replay buffer, and thereby unlock on-policy deep RL algorithms?

The paper's key insight is that parallelism itself provides decorrelation. Imagine you have 16 agents, each running in its own independent copy of the environment, each at a different point in its own episode, each following a slightly different exploration policy (different values of $\epsilon$). At any given wall-clock moment, these 16 agents are experiencing 16 different states, taking 16 different actions, receiving 16 different rewards. If they all compute gradient updates from their own recent experience and apply them to a shared model, the aggregated update is computed from data that spans many different regions of the state space and many different points in the temporal dynamics of the environment. This is decorrelation by construction — no replay buffer needed.

The authors make this argument explicitly in Section 4:

"multiple actors-learners running in parallel are likely to be exploring different parts of the environment. Moreover, one can explicitly use different exploration policies in each actor-learner to maximize this diversity. By running different exploration policies in different threads, the overall changes being made to the parameters by multiple actor-learners applying online updates in parallel are likely to be less correlated in time than a single agent applying online updates. Hence, we do not use a replay memory and rely on parallel actors employing different exploration policies to perform the stabilizing role undertaken by experience replay in the DQN training algorithm."

This is a genuinely different paradigm. Instead of storing old data to create diversity, you generate diversity by having multiple agents explore simultaneously. The theoretical justification draws on work by Tsitsiklis (1994), who studied the convergence of asynchronous Q-learning and showed that Q-learning still converges under outdated information as long as certain technical conditions hold — essentially, the asynchronous update structure has known convergence properties that this framework exploits.

Why This Matters: Practical and Theoretical Significance

The gap matters for several reasons that span practical deployment, algorithmic generality, and research direction:

Practical: Democratizing deep RL compute. In 2016, training a DQN agent on Atari required a high-end GPU (Nvidia K40) running for 8-10 days. Gorila required 130 machines. These resource requirements put deep RL research out of reach for most academic labs and individual researchers. If a single multi-core CPU machine could achieve comparable or better results in less time, it would dramatically lower the barrier to entry and accelerate research iteration cycles.

Practical: Enabling on-policy methods for deep RL. Sarsa, n-step methods, and actor-critic had been workhorses of RL with linear function approximation for decades, but the deep RL revolution had largely passed them by because no one knew how to make them stable with neural networks without experience replay. Unlocking these methods meant access to:

• Actor-critic for continuous control. In continuous action spaces, Q-learning requires solving $\max_a Q(s, a; \theta)$ at every step, which requires a separate optimization procedure (as in DDPG; Lillicrap et al., 2015). Policy gradient methods handle continuous actions natively by outputting parameters of a continuous distribution (e.g., mean and variance of a Gaussian).

• Entropy regularization for exploration. Policy-based methods can directly encourage exploration by adding an entropy bonus $-\beta H(\pi(\cdot|s))$ to the objective, which prevents the policy from collapsing to a deterministic action too quickly. This is much more principled than $\epsilon$-greedy exploration, which adds uniform noise regardless of the value landscape.

• Natural integration of recurrent policies. On-policy methods with forward-view n-step returns can naturally backpropagate through time through recurrent states, enabling LSTM-based agents that maintain internal memory across time steps.

Theoretical: Understanding why deep RL is stable. If parallelism could substitute for experience replay as a stabilization mechanism, it would clarify why experience replay works in the first place. It's not the storage and resampling per se that matters — it's the decorrelation and stationarity they provide. Parallelism provides these same properties through a different mechanism (spatial diversity across agents rather than temporal diversity across stored transitions), suggesting that decorrelation is the fundamental requirement, and experience replay is just one way to achieve it.

How the Paper Positions Itself

The paper positions itself as introducing a new paradigm for deep reinforcement learning — not an incremental improvement to existing replay-based methods, but a fundamentally different approach to the stability problem:

"In this paper we provide a very different paradigm for deep reinforcement learning. Instead of experience replay, we asynchronously execute multiple agents in parallel, on multiple instances of the environment."

The key differentiators from prior work are:

Versus DQN and its variants: Complete removal of experience replay, enabling on-policy methods, reducing memory requirements, and running entirely on CPU rather than GPU. The paper explicitly notes that its methods "can be applied robustly and effectively using deep neural networks" in ways that were previously thought impossible without experience replay.

Versus Gorila: Same idea of asynchronous parallel actors, but (a) on a single machine with multiple CPU threads instead of 130 machines, (b) using Hogwild!-style lock-free updates (Recht et al., 2011) instead of a parameter server architecture, and (c) most importantly, without experience replay in each actor. The paper frames Gorila as proof that asynchronous training helps, but notes that Gorila still relied on replay buffers within each actor and was limited to Q-learning. By removing experience replay entirely and using the parallelism itself as the sole decorrelation mechanism, the paper's framework can train a much broader class of algorithms.

Versus prior theory on asynchronous RL: Tsitsiklis (1994) proved convergence of asynchronous Q-learning under certain conditions, and Bertsekas (1982) studied distributed dynamic programming. These theoretical results suggested that asynchronous updates could work, but no one had demonstrated that they could stabilize deep neural network training on challenging domains like Atari. The gap between "converges in theory with look-up tables" and "works in practice with convolutional neural networks on 57 Atari games" was enormous, and this paper bridges it.

What the paper is NOT claiming: The authors are careful not to claim that experience replay is obsolete or useless. In the conclusion, they explicitly state:

"While this shows that stable online Q-learning is possible without experience replay, which was used for this purpose in DQN, it does not mean that experience replay is not useful. Incorporating experience replay into the asynchronous reinforcement learning framework could substantially improve the data efficiency of these methods by reusing old data."

This is an important nuance. The paper's contribution is showing that parallelism can substitute for experience replay as a stabilizer, not that it should always replace it. The two mechanisms are complementary — parallelism provides decorrelation through spatial diversity; experience replay provides decorrelation through temporal diversity and also improves sample efficiency by reusing data. The authors envision combining them for maximum benefit, but first needed to establish that parallelism alone is sufficient to make on-policy deep RL work.

3. Technical Approach

3.1 Reader Orientation

This paper builds an asynchronous parallel training system for deep reinforcement learning that replaces experience replay buffers with multiple agent threads running simultaneously on a single multi-core CPU. The problem it solves is that deep RL with neural networks was previously thought to require experience replay for stability, which locked the field into off-policy algorithms like DQN and prevented the use of on-policy methods like Sarsa and actor-critic — the "shape" of the solution is to let parallel actor-learners naturally decorrelate training data by exploring different parts of the environment at the same time, so that the aggregate gradient update computed from their combined experience is already diverse enough to train stably without storing and reshuffling old transitions.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components running on a single machine with multiple CPU threads:

1. Global shared model — a single set of neural network parameters (policy, value function, or Q-function weights depending on the algorithm) stored in shared memory accessible to all threads without locks. These are the parameters being trained.

2. Multiple actor-learner threads — each thread maintains its own copy of the environment, its own exploration policy (with different $\epsilon$ values or stochastic action sampling), and its own accumulation buffer for gradients. Threads operate independently: they interact with their environment, compute local gradient updates from their own experience, and asynchronously apply those updates to the shared global model using Hogwild!-style lock-free updates.

3. Environment instances — each thread has an independent copy of the environment (e.g., a separate Atari emulator process), ensuring that threads experience different states, different episode trajectories, and different reward sequences simultaneously.

4. Optimization engine — the algorithm for applying gradient updates to the shared parameters. After extensive comparison, the paper settles on Shared RMSProp, where the moving average of squared gradients $g$ (used to normalize the learning rate per-parameter) is itself shared across threads and updated asynchronously without locks.

Information flows as follows: each thread resets or continues its environment → receives a state $s_t$ → selects an action $a_t$ using its thread-specific exploration policy → receives reward $r_t$ and next state $s_{t+1}$ → accumulates this experience for up to $t_{\text{max}}$ steps (typically 5) or until episode termination → computes gradients of the RL loss over the accumulated trajectory → asynchronously applies these gradients to the shared global parameters → repeats. The key is that 16 such threads do this simultaneously, so the shared model receives gradient updates computed from 16 different trajectories spanning different regions of state space and different stages of learning.

3.3 Roadmap for the Deep Dive

• First, the shared optimization infrastructure — Hogwild!-style asynchronous updates and the Shared RMSProp algorithm — because every method in the paper shares this foundation and the choice of optimizer turns out to be critical for stability.
• Second, the generic design principles that apply to all four algorithms: parallel actor-learners with diverse exploration, gradient accumulation over multiple timesteps, and the removal of experience replay.
• Third, asynchronous one-step Q-learning as the simplest value-based method, since it introduces the target network mechanism and the update-timing parameters that all value-based methods share.
• Fourth, asynchronous one-step Sarsa, which differs from Q-learning only in the target value computation but is significant because it is on-policy and would be impossible with experience replay.
• Fifth, asynchronous n-step Q-learning, which introduces the forward-view multi-step return computation — a key mechanism that also appears in A3C and fundamentally changes how rewards propagate through the value function.
• Sixth, asynchronous advantage actor-critic (A3C) as the most complex and highest-performing method, covering the policy network, value network, advantage estimation, entropy regularization, and the combined policy-and-value update procedure.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and empirical methods paper whose core idea is that asynchronous parallelism across multiple actor-learners can replace experience replay as the primary mechanism for stabilizing deep reinforcement learning, thereby enabling a much broader class of RL algorithms (on-policy and off-policy, value-based and policy-based) to train neural network controllers successfully.

---

Shared Optimization Infrastructure: Hogwild! Updates and Shared RMSProp

All four asynchronous algorithms in this paper share a common optimization backbone: multiple CPU threads compute gradient updates independently and apply them to a shared parameter vector without using locks. This approach, known as Hogwild! (Recht et al., 2011), exploits the fact that stochastic gradient updates are typically sparse — each update only modifies a small subset of the parameters (e.g., the weights connected to a few active neurons) — so the probability of two threads writing to the same memory location simultaneously and corrupting each other's updates is low enough that explicit locking can be skipped entirely.

Hogwild! mechanics in this context. Each thread $i$ independently:

1. Computes its own gradient accumulation $\Delta\theta_i$ from its local trajectory.
2. Reads the current shared parameters $\theta$ (without any lock or synchronization barrier).
3. Performs the update $\theta \leftarrow \theta - \eta \cdot \text{optimizer\_step}(\Delta\theta_i)$, writing the result back to shared memory element-by-element.

There is no guarantee that the $\theta$ read in step 2 hasn't been modified by another thread between the read and the write. This means some updates are computed with respect to slightly stale parameters — the theoretical justification comes from Tsitsiklis (1994), who showed that asynchronous Q-learning converges as long as outdated information is eventually discarded, and from Hogwild! theory (Recht et al., 2011), which proves convergence for lock-free SGD under sparsity assumptions. The paper implicitly relies on these guarantees while empirically demonstrating that the approach works in practice for deep neural networks on challenging RL domains.

Why not use locking? The paper mentions that their implementations "do not use any locking in order to maximize throughput when using a large number of threads" (Supplementary Section 7). Locking — where each thread must acquire a mutex before reading or writing the shared parameters — would serialize the updates and eliminate the parallelism benefit. With 16 threads all trying to update the model simultaneously, lock contention would mean most threads spend most of their time waiting. The Hogwild! approach accepts occasional write conflicts as a trade-off for near-linear throughput scaling.

The three optimizer variants compared. The paper investigated three optimization algorithms in the asynchronous setting (Supplementary Section 7, Figure S5):

Variant 1: Momentum SGD.

m_i &= \alpha m_i + (1 - \alpha)\Delta\theta_i \\ \theta &\leftarrow \theta - \eta m_i \end{align}$$ where `$m_i$` is a **per-thread** momentum vector (each thread `$i$` maintains its own `$m_i$`), `$\alpha$` is the momentum coefficient, `$\eta$` is the learning rate, and `$\Delta\theta_i$` is the gradient accumulated by thread `$i$`. The momentum vector is not shared; each thread independently maintains its velocity term. This is straightforward to implement asynchronously but means there is no cross-thread information sharing about the geometry of the optimization landscape. **Variant 2: RMSProp (per-thread statistics).** The standard non-centered RMSProp update is: $$g = \alpha g + (1 - \alpha)\Delta\theta^2$$ $$\theta \leftarrow \theta - \eta \frac{\Delta\theta}{\sqrt{g + \epsilon}}$$ where `$g$` is a moving average of elementwise squared gradients, `$\alpha$` is the decay factor (set to 0.99 in all experiments), `$\Delta\theta^2$` denotes elementwise squaring of the gradient vector, `$\epsilon$` is a small constant for numerical stability, and all operations are elementwise. In this variant, **each thread maintains its own `$g$`**, meaning each thread independently tracks its own estimate of the scale of each parameter's gradients. This is the "natural" per-thread extension of RMSProp to the asynchronous setting, but it means each thread has a different, potentially inconsistent, view of the parameter-wise learning rate scaling. **Variant 3: Shared RMSProp (the winner).** The update equations are identical to Variant 2, but the vector `$g$` is now **shared across all threads** and updated asynchronously without locking. Each thread reads the shared `$g$`, computes its gradient `$\Delta\theta$`, updates the shared `$g$` elementwise using the RMSProp recurrence, and then applies the normalized gradient to the shared parameters `$\theta$`. This means all threads collectively maintain a single running estimate of gradient variance, informed by data from all environments and exploration policies. **How the shared `$g$` update works in practice.** When thread `$i$` has accumulated its gradient `$\Delta\theta_i$`, it: 1. Reads the current shared `$g$` vector from memory (non-atomically). 2. Computes `$g_{\text{new}} = \alpha g + (1 - \alpha)\Delta\theta_i^2$` elementwise. 3. Writes `$g_{\text{new}}$` back to the shared `$g$`, overwriting whatever is there (possibly clobbering another thread's concurrent write). 4. Uses the `$g_{\text{new}}$` it just computed (or a slightly stale version if another thread overwrote it mid-computation) to compute the normalized update `$\Delta\theta_{\text{norm}} = \Delta\theta_i / \sqrt{g_{\text{new}} + \epsilon}$`. 5. Applies `$\theta \leftarrow \theta - \eta \Delta\theta_{\text{norm}}$` to the shared parameters. There is a race condition: two threads might read the same `$g$`, each compute their own `$g_{\text{new}}$`, and one will overwrite the other's contribution. But the Hogwild! theory argues that this is acceptable because the `$g$` update is itself a moving average — losing one thread's contribution to a single update step has a negligible effect on the long-term statistics, and the threads' gradient distributions are similar enough (they all train on the same task with similar network states) that the shared statistics remain useful even with occasional dropped updates. **Comparison results (Supplementary Figure S5).** The paper compared these three optimizers on two algorithms (Async n-step Q and A3C) across four games (Breakout, Beamrider, Seaquest, Space Invaders), with 50 experiments per setting using randomly sampled learning rates and initializations. The results are presented as rank-sorted curves: the x-axis shows model rank after sorting by descending final average score, and the y-axis shows the final score. In this representation: > "RMSProp with shared statistics tends to be more robust than RMSProp with per-thread statistics, which is in turn more robust than Momentum SGD." "More robust" means the curve is flatter — a flatter curve indicates that a wider range of learning rates and initializations lead to good final performance. Shared RMSProp consistently achieves higher maximum scores and maintains performance across a broader range of hyperparameters than the other two optimizers. This is the optimizer used for all experiments in the main paper. **Why Shared RMSProp wins.** The paper does not deeply analyze this, but the likely mechanism is that sharing gradient statistics provides an additional form of stabilization. Each thread sees only a narrow slice of the environment (its own trajectory, its own exploration policy), and its local gradient magnitudes might fluctuate significantly depending on what it encounters. By sharing `$g$` across threads, the learning rate normalization is informed by gradient statistics aggregated across diverse states and exploration behaviors, yielding a more stable and representative estimate of the parameter-wise gradient scale. This complements the spatial decorrelation provided by parallel actors — not only are the gradients themselves decorrelated, but the optimization step sizes are smoothed by cross-thread statistics. **Key hyperparameters (fixed across all experiments).** The paper uses `$\alpha = 0.99$` for the RMSProp decay factor and `$\gamma = 0.99$` for the RL discount factor in all experiments. The `$\epsilon$` for numerical stability in RMSProp is standard (typically `$10^{-8}$` or similar, not explicitly specified). Learning rates are sampled from `$\text{LogUniform}(10^{-4}, 10^{-2})$` for the hyperparameter search experiments (Figures 1, S5, S7, S11) and are fixed at a tuned value for the final 57-game Atari evaluation (Table 1). All learning rates are annealed to zero over the course of training. --- #### Generic Design Principles: Parallelism as Decorrelation, Gradient Accumulation, and Exploration Diversity Beyond the specific optimizer choice, three design principles apply uniformly across all four algorithms: **Principle 1: Parallel actor-learners replace experience replay for decorrelation.** This is the paper's central hypothesis. In standard DQN, experience replay decorrelates updates by storing transitions from many time steps and sampling random minibatches. The asynchronous framework achieves decorrelation differently: at any wall-clock instant, 16 agents are at different points in different episodes in different environment instances. Agent 1 might be at the start of a game of Breakout, agent 2 might be in the middle of Pong, and agent 16 might be at the terminal state. Their combined gradient updates thus span many different regions of state space without any explicit storage or resampling. The paper states: > "By running different exploration policies in different threads, the overall changes being made to the parameters by multiple actor-learners applying online updates in parallel are likely to be less correlated in time than a single agent applying online updates." This is the critical substitution: **spatial diversity across parallel agents** (many agents in different states simultaneously) replaces **temporal diversity from a replay buffer** (one agent's historical states reshuffled). The key requirement is that the agents actually explore different parts of the environment — which is enforced by Principle 3. **Principle 2: Accumulating gradients over multiple timesteps before applying them.** Rather than updating the shared model after every single environment step, each thread accumulates gradients over `$t_{\text{max}}$` steps (or until episode termination) and then applies the accumulated gradient as a single update. The paper sets `$t_{\text{max}} = 5$` and `$I_{\text{AsyncUpdate}} = 5$` for all Atari experiments, meaning updates happen every 5 steps. This serves two purposes. First, it provides a trade-off between computational and data efficiency: applying updates every 5 steps means fewer synchronization points and less overhead from the gradient application step, without waiting so long that the agent is acting on extremely stale parameters. Second, it reduces the chances of multiple threads overwriting each other's updates by reducing the frequency of writes to shared memory. The paper explicitly notes that accumulating updates "reduces the chances of multiple actor learners overwriting each other's updates" (Section 4, one-step Q-learning description). **Principle 3: Different exploration policies per thread to maximize diversity.** Each thread is given a different exploration policy to ensure the parallel agents actually explore different parts of the environment rather than all converging to the same behavior. For value-based methods (Q-learning, Sarsa), the paper uses **$\epsilon$-greedy exploration with thread-specific `$\epsilon$` values**: > "We experiment with using `$\epsilon$`-greedy exploration with `$\epsilon$` periodically sampled from some distribution by each thread." Specifically, the value-based methods sample the exploration rate `$\epsilon$` from a discrete distribution taking three values `$\epsilon_1, \epsilon_2, \epsilon_3$` with probabilities 0.4, 0.3, 0.3 respectively. The values `$\epsilon_1, \epsilon_2, \epsilon_3$` are annealed from 1 to 0.1, 0.01, 0.5 respectively over the first four million frames (Supplementary Section 8). This means at any point in training, some threads are exploring aggressively (high `$\epsilon$`), some moderately, and some exploiting more — creating a diverse mix of behaviors that maximizes the decorrelation effect. For the policy-based A3C method, exploration diversity comes naturally from the stochastic policy — the policy network outputs a probability distribution over actions, and actions are sampled from this distribution. The entropy regularization term (described in the A3C subsection below) prevents the policy from collapsing to a deterministic strategy too quickly, maintaining exploration. Different threads sample different actions even from the same policy due to the stochasticity, and the policy evolves differently across threads because each thread sees different trajectories. --- #### Asynchronous One-Step Q-Learning **Algorithm location and structure.** Pseudocode appears in Algorithm 1 of the main paper. This is the simplest of the four methods and serves as a direct comparison point against DQN: it uses the same Q-learning update rule as DQN but replaces the experience replay buffer with asynchronous parallel actors and Hogwild! updates. **Neural network architecture.** The Q-network has the same architecture as Mnih et al. (2013): a convolutional layer with 16 filters of size `$8 \times 8$` with stride 4, followed by a convolutional layer with 32 filters of size `$4 \times 4$` with stride 2, followed by a fully connected layer with 256 hidden units. All three hidden layers use rectifier (ReLU) nonlinearities. The output layer has one linear unit per action representing the action-value `$Q(s, a; \theta)$`. Input preprocessing is identical to Mnih et al. (2015): frames are converted to grayscale, resized to `$84 \times 84$`, and the agent sees a stack of the 4 most recent frames. An action repeat of 4 is used (the agent selects an action every 4 frames, with the selected action repeated for the intervening frames). **Target network.** Following DQN, the algorithm maintains two sets of Q-network parameters: the online parameters `$\theta$` that are updated asynchronously by all threads, and a separate set of **target network parameters** `$\theta^-$` that are used to compute the Q-learning target values. The target network is updated to match the online network every `$I_{\text{target}} = 40000$` frames. This is a shared target network — there is only one copy of `$\theta^-$`, updated periodically from the shared `$\theta$` by whichever thread happens to reach the update interval. The target network stabilizes learning by providing a fixed target for the Q-learning updates over short timescales, breaking the harmful feedback loop where the online network chases its own moving predictions. **Update rule.** At each environment step, the thread computes the one-step Q-learning target: $$y = \begin{cases} r & \text{for terminal } s' \\ r + \gamma \max_{a'} Q(s', a'; \theta^-) & \text{for non-terminal } s' \end{cases}$$ where `$r$` is the reward received, `$s'$` is the next state, `$\gamma = 0.99$` is the discount factor, `$\theta^-$` are the target network parameters, and `$\max_{a'} Q(s', a'; \theta^-)$` is the maximum predicted Q-value over all actions in state `$s'$` according to the target network. **What it computes:** For a terminal state (episode end), the target is just the immediate reward — there is no future value to consider. For a non-terminal state, the target is the immediate reward plus the discounted estimate of the best possible future value. The key design choice is that the "best possible future value" is computed using the **target network** `$\theta^-$` rather than the online network `$\theta$`. This makes the target partially decoupled from the parameters being optimized. **Why this form:** Using `$\max_{a'} Q(s', a'; \theta)$` (the online network for both selection and evaluation) would create a positive feedback loop: if the network overestimates some action's value, that overestimated value becomes the target, which pushes the network to overestimate even more in the next update. The target network breaks this loop because `$\theta^-$` is fixed for many thousands of updates, providing a stable regression target while `$\theta$` catches up. This is exactly the same mechanism as DQN, transplanted into the asynchronous setting. **Loss function.** The gradient accumulated for each state-action pair is with respect to the squared error: $$\frac{\partial (y - Q(s, a; \theta))^2}{\partial \theta}$$ This is the gradient of the mean squared error (MSE) between the target `$y$` and the current Q-value prediction. The thread accumulates these gradients over `$t_{\text{max}} = 5$` steps (or fewer if the episode terminates) and then applies the accumulated gradient to the shared parameters `$\theta$` asynchronously. **Why MSE rather than Huber loss:** DQN originally used a Huber loss (squared error for small errors, absolute error for large errors) to reduce sensitivity to outliers. This paper uses plain MSE, likely because the gradient clipping used in later experiments (for A3C) provides similar robustness, and MSE is simpler to implement in the asynchronous setting. The paper does not explicitly discuss this choice. **Thread-local state.** Each thread maintains: - `$t$`: a thread-local step counter tracking how many steps since the last gradient update. - `$d\theta$`: a thread-local gradient accumulator, initialized to zero and incremented with each step's gradient. - `$s$`: the current state of the thread's environment instance. - Its own exploration policy (`$\epsilon$` value sampled from the distribution described above). **Update timing and control flow (Algorithm 1, Section 4).** The thread runs in a loop: 1. Initialize thread step counter `$t \leftarrow 0$`, gradient accumulator `$d\theta \leftarrow 0$`, and synchronize target network `$\theta^- \leftarrow \theta$`. 2. Get initial state `$s$` from the environment. 3. **Take action:** select `$a$` using `$\epsilon$`-greedy policy based on `$Q(s, a; \theta)$` — with probability `$\epsilon$`, pick a random action; otherwise, pick `$\arg\max_a Q(s, a; \theta)$`. 4. **Receive transition:** observe new state `$s'$` and reward `$r$`. 5. **Compute target `$y$`** using the formula above (terminal vs. non-terminal branch). 6. **Accumulate gradient:** `$d\theta \leftarrow d\theta + \frac{\partial (y - Q(s,a;\theta))^2}{\partial \theta}$`. 7. Set `$s \leftarrow s'$`, increment global counter `$T$` and thread counter `$t$`. 8. **Periodic target network update:** if `$T \mod I_{\text{target}} == 0$`, set `$\theta^- \leftarrow \theta$`. 9. **Periodic asynchronous update:** if `$t \mod I_{\text{AsyncUpdate}} == 0$` or if `$s$` is terminal, perform the asynchronous update of `$\theta$` using accumulated `$d\theta$`, then clear `$d\theta \leftarrow 0$`. 10. Repeat until global counter `$T > T_{\text{max}}$`. **Why the target network is updated based on the global counter `$T$` rather than per-thread steps.** The global counter `$T$` counts total frames across all threads. Updating the target network every 40,000 *global* frames means the target network is refreshed after every 40,000 environment interactions total, regardless of which thread generated them. With 16 threads, this means roughly every 2,500 frames per thread on average. Using a per-thread counter would mean the target network updates 16 times less frequently (or 16 times more frequently if each thread independently maintained its own target network — which is not what the paper does; the target network is shared). --- #### Asynchronous One-Step Sarsa **Algorithm location.** Described briefly in Section 4, with no separate pseudocode — the paper states it "is the same as asynchronous one-step Q-learning as given in Algorithm 1 except that it uses a different target value for `$Q(s, a)$`." **The critical difference: target value computation.** The target for one-step Sarsa is: $$y = r + \gamma Q(s', a'; \theta^-)$$ where `$a'$` is **the action actually taken** in state `$s'$` by the agent's exploration policy, not the maximum-Q action. Compare this with Q-learning's target `$r + \gamma \max_{a'} Q(s', a'; \theta^-)$`. The difference is small in notation but profound in implications. **What it computes:** Sarsa evaluates the policy the agent is actually following, not the optimal policy. If the agent chose a suboptimal action `$a'$` due to `$\epsilon$`-greedy exploration, the target reflects the value of that suboptimal action, not the best possible action. This means Sarsa learns the Q-function for the **behavior policy** (what the agent actually does, including exploration), whereas Q-learning learns the Q-function for the **optimal policy** regardless of what actions the agent actually takes. **Why this form:** Sarsa is an **on-policy** algorithm — its update target depends on the action actually selected by the current policy, so it requires data generated by the current policy. This is exactly why Sarsa could not work with experience replay: if you stored transitions in a buffer and replayed them later, the stored action `$a'$` would have been selected by an old policy, and using it as the target for the current Q-function would introduce off-policy bias. The asynchronous framework eliminates this problem because each thread generates data online from its current policy and immediately uses it for updates — there is no replay buffer, so the data is always on-policy by construction. **Why Sarsa matters.** Sarsa tends to learn safer policies than Q-learning in environments where exploration mistakes are costly, because it accounts for the fact that the agent will continue exploring (and thus occasionally make suboptimal choices) rather than assuming it will always act optimally after the current step. In the Atari domain with `$\epsilon$`-greedy exploration, this difference is less pronounced because `$\epsilon$` is annealed to small values, but the paper's demonstration that Sarsa trains stably and achieves competitive scores (Figure 1) is significant because it was previously unknown how to make on-policy deep Sarsa work at all. **All other details identical to one-step Q-learning.** The network architecture, target network mechanism (shared, updated every 40,000 global frames), gradient accumulation (`$t_{\text{max}} = 5$`), Shared RMSProp optimizer, exploration diversity (different `$\epsilon$` per thread), and Hogwild! update protocol are all identical. --- #### Asynchronous N-Step Q-Learning **Algorithm location.** Pseudocode appears in Supplementary Algorithm S2. This is a more sophisticated value-based method that uses multi-step returns to propagate rewards faster through the Q-function. **The forward-view n-step return.** Standard one-step Q-learning only directly updates the value of the state-action pair that immediately preceded a reward. N-step Q-learning uses **n-step returns** to update `$n$` preceding state-action pairs per reward. The n-step return is: $$R_t^{(n)} = r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \cdots + \gamma^{n-1} r_{t+n-1} + \gamma^n \max_a Q(s_{t+n}, a; \theta^-)$$ where `$r_t, r_{t+1}, \ldots, r_{t+n-1}$` are the `$n$` rewards actually received in the environment, `$\gamma$` is the discount factor, and the final term bootstraps from the target network's Q-value at the `$n$`-step horizon. **What it computes:** Instead of waiting for the Q-value at `$s'$` to propagate the reward backward one step at a time through many future updates, the n-step return directly injects the reward `$r_t$` into the Q-values of the `$n$` state-action pairs `$(s_t, a_t), (s_{t+1}, a_{t+1}), \ldots, (s_{t+n-1}, a_{t+n-1})$` in a single update. For example, with `$n = 5$`, receiving a reward at step `$t$` immediately updates the Q-values for the five actions that preceded it. **Why this form:** This accelerates credit assignment in sparse-reward environments, where the agent might take dozens of actions before receiving any feedback. One-step Q-learning would require dozens of Bellman backup iterations to propagate that reward back to the early actions; n-step returns do it in a single update. The paper notes that this "makes the process of propagating rewards to relevant state-action pairs potentially much more efficient." **The forward-view computation protocol (Supplementary Algorithm S2).** Unlike eligibility traces (the "backward view"), which maintain a running trace of recently visited states and decay them over time, this algorithm explicitly computes n-step returns in the "forward view": 1. **Experience collection phase:** The thread interacts with the environment using its `$\epsilon$`-greedy policy for up to `$t_{\text{max}}$` steps (set to 5 in experiments) or until a terminal state is reached. It stores the sequence of states, actions, and rewards encountered: `$(s_{t_{\text{start}}}, a_{t_{\text{start}}}, r_{t_{\text{start}}}), \ldots, (s_{t-1}, a_{t-1}, r_{t-1}), s_t$`. 2. **Bootstrap value computation:** At the end of the sequence, compute the bootstrap value `$R$`: - If `$s_t$` is terminal: `$R = 0$` (no future value after episode end). - If `$s_t$` is non-terminal: `$R = \max_a Q(s_t, a; \theta^-)$` (value estimate from target network). This `$R$` represents the estimated value from the end of the collected trajectory onward. 3. **Backward n-step return computation:** Iterate backward through the collected steps `$i = t-1, t-2, \ldots, t_{\text{start}}$`, updating `$R$` as: $$R \leftarrow r_i + \gamma R$$ At each step `$i$`, after updating `$R$`, the value `$R$` is the n-step return for `$(s_i, a_i)$` where `$n = t - i$`. The first step in the sequence gets the longest return (up to `$t_{\text{max}}$`-step), the last step gets a one-step return. For example, if `$t_{\text{max}} = 5$` and the sequence reaches step `$t$` without terminating: - At `$i = t-1$`: `$R \leftarrow r_{t-1} + \gamma \max_a Q(s_t, a; \theta^-)$` (one-step return for `$(s_{t-1}, a_{t-1})$`). - At `$i = t-2$`: `$R \leftarrow r_{t-2} + \gamma(r_{t-1} + \gamma \max_a Q(s_t, a; \theta^-))$` (two-step return for `$(s_{t-2}, a_{t-2})$`). - Continuing back to `$i = t_{\text{start}}$`, which gets the full `$t_{\text{max}}$`-step return. 4. **Gradient accumulation and update:** For each `$i$` in the collected sequence, accumulate the gradient: $$d\theta \leftarrow d\theta + \frac{\partial (R_i - Q(s_i, a_i; \theta'))^2}{\partial \theta'}$$ where `$R_i$` is the computed n-step return for step `$i$`. After processing all steps in the sequence, perform a single asynchronous update of the shared `$\theta$` using the accumulated `$d\theta$`. **Why forward view rather than backward view (eligibility traces).** The paper explains this design choice explicitly: > "We found that using the forward view is easier when training neural networks with momentum-based methods and backpropagation through time." Eligibility traces maintain a decaying memory of recently visited state-action pairs and apply TD errors to all of them simultaneously. This requires storing an eligibility value per parameter (or at least per recently active feature), which is memory-intensive for large neural networks and interacts poorly with adaptive optimizers like RMSProp that maintain per-parameter statistics. The forward view explicitly computes the n-step returns from the stored trajectory and applies them as explicit supervised-learning-style targets, which integrates cleanly with standard deep learning optimizers and automatic differentiation — the gradient computation `$\partial (R_i - Q(s_i, a_i; \theta'))^2 / \partial \theta'$` is just a standard regression gradient. **Thread-local parameters `$\theta'$`.** Note that in Supplementary Algorithm S2, the thread synchronizes its local copy `$\theta' \leftarrow \theta$` at the start of each update cycle, computes all gradients with respect to this local copy, and then applies the accumulated `$d\theta$` to the shared `$\theta$`. This prevents the thread's gradient computation from being affected by other threads' concurrent updates to the shared parameters mid-sequence, ensuring the n-step returns are computed consistently with respect to a single parameter snapshot. **On-policy vs. off-policy nature of n-step Q-learning.** N-step Q-learning as implemented here is partially off-policy. The n-step return uses the actual rewards `$r_i$` from the environment, which depend on the actions the agent actually took (the behavior policy). However, it bootstraps from `$\max_a Q(s_t, a; \theta^-)$`, which is the optimal action value, not the value of the action the agent would take. This creates a hybrid: the multi-step reward sequence is on-policy (it depends on the behavior policy's actions), but the bootstrap value is off-policy (it assumes optimal behavior after the n-step horizon). This is sometimes called "n-step Q-learning with off-policy corrections" or more precisely, it uses the Peng's Q(`$\lambda$`) formulation (Peng & Williams, 1996) without importance sampling corrections. The paper acknowledges this implicitly by citing Peng & Williams (1996) in the context of n-step returns. **Target network usage.** The target network `$\theta^-$` is shared across all threads and updated every `$I_{\text{target}} = 40000$` global frames, same as one-step Q-learning. The bootstrap value `$\max_a Q(s_t, a; \theta^-)$` uses the target network to reduce the moving-target problem, and the Q-values being updated use the online parameters `$\theta'$` (the thread's local synchronized copy). --- #### Asynchronous Advantage Actor-Critic (A3C) **Algorithm location.** Pseudocode appears in Supplementary Algorithm S3. This is the highest-performing method in the paper and represents the most complete instantiation of the asynchronous framework — it combines a policy network (the actor), a value network (the critic), n-step returns, advantage estimation, and entropy regularization into a single on-policy algorithm. **Why actor-critic is the natural beneficiary of asynchronous parallelism.** Actor-critic methods are fundamentally on-policy: the policy gradient `$\nabla_\theta \log \pi(a|s; \theta) A(s, a)$` is an expectation over actions sampled from the current policy `$\pi_\theta$`. Using off-policy data (actions from an old policy) requires importance sampling corrections that can have high variance, making training unstable. The asynchronous framework's online, on-policy data generation — each thread generates its own data from its current copy of the policy — provides exactly what actor-critic needs. The paper frames this as one of the key motivations: > "Since we no longer rely on experience replay for stabilizing learning we are able to use on-policy reinforcement learning methods such as Sarsa and actor-critic to train neural networks in a stable way." **Neural network architecture (dual-output).** The A3C network has the same convolutional and fully-connected backbone as the value-based methods (16 filters `$8 \times 8$` stride 4, 32 filters `$4 \times 4$` stride 2, 256-unit FC layer, all ReLU), but with two output heads sharing the convolutional and FC layers: - **Policy head:** A softmax output with one entry per action, representing `$\pi(a_t | s_t; \theta)$` — the probability of selecting each action given the current state. The softmax ensures outputs are non-negative and sum to 1. - **Value head:** A single linear output representing `$V(s_t; \theta_v)$` — the estimated value of the current state (expected cumulative discounted reward from this state onward under the current policy). The parameters `$\theta$` (policy parameters) and `$\theta_v$` (value function parameters) are treated as separate for generality in the pseudocode, but in practice the paper always shares the non-output layers (convolutional layers and fully-connected layer) between the policy and value heads. This means `$\theta$` and `$\theta_v$` overlap in the shared layers and diverge only in the output-layer weights. Sharing the representation is beneficial because the features useful for valuing states (recognizing game objects, understanding game dynamics) are also useful for selecting actions, and training both heads jointly from the shared representation provides a form of multi-task learning that can improve generalization. **LSTM variant.** For the recurrent agent evaluated in Table 1, an additional 256 LSTM cells are inserted after the final hidden (FC) layer, before the policy and value outputs. This allows the agent to maintain an internal memory across time steps, which is useful for partially observable environments where the current frame does not contain all information needed for optimal decisions (e.g., games where the ball's velocity matters and cannot be inferred from a single frame). The LSTM is trained via backpropagation through time, using the same forward-view n-step unrolling as the feedforward agent. **The advantage function and n-step advantage estimation.** The core learning signal for A3C is the **advantage function** `$A(s_t, a_t)$`, which measures how much better (or worse) action `$a_t$` is compared to the average action in state `$s_t$`: $$A(s_t, a_t) = Q(s_t, a_t) - V(s_t)$$ Since the agent does not have access to the true Q-function or value function, it uses an n-step **advantage estimate** computed from the trajectory: $$A(s_t, a_t; \theta, \theta_v) = \sum_{i=0}^{k-1} \gamma^i r_{t+i} + \gamma^k V(s_{t+k}; \theta_v) - V(s_t; \theta_v)$$ where the sum `$\sum_{i=0}^{k-1} \gamma^i r_{t+i}$` is the discounted sum of `$k$` actual rewards received from step `$t$` to step `$t+k-1$`, the term `$\gamma^k V(s_{t+k}; \theta_v)$` is the bootstrapped value estimate at the `$k$`-step horizon, and `$V(s_t; \theta_v)$` is the current value estimate for the starting state. The integer `$k$` varies from state to state and is upper-bounded by `$t_{\text{max}}$` (which is set to 5). Specifically, for a sequence collected from `$t_{\text{start}}$` to `$t$`, the advantage estimate for step `$i$` uses `$k = t - i$`, meaning earlier steps in the sequence get longer-horizon advantage estimates. **What it computes:** `$\sum_{i=0}^{k-1} \gamma^i r_{t+i} + \gamma^k V(s_{t+k}; \theta_v)$` is an estimate of `$Q(s_t, a_t)$` — the expected return from taking action `$a_t$` and then following the current policy. It combines `$k$` steps of actually experienced rewards with a value function bootstrap for the remaining future. Subtracting `$V(s_t; \theta_v)$` yields the advantage: a positive advantage means the action led to better-than-expected outcomes; a negative advantage means worse-than-expected. This centered signal has lower variance than using the raw return `$R_t$` alone because `$V(s_t)$` absorbs the component of the return that is predictable from the state regardless of which action was taken. **Why this form:** Using the advantage rather than the raw return (as in REINFORCE) is a variance reduction technique. The raw return `$R_t = \sum_{i=0}^{\infty} \gamma^i r_{t+i}$` has high variance because it depends on all future stochastic events — lucky breaks and unlucky accidents far in the future both affect the update. The baseline `$V(s_t)$` subtracts out the expected value given the state, so the policy gradient update is driven only by whether the action performed better or worse than expected, not by whether the agent happened to be in a good or bad state. The paper explains: > "A learned estimate of the value function is commonly used as the baseline `$b_t(s_t) \approx V^\pi(s_t)$` leading to a much lower variance estimate of the policy gradient." The n-step formulation further reduces variance compared to Monte Carlo returns by bootstrapping from `$V(s_{t+k})$` after `$k$` steps rather than summing rewards all the way to the end of the episode. This truncates the variance accumulation from future stochasticity while still providing multi-step credit assignment. **Policy gradient (actor update).** The policy parameters `$\theta$` are updated in the direction that increases the log-probability of actions with positive advantage and decreases the log-probability of actions with negative advantage, scaled by the magnitude of the advantage: $$\nabla_\theta J(\theta) = \nabla_\theta \log \pi(a_t | s_t; \theta) \cdot A(s_t, a_t; \theta, \theta_v)$$ **What it computes:** For each action taken in the trajectory, compute the gradient of the log-probability of that action under the current policy, and multiply it by the advantage estimate. If the advantage is positive (good action), the policy parameters move to make that action more likely in similar states. If the advantage is negative (bad action), the parameters move to make it less likely. The magnitude of the update is proportional to `$|A|$`, so surprisingly good or bad outcomes cause larger policy adjustments. **Why this form:** This is the standard policy gradient theorem (Sutton & Barto, 1998), with the advantage substituted for the return as a baseline. The gradient `$\nabla_\theta \log \pi(a|s; \theta)$` is the direction in parameter space that most rapidly increases the probability of action `$a$` in state `$s$`. Multiplying by the advantage ensures that the probability increases for good actions and decreases for bad ones, and the advantage magnitude determines how strongly to adjust. The expected value of this gradient (averaged over actions sampled from `$\pi$`) is the gradient of the expected return `$\mathbb{E}[R_t]$`, so this is a stochastic gradient ascent on the RL objective. **Value function update (critic update).** The value function parameters `$\theta_v$` are updated to minimize the squared error between the estimated value `$V(s_i; \theta_v)$` and the n-step return: $$\text{Loss}_{\text{value}} = (R_i - V(s_i; \theta_v))^2$$ where `$R_i = \sum_{j=0}^{k-1} \gamma^j r_{i+j} + \gamma^k V(s_{i+k}; \theta_v)$` is the same n-step return used in the advantage estimate (but without subtracting the baseline, since we want the value function to predict the full return, not the advantage). The gradient: $$\frac{\partial (R_i - V(s_i; \theta_v))^2}{\partial \theta_v}$$ is accumulated alongside the policy gradient and applied simultaneously. **What it computes:** This is standard TD learning for the value function: update `$V(s_i)$` to be closer to the n-step return `$R_i$`, which is a better (lower-variance, more up-to-date) estimate of the true expected return than the current `$V(s_i)$`. Since `$R_i$` includes actually experienced rewards, it incorporates new information from the environment. **Why this form:** The value function serves as both a baseline for the policy gradient (reducing variance) and as a bootstrap for computing n-step returns. Training it with the same n-step returns used for the policy ensures consistency — both the actor and critic are learning from the same multi-step reward signal. Using squared error is standard for regression targets. **Entropy regularization.** The objective function includes an additional term that encourages the policy to maintain entropy (randomness), preventing premature convergence to a deterministic policy: $$\text{Entropy bonus} = \beta \nabla_\theta H(\pi(s_t; \theta))$$ where `$H(\pi(s_t; \theta)) = -\sum_a \pi(a|s_t; \theta) \log \pi(a|s_t; \theta)$` is the entropy of the policy distribution in state `$s_t$`, and `$\beta$` is a hyperparameter controlling the strength of regularization. In all Atari and TORCS experiments, `$\beta = 0.01$`. **What it computes:** The entropy `$H$` measures how spread out the action probabilities are — a uniform distribution (all actions equally likely) has maximum entropy `$\log(|\mathcal{A}|)$`; a deterministic policy (one action with probability 1) has zero entropy. The gradient `$\nabla_\theta H$` points in the direction that increases entropy. Adding `$\beta \nabla_\theta H$` to the policy gradient pushes the parameters toward more stochastic policies. **Why this form:** Without entropy regularization, the policy can collapse to a deterministic strategy too early in training. Once the policy assigns near-100% probability to a particular action in a state, it stops exploring that state — the policy gradient for other actions is multiplied by near-zero probabilities, so they receive essentially no updates. If the deterministic choice is suboptimal (which is likely early in training), the agent gets stuck. The entropy bonus counteracts this by rewarding the policy for maintaining uncertainty. The paper credits Williams & Peng (1991) for this technique, noting: > "We also found that adding the entropy of the policy `$\pi$` to the objective function improved exploration by discouraging premature convergence to suboptimal deterministic policies. This technique was originally proposed by (Williams & Peng, 1991), who found that it was particularly helpful on tasks requiring hierarchical behavior." The entropy bonus also provides a more principled exploration mechanism than `$\epsilon$`-greedy for policy-based methods. Rather than adding uniform random noise regardless of the value landscape, the entropy bonus encourages the policy to maintain high entropy specifically in states where it is uncertain, while allowing it to become deterministic in states where one action is clearly superior. **Complete A3C update procedure (Supplementary Algorithm S3).** Each thread executes the following loop: 1. **Synchronize:** Set thread-local parameters `$\theta' \leftarrow \theta$` and `$\theta'_v \leftarrow \theta_v$` (copy the shared global parameters). Reset gradient accumulators `$d\theta \leftarrow 0$`, `$d\theta_v \leftarrow 0$`. 2. **Experience collection:** Starting from state `$s_{t_{\text{start}}}$`, interact with the environment for up to `$t_{\text{max}}$` steps (set to 5) or until a terminal state is reached. At each step `$t$`: - Sample action `$a_t \sim \pi(a_t | s_t; \theta')$` (stochastic policy — exploration is driven by the policy's entropy, not external `$\epsilon$`-greedy). - Receive reward `$r_t$` and next state `$s_{t+1}$`. - Store the transition. 3. **Bootstrap value:** At the end of the trajectory (step `$t$`, which is either terminal or `$t_{\text{start}} + t_{\text{max}}$`): - If terminal: `$R \leftarrow 0$`. - If non-terminal: `$R \leftarrow V(s_t; \theta'_v)$` (bootstrap from the value function). 4. **Backward accumulation of policy and value gradients:** For `$i = t-1, t-2, \ldots, t_{\text{start}}$`: - `$R \leftarrow r_i + \gamma R$` (n-step return for step `$i$`). - Accumulate policy gradient: $$d\theta \leftarrow d\theta + \nabla_{\theta'} \log \pi(a_i | s_i; \theta') \cdot (R - V(s_i; \theta'_v))$$ where `$R - V(s_i; \theta'_v)$` is the advantage estimate. - Accumulate value gradient: $$d\theta_v \leftarrow d\theta_v + \frac{\partial (R - V(s_i; \theta'_v))^2}{\partial \theta'_v}$$ 5. **Apply entropy gradient:** Add the entropy regularization gradient `$\beta \nabla_{\theta'} H(\pi(s_i; \theta'))$` to `$d\theta$`. This is typically done during the backward loop, accumulated per-step. 6. **Asynchronous update:** Perform lock-free update of shared `$\theta$` using `$d\theta$` and shared `$\theta_v$` using `$d\theta_v$`. 7. Repeat until global counter `$T > T_{\text{max}}$`. **Key differences from the value-based methods.** A3C has no target network (there is no `$\theta^-$`), no experience replay buffer, and no `$\epsilon$`-greedy exploration. Stabilization comes entirely from the asynchronous parallelism, the use of a value function baseline (which reduces gradient variance), and entropy regularization (which prevents policy collapse). The lack of a target network is possible because the value function bootstrap `$V(s_{t+k}; \theta_v)$` uses the same parameters being updated, but the n-step truncation and the asynchronous nature of updates mean the value estimates don't need the same kind of decoupling that Q-learning's max operator requires. **Continuous action extension (MuJoCo experiments, Supplementary Section 9).** For continuous control tasks, the policy network outputs the parameters of a multivariate normal distribution with a spherical (diagonal) covariance. Specifically: - The network outputs two vectors: a mean vector `$\mu$` and a scalar variance `$\sigma^2$`. - `$\mu$` is produced by a linear layer. - `$\sigma^2$` is produced by a SoftPlus operation `$\log(1 + \exp(x))$` applied to the output of a linear layer, ensuring positivity. - Actions are sampled as `$a \sim \mathcal{N}(\mu, \sigma^2 I)$` during experience collection. The policy and value networks do not share parameters for the continuous control experiments (unlike Atari, where they share convolutional and FC layers). This is noted as "unlikely to be crucial" by the authors. For the low-dimensional physical state input case (joint positions and velocities), the state is mapped through a single hidden layer of 200 ReLU units before being fed to an LSTM layer of 128 cells, then to the policy and value heads. For pixel-based input cases (pendulum, pointmass2D, gripper), the input is passed through two spatial convolution layers (without nonlinearity or pooling) and then to the 128-cell LSTM. The entropy regularization for continuous actions uses the differential entropy of the normal distribution: `$-\frac{1}{2}(\log(2\pi\sigma^2) + 1)$`, with a weight of `$10^{-4}$` across all tasks (much smaller than the Atari `$\beta = 0.01$`, reflecting the different scale of entropy for continuous vs. discrete distributions). Since episodes in MuJoCo are typically at most a few hundred time steps, the paper does not use bootstrapping — each episode is batched into a single update using the full Monte Carlo return (no truncation at `$t_{\text{max}}$`). This is mentioned briefly: "since the episodes were typically at most several hundred time steps long, we did not use any bootstrapping in the policy or value function updates and batched each episode into a single update." **Hyperparameter summary for A3C (all domains).** - `$\gamma = 0.99$` (discount factor, all experiments). - `$t_{\text{max}} = 5$` (Atari and TORCS), full episode for MuJoCo. - `$I_{\text{AsyncUpdate}} = 5$` for value-based methods; A3C updates after every `$t_{\text{max}}$` sequence. - Shared RMSProp optimizer with `$\alpha = 0.99$`. - Entropy weight `$\beta = 0.01$` for Atari and TORCS, `$10^{-4}$` for MuJoCo continuous control. - Learning rate sampled from `$\text{LogUniform}(10^{-4}, 10^{-2})$` for hyperparameter search, annealed to zero. Fixed learning rate (tuned per-game) for the final 57-game evaluation. - Gradient norm clipping: the paper mentions clipping in the context of the 57-game evaluation (Section 5.1: "tuned hyperparameters (learning rate and amount of gradient norm clipping) using a search on six Atari games"), but exact clipping thresholds are not provided in the main text. --- #### Summary of Design Choices and Their Justifications - **Asynchronous parallelism over experience replay:** Decorrelates training data through spatial diversity (16 agents in different states simultaneously) rather than temporal diversity (reshuffling stored transitions), enabling on-policy methods and reducing memory requirements. The empirical evidence that this works for deep neural networks is the paper's primary contribution. - **Hogwild! lock-free updates over parameter-server architectures:** Maximizes throughput on a single multi-core CPU by avoiding locking overhead. The theoretical convergence guarantees from prior work (Tsitsiklis, 1994; Recht et al., 2011) provide justification, and the empirical results validate that the approach works in practice for RL. - **Shared RMSProp over per-thread RMSProp or Momentum SGD:** The shared gradient statistics provide an additional stabilization mechanism by aggregating gradient scale information across diverse threads, complementing the decorrelation provided by parallel actors. This was selected through empirical comparison (Figure S5). - **Gradient accumulation over `$t_{\text{max}} = 5$` steps:** Balances computational efficiency (fewer synchronization points) against the staleness of parameters (the policy doesn't drift too far between updates). Also reduces write conflicts in the Hogwild! setting. - **Thread-specific exploration (`$\epsilon$`-greedy with different `$\epsilon$` values for value-based methods; entropy regularization for policy-based):** Ensures that parallel agents actually explore different behaviors rather than converging to the same policy, maximizing the decorrelation benefit of parallelism. - **Shared target network updated every 40,000 global frames (value-based methods only):** Decouples the Q-learning targets from the rapidly-updating online parameters, preventing the moving-target instability that Q-learning is prone to. Using a global update counter ensures all threads use the same target network at any given time. - **Forward-view n-step returns over backward-view eligibility traces:** Integrates cleanly with momentum-based optimizers and automatic differentiation — the explicit n-step returns become standard regression targets that any deep learning optimizer can handle. This is a pragmatic choice driven by implementation simplicity with neural networks. - **Entropy regularization (A3C) over `$\epsilon$`-greedy for policy-based exploration:** Provides a gradient-based exploration incentive that is more principled for stochastic policies, encouraging the policy to maintain uncertainty in proportion to the value landscape rather than adding uniform noise. - **Shared convolutional and FC layers between policy and value networks (A3C):** Multi-task learning from a shared representation improves generalization and reduces parameter count. The policy and value functions benefit from the same visual features. - **Dual-output architecture (softmax policy head + linear value head) for A3C:** Clean separation of concerns — the policy head produces action probabilities, the value head produces scalar state values. Both are trained from the same trajectory data with different loss functions, computed simultaneously during the backward pass. ## 4. Key Insights and Innovations ### Innovation 1: Parallelism IS the Stabilizer — A Substitute, Not an Accelerator This paper's deepest conceptual move is reframing parallelism from a computational speedup technique into a **stabilization mechanism** that serves the same fundamental role as experience replay. This is not an incremental engineering improvement — it is a category shift in how we think about what makes deep RL training work. Before this paper, the dominant assumption in deep RL was clear: experience replay was *necessary* for stability when using deep neural networks as function approximators. DQN (Mnih et al., 2015) demonstrated this, and every subsequent improvement — Double DQN, Dueling DQN, Prioritized Replay — accepted the replay buffer as a non-negotiable foundation. Even Gorila (Nair et al., 2015), which distributed DQN across 130 machines, preserved experience replay within each actor. The reasoning was intuitive: online RL generates strongly correlated, non-stationary data, and neural networks trained on such data diverge. Experience replay breaks the correlation by shuffling and resampling. The field had converged on a causal chain: **stability requires decorrelation → decorrelation requires a replay buffer → replay buffers require off-policy algorithms**. This paper breaks that chain at the second link. It demonstrates that decorrelation can be achieved through an entirely different mechanism: **spatial diversity across parallel actors**. At any wall-clock instant, 16 agents running in separate environment instances occupy 16 different states, follow 16 different exploration trajectories, and compute gradients from 16 decorrelated data points — even though each individual agent's data stream remains temporally correlated. The aggregate gradient applied to the shared model is therefore computed from data as diverse as what you'd get by sampling randomly from a replay buffer, but without storing, indexing, or resampling anything. The significance of this reframing extends far beyond the specific algorithms in the paper. It reveals that experience replay is not a unique solution to the stability problem but rather one instance of a broader principle: **training data must be sufficiently decorrelated for neural network optimization to work in RL**. The replay buffer achieves this through temporal diversity (mixing old and new data). Parallelism achieves it through spatial diversity (mixing data from different environment instances). Understanding this equivalence opens the door to other stabilization mechanisms not yet explored — and critically, it explains *why* parallelism works, not just that it works. The paper speculates on this mechanism explicitly: > "By running different exploration policies in different threads, the overall changes being made to the parameters by multiple actor-learners applying online updates in parallel are likely to be less correlated in time than a single agent applying online updates." This is a **diagnostic insight**, not just an empirical observation. It tells us what problem experience replay was solving at a functional level (decorrelation), which means we can now evaluate any proposed alternative by asking whether it achieves sufficient decorrelation, rather than testing it ad hoc. What makes this a fundamental rather than incremental contribution is the **unlocking effect** it has on the algorithm design space. By removing the replay buffer requirement, the framework makes possible what was previously impossible: stable training of on-policy algorithms (Sarsa, actor-critic) with deep neural networks on challenging domains. This is not a small refinement — it expands the set of usable RL algorithms from "off-policy value-based methods that happen to work with replay buffers" to "the entire RL toolkit, value-based and policy-based, on-policy and off-policy." The paper demonstrates four algorithms spanning this space, all training successfully on Atari (Figure 1), which would not have been possible under the replay-buffer paradigm. The evidence for this reframing is primarily the success of one-step Sarsa and A3C — both on-policy methods that fundamentally cannot work with experience replay (their update targets depend on the current policy's actions, not historical actions from old policies). That these methods achieve competitive or superior performance to DQN (Figure 1, Table 1) is direct evidence that parallelism alone provides sufficient stabilization. The fact that the asynchronous one-step Q-learning variant — which is architecturally identical to DQN except for using parallelism instead of a replay buffer — also trains successfully (Figure 1) confirms that the parallelism is doing the same functional work as the replay buffer. ### Innovation 2: Hogwild!-Style Async Updates Are Not Just a Performance Hack — They Are a Stability Mechanism A subtler but equally important insight emerges from the paper's optimizer comparison (Supplementary Figure S5): the choice of **Shared RMSProp** as the optimization algorithm is not merely about training speed — it provides an additional layer of stabilization through cross-thread gradient statistics sharing that complements the data decorrelation from parallel actors. The field's default assumption, inherited from supervised deep learning, was that optimizer choice was primarily about convergence speed and hyperparameter robustness. The paper's comparison of three optimizers (Momentum SGD, per-thread RMSProp, Shared RMSProp) on two algorithms across four games reveals something more profound: in the asynchronous RL setting, the optimizer's internal state becomes part of the stabilization infrastructure. Shared RMSProp, where the moving average of squared gradients `$g$` is maintained globally and updated by all threads without locks, significantly outperforms both Momentum SGD (which maintains no cross-thread statistics) and per-thread RMSProp (where each thread has its own independent `$g$`). > "RMSProp with shared statistics tends to be more robust than RMSProp with per-thread statistics, which is in turn more robust than Momentum SGD." Why does sharing gradient statistics matter? Each thread sees only a narrow slice of the environment — its own trajectory under its own exploration policy. The gradient magnitudes computed by a single thread might fluctuate wildly depending on whether it happened to encounter high-reward or low-reward states, an episode start or end, or a rare game event. Per-thread RMSProp normalizes learning rates based on these local statistics, which can be noisy and unrepresentative. Shared `$g$`, by contrast, aggregates gradient magnitude information across all threads simultaneously — 16 different trajectories spanning different states, different exploration behaviors, and different stages of episodes. The resulting per-parameter learning rate scaling is informed by a much more representative sample of the gradient distribution, smoothing out the idiosyncrasies of individual threads. This is a **systems-level insight** with theoretical implications. It suggests that in asynchronous training, the optimizer is not just finding a minimum but actively filtering out noise in the gradient signal. The shared statistics act as a form of **implicit ensemble**: each thread contributes to a collective estimate of parameter-wise gradient variance, and the normalized updates are less susceptible to the peculiarities of any single trajectory. This mechanism operates orthogonally to the data decorrelation from parallel actors. The parallelism decorrelates *what* gradients are computed; the shared optimizer statistics decorrelate *how much* each parameter is moved. Both contribute to stability, and Figure S5 shows that the combination is what makes the framework robust across a wide range of learning rates and random initializations. The significance of this finding is amplified by what it enables: the complete removal of locking in the optimization step. Hogwild! (Recht et al., 2011) had shown that lock-free SGD works under sparsity assumptions for supervised learning, but its application to deep RL — with all the non-stationarity, temporal correlation, and policy-dependent data distributions that RL entails — was far from obvious. The fact that Shared RMSProp with Hogwild!-style updates succeeds on 57 Atari games without divergence (Figure 2, Supplementary Figure S11) is experimental validation that the Hogwild! theory extends to this much more challenging setting. The paper doesn't prove this theoretically, but the empirical demonstration that there are "virtually no points with scores of 0 in regions with good learning rates" (Section 5.6) provides compelling evidence that the optimization does not collapse. ### Innovation 3: Difficulty-Adaptive Exploration via Thread-Specific Policies Is Implicit Curriculum Learning The paper's mechanism for ensuring diversity across parallel actors — giving each value-based thread a different `$\epsilon$` value sampled from a distribution, and relying on stochastic policy sampling with entropy regularization for A3C — appears at first glance to be a minor implementation detail. But it represents a more interesting conceptual contribution: **implicit curriculum learning through heterogeneous exploration**. The parallel actors collectively implement a spectrum of exploration-exploitation tradeoffs, and the shared model benefits from the combined experience of agents at different points on this spectrum. In standard single-agent RL with `$\epsilon$`-greedy exploration, the agent faces a tension: high `$\epsilon$` provides better exploration but noisy learning (since many actions are random), while low `$\epsilon$` provides cleaner learning signals (actions are mostly optimal) but risks getting stuck in local optima. The typical solution is to anneal `$\epsilon$` from high to low over time, trading off exploration for exploitation as learning progresses. But this means the agent never gets the benefit of both high-exploration and high-exploitation experience simultaneously. The asynchronous framework sidesteps this tradeoff entirely. By assigning different `$\epsilon$` values to different threads — sampling from three values `$\epsilon_1, \epsilon_2, \epsilon_3$` with different annealing schedules (from 1 to 0.1, 0.01, 0.5 respectively) — the system simultaneously runs highly exploratory agents (who discover new strategies and escape local optima), highly exploitative agents (who refine and deepen existing knowledge), and intermediate agents. The shared model receives gradients from all of them, effectively learning from an **exploration curriculum** where exploratory agents push the model into new regions of state space while exploitative agents consolidate knowledge in known regions. This is not merely a computational trick to keep threads diverse. It is a qualitatively different form of exploration than what single-agent RL can achieve. Single-agent exploration must be sequential: explore, then exploit. Multi-agent heterogeneous exploration is **parallel**: explore and exploit simultaneously, with the model integrating both types of experience in every update batch. The data efficiency gains observed for one-step methods with more threads (Figure 3, showing that 16 threads need fewer total frames to reach a given score than 1 thread) may be partly explained by this effect — with more threads, the model has access to a richer mix of exploration behaviors at every training step. For A3C, the mechanism is even more elegant: rather than externally imposing an `$\epsilon$` schedule, the policy's own entropy (maintained by the entropy regularization term `$\beta = 0.01$`) provides a natural diversity mechanism. Different threads naturally diverge in their exploration behavior because they experience different trajectories, which leads to different policy updates, which leads to different action distributions. The entropy bonus ensures that no thread collapses to a deterministic policy too quickly, maintaining the diversity that makes the parallel framework work. This is a more principled approach — rather than forcing exploration through external noise, it incentivizes the policy to remain appropriately uncertain. This insight connects to the broader curriculum learning literature, but with a twist: the curriculum is not designed by a human or a separate scheduling algorithm. It emerges naturally from the combination of heterogeneous exploration parameters and asynchronous parallel updates. The "curriculum" is the set of diverse experiences that the shared model sees at each update step, which spans the full exploration-exploitation spectrum. ### Innovation 4: Reconciling Conflicting Priors About On-Policy Deep RL A significant but understated contribution of this paper is that it provides an **empirical resolution to a field-level contradiction** about whether on-policy methods can work with deep neural networks for challenging RL tasks. Before this paper, the evidence was largely negative: attempts to use on-policy algorithms like REINFORCE or actor-critic with deep networks on complex domains had generally failed or required elaborate stabilization machinery (trust regions, natural gradients, or massive batch sizes). The success of DQN — an off-policy method — reinforced the belief that off-policy learning was somehow more compatible with deep function approximation. This paper demonstrates that the problem was never about on-policy vs. off-policy per se. It was about **data correlation**. On-policy methods failed with deep networks not because their update rules are fundamentally incompatible with neural network optimization, but because the standard single-agent training regime produces highly correlated data that those updates cannot handle. Once the correlation is broken — through parallelism instead of experience replay — on-policy methods not only work, they can outperform off-policy methods. A3C, an on-policy actor-critic method, achieves the highest scores in the paper (Table 1: 496.8% mean human-normalized for feedforward, 623.0% for LSTM, surpassing all DQN variants including the off-policy Prioritized DQN at 463.6%). This resolution is important because it reverses the default assumption that had taken hold in the field. It suggests that on-policy methods' previous failures were not evidence of a fundamental limitation but evidence that the training infrastructure (single-agent, sequential data collection) was the bottleneck, not the algorithm. The paper doesn't make this argument explicitly, but it is the clear implication: if your stabilization mechanism works (replay buffer for off-policy, parallelism for on-policy), then *both families of algorithms train successfully*. The choice between them should be driven by their inherent properties (continuous actions, exploration behavior, bias-variance characteristics), not by whether they happen to be trainable with deep networks. Furthermore, the paper demonstrates that the supposed stability advantage of off-policy methods (they can reuse old data) comes with a hidden cost: they cannot use on-policy multi-step returns without off-policy corrections. The paper's n-step Q-learning variant uses the forward-view n-step return, which is partially on-policy — the reward sequence comes from the behavior policy. This hybrid approach accelerates credit assignment in a way that purely off-policy one-step Q-learning cannot match, and Figure 1 shows that n-step Q-learning consistently learns faster than one-step Q-learning. The asynchronous framework thus not only enables purely on-policy methods but also enables partially-on-policy enhancements to off-policy methods, blurring the boundary in productive ways. ## 5. Experimental Analysis ### Evaluation Methodology - **Dataset.** The primary benchmark is the **Arcade Learning Environment (Bellemare et al., 2012)**, providing a simulator for Atari 2600 games. The paper evaluates on a subset of games for algorithm comparison and learning speed analysis (Figures 1, 3, 4), and on the full set of 57 Atari games for the state-of-the-art comparison against prior work (Table 1, Supplementary Table S3). Three additional domains are used: **TORCS 3D car racing simulator** (Wymann et al., 2013) for comparing all four asynchronous methods on a more visually realistic task; **MuJoCo physics simulator** (Todorov, 2015) for evaluating A3C on continuous motor control tasks with contact dynamics; and **Labyrinth**, a new custom 3D environment where agents must find rewards (apples worth +1, portals worth +10) in randomly generated mazes from visual input only. The Labyrinth task tests whether the agent can learn a general exploration strategy for novel maze layouts at each episode. - **Base model(s).** All experiments use the same neural network architecture as Mnih et al. (2013; 2015): a convolutional layer with 16 filters of size 8 × 8 with stride 4, followed by a convolutional layer with 32 filters of size 4 × 4 with stride 2, followed by a fully connected layer with 256 hidden units, all with ReLU nonlinearities. For value-based methods, the output is a linear layer with one unit per action representing Q-values. For A3C, the network has two output heads sharing the convolutional and FC layers: a softmax policy output and a single linear value output. An LSTM variant of A3C adds 256 LSTM cells after the final hidden layer for handling partial observability. Input preprocessing matches DQN: grayscale conversion, resizing to 84 × 84, and stacking 4 most recent frames; action repeat of 4 is used. For MuJoCo continuous control from physical state input, the network uses a single 200-unit ReLU hidden layer followed by 128 LSTM cells; from pixels, two spatial convolution layers (without nonlinearity or pooling) feed into the same LSTM. The policy head for continuous actions outputs the mean μ (linear layer) and variance σ^2 (SoftPlus activation) of a multivariate normal distribution with spherical covariance. - **Metrics.** The primary metric is **game score** — the total reward accumulated per episode. For the Atari domain, scores are reported using the **human starts evaluation metric** (30 minutes of emulator time per game, with random no-op starts as in Mnih et al., 2015). For comparison across all 57 games, scores are converted to **human-normalized scores**: 100 × (agent_score − random_score) / (human_score − random_score), where human and random scores come from Bellemare et al. (2012). Mean and median human-normalized scores across all 57 games are the aggregate metrics reported in Table 1. For TORCS, score is the cumulative reward proportional to the agent's velocity along the center of the track. For MuJoCo, score is the cumulative task-specific reward (details in Supplementary Section 9). For Labyrinth, score is the total points collected in 60-second episodes. Training speed is measured in **wall-clock hours** on a single machine. Data efficiency is measured in **training epochs**, where one epoch corresponds to 4 million frames across all threads (Section 5.5), or in total training frames. - **Baselines.** The paper compares against several published methods from the deep RL literature: **DQN** (Mnih et al., 2015), trained for 8 days on a single Nvidia K40 GPU; **Gorila** (Nair et al., 2015), trained for 4 days on 100 machines (plus 30 parameter servers); **Double DQN** (Van Hasselt et al., 2015), trained for 8 days on a GPU; **Dueling Double DQN** (Wang et al., 2015), trained for 8 days on a GPU; and **Prioritized DQN** (Schaul et al., 2015), trained for 8 days on a GPU. These baselines represent the state-of-the-art at the time of writing, all using experience replay and GPU hardware. The paper also uses **DQN-trained-on-GPU** as a direct learning speed comparison against the asynchronous methods trained on 16 CPU cores (Figure 1). For the TORCS experiments, a **human tester** baseline is included to provide an approximate performance ceiling. Within the asynchronous framework, the four proposed methods serve as baselines for each other: one-step Q-learning and one-step Sarsa as one-step value-based baselines, n-step Q-learning as a multi-step value-based baseline, and A3C as the policy-based method. The one-step Sarsa variant serves specifically as the on-policy baseline that would be impossible under experience replay. - **Generation budget / compute accounting.** The paper uses **wall-clock training time** (hours) as the primary fairness metric for comparing learning speed between methods. For the scalability analysis (Table 2), training speedup is measured as the ratio of time required to reach a fixed reference score using one thread divided by the time required using n threads. For data efficiency (Figure 3), the metric is total training frames across all threads, measured in epochs of 4 million frames each. The asynchronous methods use a fixed architecture of 16 actor-learner threads for all main experiments (Figures 1, 3, 4, Tables 1, 2), with each thread running on a single CPU core and no GPU. The key compute comparison is not FLOPs-matched but rather **resource-type-matched**: the asynchronous methods run on 16 CPU cores (a single multi-core machine) while DQN and its variants run on a single Nvidia K40 GPU. The paper argues that 16 CPU cores represent a significantly lower resource requirement than a high-end GPU or a 130-machine cluster. All four asynchronous methods use identical update frequencies: gradients are accumulated over t_max = 5 steps (or until episode termination) before being applied, the shared target network (for value-based methods) is updated every I_target = 40,000 global frames, and the Shared RMSProp optimizer uses decay factor α = 0.99 across all experiments. Learning rates are sampled from LogUniform(10^(-4), 10^(-2)) for the hyperparameter sensitivity analysis (50 experiments per game per method), with learning rate annealing to zero over training. - **Cross-validation / statistical protocol.** For the learning speed comparison (Figure 1), DQN results are averaged over 5 runs with different random seeds and fixed hyperparameters; asynchronous methods are averaged over the best 5 models from 50 experiments where learning rates were randomly sampled from LogUniform(10^(-4), 10^(-2)) with all other hyperparameters fixed. For the robustness analysis (Figure 2, Supplementary Figure S11), 50 experiments per algorithm per game are run with different learning rates and random initializations. For the final 57-game evaluation (Table 1), hyperparameters (learning rate and gradient norm clipping) were tuned using a search on six Atari games (Beamrider, Breakout, Pong, Q*bert, Seaquest, Space Invaders) and then fixed for all 57 games — a standard protocol following Van Hasselt et al. (2015). Both feedforward and LSTM A3C agents were trained for four days using 16 CPU cores. The human-normalized scores in Table 1 use the human starts evaluation protocol from Bellemare et al. (2012). For the scalability experiments (Table 2), speedup is averaged over seven Atari games (Beamrider, Breakout, Enduro, Pong, Q*bert, Seaquest, and Space Invaders) with the speedup defined as single-thread time divided by n-thread time to reach a fixed reference score. ### Main Quantitative Results #### Learning Speed Comparison: Asynchronous Methods vs. DQN (Figure 1, Section 5.1) **Headline result:** All four asynchronous methods successfully train neural network controllers on five Atari games (Beamrider, Breakout, Pong, Q*bert, Space Invaders) using only 16 CPU cores, and all learn faster than DQN trained on a single Nvidia K40 GPU. The policy-based A3C significantly outperforms all three value-based methods. Figure 1 shows training curves (score vs. wall-clock time in hours) for DQN and the four asynchronous methods. On **Beamrider**, A3C reaches approximately 10,000-12,000 score within 4-6 hours while DQN requires approximately 10-12 hours to reach comparable performance; n-step Q-learning reaches approximately 8,000-10,000 by 8 hours. On **Breakout**, A3C reaches roughly 400-500 score by 6 hours while DQN reaches only about 100-150 at the same wall-clock time. On **Pong**, all asynchronous methods reach scores of approximately 15-20 within 2-4 hours while DQN requires about 8-10 hours. On **Q*bert**, A3C reaches approximately 8,000-10,000 by 8 hours, significantly faster than DQN. On **Space Invaders**, A3C and n-step Q both reach approximately 800-1,200 score by 6 hours, again faster than DQN. The n-step Q-learning variant consistently learns faster than both one-step Q-learning and one-step Sarsa, confirming the value of multi-step returns for accelerating credit assignment. One-step Sarsa (on-policy) performs comparably to one-step Q-learning (off-policy), demonstrating that on-policy value-based methods are viable for deep RL when trained asynchronously. The paper notes: "the results suggest that n-step methods learn faster than one-step methods on some games. Overall, the policy-based advantage actor-critic method significantly outperforms all three value-based methods." A critical detail: the number of CPU cores (16) was used for all asynchronous methods, while DQN used a single GPU. This is not a FLOPs-matched comparison — it is a demonstration that asynchronous CPU training can be faster in wall-clock time than GPU training for DQN, with lower hardware requirements. #### State-of-the-Art Comparison on 57 Atari Games (Table 1, Supplementary Table S3, Section 5.1) **Headline result:** A3C with an LSTM achieves a mean human-normalized score of 623.0% and median of 112.6% across 57 Atari games after four days of training on 16 CPU cores, surpassing all prior methods including Prioritized DQN (463.6% mean, 127.6% median, trained for 8 days on GPU). A feedforward A3C trained for four days achieves 496.8% mean and 116.6% median. Even after just one day of training, feedforward A3C (344.1% mean, 68.2% median) matches or exceeds the mean score of Dueling Double DQN (343.8% mean) and nearly reaches the median of Gorila (71.3%). Table 1 provides the comparison: | Method | Training Time | Mean | Median | |---|---|---|---| | DQN | 8 days on GPU | 121.9% | 47.5% | | Gorila | 4 days, 100 machines | 215.2% | 71.3% | | Double DQN (D-DQN) | 8 days on GPU | 332.9% | 110.9% | | Dueling D-DQN | 8 days on GPU | 343.8% | 117.1% | | Prioritized DQN | 8 days on GPU | 463.6% | 127.6% | | **A3C, FF (1 day)** | **1 day on CPU** | **344.1%** | **68.2%** | | **A3C, FF (4 days)** | **4 days on CPU** | **496.8%** | **116.6%** | | **A3C, LSTM (4 days)** | **4 days on CPU** | **623.0%** | **112.6%** | The key comparisons: A3C FF (4 days) achieves a mean score 44.8% higher than Prioritized DQN (496.8% vs. 463.6%) while training for half the time (4 days vs. 8 days) on CPU instead of GPU. A3C LSTM achieves a mean score 34.4% higher than Prioritized DQN (623.0% vs. 463.6%). The LSTM variant provides a substantial boost over the feedforward architecture, particularly on games requiring memory and temporal reasoning. Several important nuances in these results: - **The median is more variable than the mean.** A3C LSTM achieves 112.6% median, which is slightly lower than Prioritized DQN's 127.6% median, even though its mean is much higher. This indicates that A3C's advantage is driven by particularly strong performance on a subset of games while possibly underperforming on others — the LSTM variant achieves very high scores on some games (pulling up the mean) but may struggle on games where recurrent state is less useful or where the LSTM makes optimization harder. - **Supplementary Table S3 reveals per-game patterns.** A3C LSTM achieves dramatically higher scores than all prior methods on several games: Alien (945.3 vs. DQN's 570.2 and Prioritized's 900.5), Assault (14,497.9 vs. Prioritized's 7,748.5), Asteroids (5,093.1 vs. Prioritized's 1,654.0), Chopper Command (10,150.0 vs. Prioritized's 6,604.0), and Video Pinball (470,310.5 vs. Prioritized's 374,886.9). However, it underperforms on some games: Enduro (-82.5 vs. Prioritized's 1,884.4), Freeway (0.1 vs. Prioritized's 27.9), and Kung-Fu Master (40,835.0 vs. Prioritized's 31,244.0 but lower than Double DQN's 30,207.0 — actually A3C LSTM wins here, but FF A3C gets only 28,819.0). These per-game variations suggest that A3C's overall advantage comes from a subset of games where policy-based learning with entropy regularization and LSTM memory provides significant benefits. - **The one-day result is striking.** Feedforward A3C trained for just one day (344.1% mean) essentially matches Dueling Double DQN's final performance (343.8% mean after 8 days), representing an approximately 8× reduction in training time while using less specialized hardware. This has obvious practical implications for research iteration speed. The paper notes that the improvements from Double DQN (reducing overestimation bias) and Dueling DQN (separate value and advantage streams) could potentially be incorporated into the asynchronous one-step Q and n-step Q methods, suggesting the value-based asynchronous results in Table 1 (which don't use these techniques) are lower bounds on what asynchronous value-based methods could achieve. #### TORCS Car Racing Simulator (Section 5.2, Supplementary Figure S6) **Headline result:** A3C is the best-performing method across all four TORCS configurations (slow/fast car, with/without opponent bots), reaching between roughly 75% and 90% of the score obtained by a human tester in about 12 hours of training. Supplementary Figure S6 shows training curves (score vs. wall-clock hours) for all four asynchronous methods across the four TORCS variants. The key findings: - **Multi-step methods learn faster than one-step methods.** Both n-step Q and A3C (which uses n-step returns) consistently outperform one-step Q and one-step Sarsa across all four configurations. For the "slow car, no bots" setting, A3C reaches approximately 3,500-4,000 score by 12 hours, while one-step Q plateaus around 2,000-2,500. - **A3C is the top performer.** On "fast car, bots" — the most challenging configuration — A3C reaches approximately 4,500-5,000 score by 20-25 hours, while n-step Q reaches roughly 3,500-4,000, and the one-step methods reach only about 2,000-3,000. The human tester baseline is approximately 5,500-6,000, meaning A3C reaches roughly 75-80% of human performance. - **One-step Sarsa and one-step Q-learning perform similarly**, confirming that on-policy and off-policy value-based methods are comparably effective when both can be trained stably via the asynchronous framework. This reinforces the paper's claim that the framework unlocks on-policy methods without performance penalties. The TORCS domain tests generalization beyond Atari: it has more realistic graphics, requires learning vehicle dynamics, and involves continuous control aspects (steering, acceleration, braking) in a 3D environment. The success of all four asynchronous methods on TORCS demonstrates that the framework is not specific to the discrete-action, relatively simple Atari domain. #### Continuous Action Control Using MuJoCo (Section 5.3, Supplementary Figures S7, S8) **Headline result:** A3C successfully solves a variety of continuous motor control tasks (pendulum, pointmass2D, gripper, and locomotion tasks) from both physical state and pixel inputs on CPU, typically within a few hours from states and within 24 hours from pixels. Supplementary Figure S8 shows score per episode vs. wall-clock time for the MuJoCo tasks, with error bars for the top 5 experiments. The paper reports that "in all problems, using either the physical state or pixels as input, Asynchronous Advantage-Critic found good solutions in less than 24 hours of training and typically in under a few hours." Supplementary Figure S7 shows scatter plots of best score vs. learning rate, demonstrating that for most tasks there is a wide range of learning rates (from roughly 10^(-5) to 10^(-2)) that lead to good performance, indicating robustness to hyperparameter choice in the continuous domain as well. Specific implementation details for continuous actions (Supplementary Section 9): the policy network outputs the mean vector μ and scalar variance σ^2 of a multidimensional normal distribution with spherical covariance. The entropy regularization for continuous actions uses the differential entropy of the normal distribution, -1/2(log(2πσ^2) + 1), with a weight of 10^(-4) across all tasks (much smaller than the Atari β = 0.01). Since MuJoCo episodes are typically at most a few hundred steps, the agents do not use bootstrapping — "we did not use any bootstrapping in the policy or value function updates and batched each episode into a single update." This makes the MuJoCo variant essentially a Monte Carlo policy gradient method rather than an n-step TD method. The significance of these results is that they extend the asynchronous framework to **continuous action spaces**, which value-based methods like Q-learning handle poorly (they require solving an optimization problem max_a Q(s, a) at each step). A3C's policy gradient approach handles continuous actions natively by outputting distribution parameters. The fact that A3C succeeds on MuJoCo — a standard benchmark for continuous control that later became central to algorithms like TRPO, PPO, and DDPG — positions it as a general-purpose algorithm spanning discrete and continuous domains. The paper claims that this makes A3C "the most general and successful reinforcement learning agent to date" (Section 1). #### Labyrinth: Learning General Exploration Strategies in Random 3D Mazes (Section 5.4) **Headline result:** An A3C LSTM agent trained purely from 84 × 84 RGB visual input achieves a final average score of around 50 in the Labyrinth environment, demonstrating that it can learn a general strategy for exploring randomly generated 3D mazes and collecting rewards. The Labyrinth task is structured as follows (Section 5.4): at the start of each episode, the agent is placed in a new randomly generated maze of rooms and corridors. The maze contains apples (reward +1 each) and portals (reward +10, after which the agent is respawned at a random location and all apples regenerate). Episodes last 60 seconds. The optimal strategy involves first finding the portal and then repeatedly returning to it after each respawn, collecting apples along the way. The key challenge is that the agent sees a **new maze layout every episode**, so it must learn a general exploration strategy rather than memorizing specific maze configurations. The reported final average score of approximately 50 indicates that the agent learns a reasonable strategy — collecting the portal at least once (for +10) and consistently picking up apples between respawns. The paper notes that "this task is much more challenging than the TORCS driving domain because the agent is faced with a new maze in each episode and must learn a general strategy for exploring random mazes." This experiment is significant because it tests generalization to novel environments — a capability that goes beyond the standard RL benchmark of learning a fixed policy for a single environment. The LSTM is crucial here because the agent needs to remember which areas of the current maze it has already explored, a form of episodic memory that recurrent state can provide. The paper does not report an ablation with a feedforward network on Labyrinth, so the contribution of the LSTM is not directly quantified, but the visual input alone (no map, no coordinates) makes the task sufficiently challenging that recurrent state is likely essential. #### Scalability Analysis (Table 2, Figures 3 and 4, Section 5.5) **Headline result:** All four methods achieve substantial speedups from using multiple worker threads, with 16 threads providing at least an order of magnitude speedup across all methods. One-step methods exhibit **superlinear speedups** that suggest improved data efficiency with more parallel workers, beyond pure computational parallelism. Table 2 reports the average training speedup for each method and number of threads, averaged over seven Atari games: | Method | 1 thread | 2 threads | 4 threads | 8 threads | 16 threads | |---|---|---|---|---|---| | 1-step Q | 1.0 | 3.0 | 6.3 | 13.3 | 24.1 | | 1-step SARSA | 1.0 | 2.8 | 5.9 | 13.1 | 22.1 | | n-step Q | 1.0 | 2.7 | 5.9 | 10.7 | 17.2 | | A3C | 1.0 | 2.1 | 3.7 | 6.9 | 12.5 | Several important patterns emerge: - **One-step methods scale superlinearly.** With 16 threads, one-step Q-learning achieves a 24.1× speedup (far exceeding the ideal linear 16×), and one-step Sarsa achieves 22.1×. Pure computational parallelism cannot explain speedups beyond the number of threads — this means one-step methods actually require **fewer total training frames** to reach the same score when using more parallel workers. The paper explains: "we observe that one-step methods (one-step Q and one-step Sarsa) often require less data to achieve a particular score when using more parallel actor-learners. We believe this is due to positive effect of multiple threads to reduce the bias in one-step methods." - **n-step Q and A3C scale roughly linearly or sublinearly.** n-step Q achieves 17.2× speedup with 16 threads (slightly superlinear), while A3C achieves 12.5× (sublinear). This sublinearity for A3C is not explained in detail but may be due to the on-policy nature of the algorithm — with more threads, each thread's local policy diverges more from the shared policy between synchronization steps, potentially reducing the effectiveness of the policy gradient updates. Alternatively, the increased write conflicts in the Hogwild! update scheme at higher thread counts may affect A3C (which updates both policy and value parameters) more than the value-based methods. - **Even sublinear speedups are substantial.** A3C's 12.5× speedup with 16 threads means training time is reduced by more than an order of magnitude compared to single-thread training. Figure 3 shows data efficiency curves (score vs. total training epochs, where one epoch = 4 million frames across all threads) for one-step Q, n-step Q, and A3C with varying thread counts on five Atari games. The key finding: for one-step Q-learning, the 16-thread curve consistently achieves higher scores for the same number of total training frames compared to 1-thread — the curve shifts upward, not just leftward, as threads increase. This confirms the superlinear speedup: more threads not only consume frames faster (wall-clock speedup) but also make better use of each frame (data efficiency improvement). For n-step Q and A3C, the curves for different thread counts are closer together in data efficiency, indicating that most of their speedup comes from pure computational parallelism rather than improved per-frame learning. Figure 4 shows the same data as wall-clock time curves (score vs. training hours), confirming that all methods benefit substantially from additional threads in terms of time to solution. The vertical gaps between thread-count curves are largest for one-step methods, consistent with Table 2. Supplementary Figure S9 shows data efficiency for one-step Sarsa (analogous to Figure 3), and Supplementary Figure S10 shows wall-clock speed for one-step Sarsa (analogous to Figure 4). The Sarsa results follow the same pattern as one-step Q-learning: superlinear speedups and improved data efficiency with more threads. **Why do one-step methods benefit more from parallelism?** The paper's explanation — "the positive effect of multiple threads to reduce the bias in one-step methods" — requires unpacking. One-step methods have higher bias in their value estimates because they propagate rewards only one step per update. With more parallel threads exploring simultaneously, the shared model receives updates from more diverse trajectories, which may help average out the bias in individual value estimates more effectively than for n-step methods (which already reduce bias through multi-step returns). This is a speculative explanation, but the empirical effect is clear and consistent. #### Robustness and Stability Analysis (Figure 2, Supplementary Figure S11, Section 5.6) **Headline result:** All four asynchronous methods are robust to the choice of learning rate and random initialization, with wide ranges of learning rates producing good scores and virtually no experiments collapsing to zero score when the learning rate was reasonable. A3C shows particularly strong robustness, with large regions of consistently high scores across five games. Figure 2 shows scatter plots of final score vs. learning rate for A3C on five games (Beamrider, Breakout, Pong, Q*bert, Space Invaders), based on 50 experiments per game with randomly sampled learning rates from LogUniform(10^(-4), 10^(-2)) and random initializations. The key observations: - **Wide viable learning rate ranges.** For Beamrider, learning rates from roughly 3 × 10^(-4) to 6 × 10^(-3) all produce scores above 10,000, with many above 14,000. For Breakout, rates from 10^(-4) to 3 × 10^(-3) produce scores above 400, with a cluster around 600-800. For Pong, rates from 10^(-4) to 5 × 10^(-3) almost all produce scores of 15-21 (near the maximum). For Q*bert, rates from 2 × 10^(-4) to 5 × 10^(-3) produce scores above 6,000-8,000. For Space Invaders, rates from 3 × 10^(-4) to 3 × 10^(-3) produce scores above 800-1,000. - **No catastrophic failures at reasonable learning rates.** The paper notes: "there are virtually no points with scores of 0 in regions with good learning rates, indicating that the methods are stable and do not collapse or diverge once they are learning." This is significant because deep RL algorithms are notoriously brittle — small changes in hyperparameters can cause complete training collapse. The absence of zero-score experiments in the viable learning rate regions is evidence that the asynchronous framework genuinely stabilizes training, not just accelerates it. - **Games vary in sensitivity.** Pong is extremely robust — nearly every learning rate produces a good score. Q*bert shows more variance (scores range from roughly 2,000 to 12,000 at similar learning rates), likely due to sensitivity to random initialization or exploration luck in early training. Supplementary Figure S11 shows the same scatter plots for one-step Q, one-step Sarsa, and n-step Q. The value-based methods also show reasonable robustness, though with somewhat more scatter than A3C. The key pattern: all methods have learning rate ranges where good scores are consistently achieved, and the density of high-scoring runs is high within those ranges. This supports the paper's claim that the asynchronous framework provides a general stabilization benefit across algorithm families, not just for a single method. The 50-run analysis per game per method is a substantial computational commitment (50 × 4 methods × 5 games = 1,000 training runs just for this section), which gives the robustness claims statistical weight beyond anecdotal evidence. ### Ablation Studies and Robustness Checks **Optimizer choice: Shared RMSProp vs. per-thread RMSProp vs. Momentum SGD (Supplementary Figure S5):** Shared RMSProp is substantially more robust than both alternatives across two algorithms (n-step Q and A3C) and four Atari games. The comparison uses rank-sorted performance curves from 50 experiments per setting with different learning rates and initializations. Shared RMSProp produces flatter curves (indicating consistent performance across a wide range of hyperparameters) and higher maximum scores compared to per-thread RMSProp and Momentum SGD. The paper states that "RMSProp with shared statistics tends to be more robust than RMSProp with per-thread statistics, which is in turn more robust than Momentum SGD." This is evaluated on Breakout, Beamrider, Seaquest, and Space Invaders. **Number of actor-learner threads: 1 vs. 2 vs. 4 vs. 8 vs. 16 (Table 2, Figures 3, 4, Supplementary Figures S9, S10):** Increasing thread count consistently improves both wall-clock training speed and (for one-step methods) data efficiency. Table 2 quantifies the speedup: one-step Q achieves 24.1× speedup with 16 threads (superlinear), while A3C achieves 12.5× (sublinear but still an order of magnitude). Figures 3 and S9 show that one-step methods require fewer total training frames to reach a given score with more threads, confirming the data efficiency benefit beyond pure parallelism. Figures 4 and S10 confirm the wall-clock time advantage across all methods and thread counts. The superlinear scaling for one-step methods is a non-obvious finding — more threads not only provide more computational throughput but actually improve per-frame learning, likely through reduced bias from parallel exploration diversity. **On-policy vs. off-policy: one-step Sarsa vs. one-step Q-learning (Figure 1, Supplementary Figures S6, S9-S11):** The on-policy Sarsa variant performs comparably to the off-policy Q-learning variant across Atari games and TORCS configurations, with no consistent performance penalty for being on-policy. Figure 1 shows nearly overlapping curves for Sarsa and Q-learning on Breakout, Pong, and Space Invaders, and comparable performance on Beamrider and Q*bert. Supplementary Figure S6 shows similar results for TORCS. This is a crucial finding because it demonstrates that on-policy methods — previously thought incompatible with deep neural network training — achieve competitive performance when trained asynchronously, validating the paper's central claim that the asynchronous framework unlocks on-policy deep RL. **One-step vs. n-step returns: one-step Q/n-step Q comparison (Figure 1, Supplementary Figure S6):** N-step methods consistently learn faster than one-step methods, with larger advantages on some games (Beamrider, Space Invaders) than others (Breakout). Figure 1 shows n-step Q curves rising faster than one-step Q curves on all five games, with particularly large gaps on Beamrider and Space Invaders. Supplementary Figure S6 shows similar patterns on TORCS, where n-step Q substantially outperforms one-step Q in all four configurations. This confirms that multi-step returns accelerate credit assignment, and the forward-view n-step implementation (chosen for compatibility with momentum-based optimizers and backpropagation) is effective. **Value-based vs. policy-based: A3C vs. all value-based methods (Figure 1, Table 1, Supplementary Figure S6):** A3C generally outperforms the three value-based methods across all domains tested. On Atari (Figure 1), A3C achieves the highest scores on Breakout, Beamrider, Q*bert, and Space Invaders, sometimes by large margins (e.g., Breakout: A3C reaches ~600-800 while n-step Q reaches ~300-400). On TORCS (Supplementary Figure S6), A3C is the top performer in all four configurations. The 57-game evaluation (Table 1) shows A3C surpassing all prior DQN variants. This suggests that policy-based methods, when stabilized by the asynchronous framework, have inherent advantages (native stochastic exploration via entropy regularization, no max-operator bias, natural extension to continuous actions) that value-based methods cannot fully match with the architectures tested. **Feedforward vs. LSTM for A3C (Table 1, Supplementary Table S3):** The LSTM variant of A3C substantially outperforms the feedforward variant in mean human-normalized score across 57 Atari games (623.0% vs. 496.8%) but achieves a slightly lower median (112.6% vs. 116.6%). This means the LSTM provides dramatic improvements on a subset of games (Alien: 945.3 vs. 518.4; Assault: 14,497.9 vs. 5,474.9; Asteroids: 5,093.1 vs. 4,474.5) while performing similarly or slightly worse on others. The LSTM is particularly beneficial on games requiring memory of past events (e.g., tracking moving objects that leave the screen, remembering visited locations), which aligns with the theoretical expectation that recurrent state addresses partial observability. **Labyrinth generalization to novel environments (Section 5.4):** The A3C LSTM agent learns a general maze exploration strategy from visual input, achieving an average score of approximately 50 in randomly generated mazes. Since each episode features a new maze, the agent cannot memorize specific layouts and must learn transferable exploration behaviors (e.g., systematically checking corridors, remembering visited rooms). No feedforward baseline is reported for Labyrinth, so the contribution of the LSTM to this capability is not isolated. No quantitative comparison to alternative exploration strategies or human performance is provided. **Continuous control from pixels vs. states (Supplementary Section 9, Supplementary Figures S7, S8):** A3C successfully learns continuous control tasks both from low-dimensional physical state input (joint positions and velocities) and from high-dimensional pixel input. From states, most tasks are solved within a few hours. From pixels, solutions are found within 24 hours for the three tasks tested (pendulum, pointmass2D, gripper). This demonstrates that the asynchronous framework works across input modalities, but the paper does not provide quantitative comparisons of final performance or sample efficiency between state-based and pixel-based training. ### Critical Assessment **Claim: "Parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers" (Abstract).** This claim is well-supported for the specific setup tested: 16 CPU threads, Shared RMSProp, Atari/TORCS/MuJoCo/Labyrinth domains, and the specific network architecture from Mnih et al. (2013). Figures 1, 2, 3, and Supplementary Figure S11 demonstrate that all four methods train stably — scores improve over time without collapse, and the robustness analysis shows no catastrophic failures at reasonable learning rates. However, the paper does not isolate the stabilizing effect of parallelism from the stabilizing effect of **Shared RMSProp**. The optimizer comparison (Supplementary Figure S5) shows that optimizer choice matters significantly — Momentum SGD is much less robust than Shared RMSProp. It is possible that Shared RMSProp alone (without parallelism) provides substantial stabilization, and the paper does not include a baseline that uses Shared RMSProp with a single thread and experience replay to disentangle these effects. The claim is more precisely: "parallelism + Shared RMSProp + diverse exploration has a stabilizing effect" — the contribution of each component is not individually ablated. **Claim: "Asynchronous actor-critic surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU" (Abstract).** This claim is supported by Table 1, which shows A3C FF (4 days, CPU) achieving 496.8% mean human-normalized vs. Prioritized DQN's 463.6% (8 days, GPU). However, the comparison has several caveats that the paper acknowledges only partially: - **Training time is not compute-matched.** Four days on 16 CPU cores vs. 8 days on a K40 GPU are not directly comparable — the total FLOPs or computational cost may differ substantially. The paper frames this as an advantage (CPU is cheaper/more accessible than GPU), but from a pure computational efficiency standpoint, it is not established that A3C uses fewer total operations. - **The prior methods were not re-tuned for a 4-day training budget.** DQN, Double DQN, Dueling DQN, and Prioritized DQN were all trained for 8 days because that was the standard protocol at the time. If these methods were trained for only 4 days, their performance might be lower than reported — but conversely, if they were given A3C's wall-clock budget on their native hardware, they might perform differently. A fairer comparison would be: method A on its optimal hardware and budget vs. method B on its optimal hardware and budget, OR both methods at equal wall-clock time on the same hardware. The paper provides neither. - **A3C's advantage is concentrated on specific games.** The LSTM variant achieves 623.0% mean but only 112.6% median (lower than Prioritized DQN's 127.6% median). This means A3C's mean advantage is driven by extreme scores on a subset of games. A method that achieves superhuman scores on a few games while being mediocre on others might have a high mean but may not be "better" in a practical sense depending on the deployment context. The paper emphasizes the mean score throughout the abstract and introduction, with the median relegated to Table 1. **Claim: "Asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes" (Abstract).** The MuJoCo results (Section 5.3, Supplementary Figures S7, S8) demonstrate that A3C can learn continuous control tasks from both states and pixels. However, the evaluation is less rigorous than the Atari experiments: - **Only A3C was tested on MuJoCo.** The value-based methods are not extended to continuous actions (the paper notes this is because they "are easily extended to continuous actions" — referring to A3C). This leaves open the question of whether the asynchronous framework benefits continuous control specifically, or whether A3C would work equally well on these tasks without parallelism. - **No quantitative comparison to prior continuous control methods.** In 2016, algorithms like DDPG (Lillicrap et al., 2015), TRPO (Schulman et al., 2015a), and GPS (Levine et al., 2015) had established benchmarks on MuJoCo tasks. The paper does not compare A3C's final performance, sample efficiency, or wall-clock time against these methods. The claim of "success" is qualitative — the agent finds "good solutions" — without quantitative benchmarking against the state of the art. - **The Labyrinth task has no baselines at all.** The reported score of approximately 50 is presented without comparison to any alternative method, human performance, or even a random agent. It is unclear whether 50 is a good score, a mediocre score, or trivially achievable by simpler methods. The environment is also custom (not a standard benchmark), making it impossible for readers to calibrate the result against known reference points. This experiment serves as a qualitative demonstration that A3C can learn from visual input in a 3D maze, but does not provide evidence that the asynchronous framework is necessary or beneficial for this task. **Claim: "Using parallel actor-learners to update a shared model had a stabilizing effect on the learning process of the three value-based methods" (Section 6).** This is the paper's central causal claim — that parallelism causes stability. The experimental evidence for causality is limited: - **No experiment compares: single-thread with replay buffer vs. multi-thread without replay buffer, holding all else equal.** The paper compares multi-thread asynchronous methods against DQN (single-thread with replay buffer), but these differ in the RL algorithm (DQN uses a specific Q-learning variant with target network and specific hyperparameters) and the hardware (GPU vs. CPU). The observed stability of asynchronous methods could be due to Shared RMSProp, the target network update frequency, the learning rate schedule, or other confounds rather than parallelism per se. - **The one-step Q-learning variant is the closest to a controlled comparison.** It uses the same Q-learning update as DQN but replaces the replay buffer with asynchronous parallelism. However, it also uses Shared RMSProp (vs. DQN's RMSProp or Adam — the paper doesn't specify DQN's optimizer), different exploration (thread-specific ε vs. DQN's annealed ε), gradient accumulation over 5 steps (vs. DQN's minibatch sampling from replay), and Hogwild! updates (vs. DQN's standard minibatch SGD). Any of these differences could contribute to stability independently of parallelism. - **The paper does not include a "single-thread with Shared RMSProp but no replay buffer" baseline.** This is the critical missing experiment. If a single thread with Shared RMSProp and no replay buffer also trains stably, then parallelism is not the stabilizer — Shared RMSProp is. If a single thread without replay buffer diverges even with Shared RMSProp, then parallelism is necessary. Without this ablation, the causal attribution of stability to parallelism is correlational rather than demonstrated. A fairer characterization of what the experiments demonstrate: the combination of asynchronous parallelism, Shared RMSProp, diverse exploration, gradient accumulation, and target networks (for value-based methods) enables stable training of on-policy and off-policy deep RL algorithms on the tested domains. The specific contribution of each component is not isolated, and claims that parallelism alone is the stabilizer are not rigorously tested. **Claim: "All four methods achieve substantial speedups from using multiple worker threads, with 16 threads leading to at least an order of magnitude speedup" (Section 5.5).** This claim is quantitatively supported by Table 2. However, the speedup metric has an important limitation: it measures speedup relative to single-thread performance of the **same method**, not relative to the best single-thread method. If single-thread A3C is slower than single-thread DQN, a 12.5× speedup over single-thread A3C might still be slower than DQN. The paper does not provide single-thread baselines for DQN's training time on the same hardware, so the absolute performance of the multi-thread system vs. optimized single-thread alternatives is not established. **Missing experiments that would strengthen the paper:** 1. **Single-thread + Shared RMSProp + no replay buffer** to isolate the stabilizing effect of parallelism from the optimizer. 2. **Single-thread + Shared RMSProp + experience replay** to test whether the benefit comes from parallelism or just from removing the replay buffer's constraints while using a good optimizer. 3. **Varying t_max** to understand the sensitivity to the gradient accumulation interval — is 5 steps optimal, or could larger values provide more stability with minimal cost? 4. **Varying the number of threads while controlling for total environment steps**, to cleanly separate the data efficiency effect from the computational speedup effect. 5. **Ablation of diverse exploration per thread**: what happens if all threads use the same ε schedule? This would test whether exploration diversity is necessary or merely beneficial. 6. **Comparison of the asynchronous one-step Q-learning with Double DQN and Dueling architectures** to establish whether the framework's benefits are orthogonal to existing Q-learning improvements. 7. **Performance of A3C on MuJoCo compared to published DDPG/TRPO results** to contextualize the continuous control results quantitatively. **Conditions under which the paper's claims hold (based on experimental evidence):** - **The framework works across algorithm families** (value-based, policy-based, on-policy, off-policy) — demonstrated for the specific four algorithms tested, on the specific domains tested. Generalization to other algorithms (e.g., TRPO, DDPG, policy gradient with GAE) is plausible but not tested. - **16 CPU cores provide approximately 12-24× speedup over single-thread** — holds for the seven Atari games in Table 2. Scaling beyond 16 threads is not tested. - **A3C outperforms DQN variants on Atari** — holds for the specific training budgets (4 days CPU vs. 8 days GPU) and evaluation protocol used. Different budget allocations or hardware configurations might yield different rankings. - **Shared RMSProp is more robust than per-thread RMSProp or Momentum SGD** — holds for the four games and two algorithms tested in Supplementary Figure S5. Generalization to other domains or algorithms is untested. - **The framework works on continuous control** — holds for A3C on the MuJoCo tasks tested, with the caveat of no quantitative baselines and no testing of value-based methods on continuous actions. ## 6. Limitations and Trade-offs ### The Single-Machine Constraint: Scaling Beyond One CPU Box Is Unexplored **The assumption or constraint.** The entire framework is designed around a single multi-core CPU machine with shared memory, exploiting Hogwild!-style lock-free updates that assume all threads can read and write a common parameter vector without explicit synchronization. The paper explicitly positions this as an advantage over Gorila's 130-machine distributed architecture, but it simultaneously imposes a hard ceiling: the number of parallel actor-learners is bounded by the number of CPU cores on a single machine. The paper tests only up to 16 threads and does not explore whether the approach degrades — or even functions — when actors must communicate over a network rather than through shared memory: > "Keeping the learners on a single machine removes the communication costs of sending gradients and parameters and enables us to use Hogwild! (Recht et al., 2011) style updates for training." This constraint is not presented as a limitation but as a deliberate design choice. However, it means the approach does not scale horizontally — you cannot simply add more machines to train faster or handle more complex environments. **The consequence.** For environments where 16 parallel actors do not provide sufficient throughput (e.g., domains requiring billions of timesteps of experience, or environments where each step is computationally expensive to simulate), the single-machine approach hits a wall. The Hogwild! mechanism relies on the assumption that parameter writes from different threads rarely conflict because the shared memory bus serializes them at the hardware level. Distributing across multiple machines would reintroduce the communication overhead and parameter-server complexity that the paper deliberately avoided, and it is unclear whether the stabilization benefits of asynchronous parallelism would survive the increased staleness and communication delays of a distributed setting. The paper cannot claim to have "solved" the distributed deep RL problem in the way Gorila attempted — it has solved a different, more constrained version of it. **What evidence exists in the paper.** Table 2 shows speedup scaling from 1 to 16 threads, with A3C achieving only 12.5× speedup at 16 threads (sublinear scaling). The trend suggests diminishing returns as thread count increases — A3C goes from 1.0 to 2.1 to 3.7 to 6.9 to 12.5, meaning each doubling of threads from 8 to 16 yielded only 1.8× additional speedup rather than the ideal 2×. Extrapolating, 32 threads might yield even less marginal benefit, and the sublinearity may worsen as write conflicts increase. No experiments beyond 16 threads are conducted, and no distributed (multi-machine) variant is tested. The paper provides no guidance on what happens when the shared memory assumption breaks. **Mitigation status.** The paper does not address this limitation. It frames the single-machine design as a feature (lower resource requirements, no communication overhead), which it is for the scale of problems tested. But for practitioners needing to scale beyond what fits in one box — training on thousands of environments simultaneously, or tackling domains where sample efficiency demands massive parallelism — the paper offers no path forward. The conclusion nods to combining experience replay with the asynchronous framework for data efficiency, but does not discuss distributed scaling. A natural extension (distributed A3C with parameter servers) is left entirely to future work. ### Data Efficiency: Replaying Old Experience Would Likely Help, But Is Not Done **The assumption or constraint.** The framework achieves stabilization by replacing experience replay with parallel actors, but in doing so, it **throws away all historical data**. Each transition is used exactly once — to compute the on-policy update for the thread that generated it — and then discarded. The paper acknowledges this tradeoff explicitly in the conclusion: > "While this shows that stable online Q-learning is possible without experience replay, which was used for this purpose in DQN, it does not mean that experience replay is not useful. Incorporating experience replay into the asynchronous reinforcement learning framework could substantially improve the data efficiency of these methods by reusing old data." This is an admission that the paper's framework is **sample-inefficient by design** — it trades data efficiency for algorithmic generality (enabling on-policy methods) and computational simplicity (no replay buffer management). **The consequence.** In domains where environment interaction is expensive — robotics, real-world simulators with high-fidelity physics, or any setting where the cost of generating a single transition dominates the cost of a gradient update — throwing away data after one use is wasteful. DQN's replay buffer allows each transition to contribute to many gradient updates, amortizing the cost of data collection. A3C and the other asynchronous methods cannot do this: to achieve the same effective number of gradient updates per environment interaction, they would need to generate proportionally more data, which may be prohibitively expensive. The paper's TORCS results hint at this tension — Supplementary Figure S6 shows all methods requiring approximately 12-40 hours of training for a driving simulator where interacting with the environment is relatively expensive compared to Atari (which runs at thousands of frames per second). The paper notes this directly in the context of TORCS: > "This could in turn lead to much faster training times in domains like TORCS where interacting with the environment is more expensive than updating the model for the architecture we used." **What evidence exists in the paper.** The paper provides no direct comparison of sample efficiency (total environment frames to reach a given performance) between the asynchronous methods and replay-based methods. Figure 3 shows data efficiency across thread counts for the same method, but does not compare against DQN's sample efficiency. The scalability analysis (Table 2, Figures 3-4) measures speedup in wall-clock time and total frames (across all threads), but does not report **transitions-per-update** or **updates-per-transition** — the standard metrics for comparing how effectively different methods use environment data. The one-step methods' superlinear speedup with more threads (24.1× for one-step Q with 16 threads) suggests that the baseline single-thread data efficiency was poor and that parallelism partly compensates — but even the improved multi-thread efficiency may not match what experience replay could achieve on the same data. **Mitigation status.** The paper explicitly calls out this limitation and suggests future work combining experience replay with asynchronous parallelism. However, it provides no experimental evidence for how such a combination would perform or whether the two stabilization mechanisms (parallelism for decorrelation, replay for data reuse) would interact synergistically or conflict. The suggestion is speculative — a practitioner reading this paper in 2016 would not know whether adding replay to A3C would improve sample efficiency, degrade performance due to off-policy bias in the policy gradient, or have no effect. ### The "Stabilizing Effect" of Parallelism Is Not Isolated from Optimizer Choice **The assumption or constraint.** The paper's central claim is that parallel actor-learners provide a stabilizing effect that substitutes for experience replay. However, the experimental design conflates parallelism with a specific optimizer choice: Shared RMSProp. Every experiment demonstrating the success of the asynchronous framework uses Shared RMSProp, and the comparison of optimizers (Supplementary Figure S5) reveals that the choice of optimizer significantly impacts robustness and final performance. The paper states: > "A comparison on a subset of Atari 2600 games showed that a variant of RMSProp where statistics g are shared across threads is considerably more robust than the other two methods." The causal chain is ambiguous: does the framework work because parallelism decorrelates the data, because Shared RMSProp smooths the optimization, or because the combination of both is necessary? The paper does not provide the experiment that would disambiguate these: **single-thread training with Shared RMSProp and no replay buffer**. If a single thread with Shared RMSProp also trains stably on Atari, then parallelism is not the primary stabilizer — the optimizer is. If it diverges, parallelism is necessary. Without this ablation, attributing stability to parallelism is an overstatement of what the experiments demonstrate. **The consequence.** A practitioner attempting to replicate or extend this work might reasonably conclude that parallelism is the key ingredient and apply it with a different optimizer (e.g., Adam, which is more common in later deep RL work). If Shared RMSProp's gradient statistics sharing was actually doing much of the stabilization work, switching to an optimizer without this property could cause training to fail or become much more brittle — even with 16 parallel threads. Conversely, if the optimizer is the key, then the entire parallelism infrastructure might be unnecessary for stability (though still valuable for wall-clock speedup), and a simpler single-thread setup with the right optimizer and perhaps a small replay buffer might suffice. **What evidence exists in the paper.** Supplementary Figure S5 shows that Shared RMSProp outperforms both per-thread RMSProp and Momentum SGD on two algorithms (n-step Q and A3C) across four games. The rank-sorted performance curves show Shared RMSProp achieving higher maximum scores and maintaining performance across a wider range of hyperparameter configurations. However, this comparison is conducted **only in the multi-threaded setting** — all three optimizers are tested with 16 threads. There is no single-thread condition for any optimizer, so the interaction between thread count and optimizer robustness is not measured. The paper does not report whether per-thread RMSProp or Momentum SGD would perform better or worse with fewer threads, or whether Shared RMSProp's advantage is specific to the multi-thread scenario. **Mitigation status.** The paper does not address this confound. The conclusion states that "using parallel actor-learners to update a shared model had a stabilizing effect on the learning process of the three value-based methods we considered" without acknowledging that the effect could be partially or primarily due to Shared RMSProp rather than the parallelism itself. The optimizer comparison in the supplementary material frames the question as "which optimizer works best for asynchronous training?" rather than "does asynchronous training provide stabilization beyond what the optimizer provides?" — so the critical ablation is not recognized as missing. A future study could resolve this with a 2×2 factorial experiment: {single-thread, multi-thread} × {replay buffer, no replay buffer}, all with Shared RMSProp, to isolate the effects. ### The Difficulty Generalization Gap: Atari Success Does Not Guarantee Broader Applicability **The assumption or constraint.** The paper's strongest and most quantitative results are on the Atari 2600 domain — 57 games, standardized evaluation protocols, comparison against multiple published baselines, detailed robustness analyses. The other domains (TORCS, MuJoCo, Labyrinth) receive substantially less rigorous treatment: fewer baselines, no quantitative comparison to the state of the art, and in the case of Labyrinth, no baselines at all. The Atari results are presented as evidence that the framework works broadly, but Atari has specific properties that may make it particularly amenable to the asynchronous approach: - **Fast simulation:** Atari games run at thousands of frames per second, so 16 parallel actors can generate millions of frames of experience rapidly. The framework's data inefficiency (discussed above) is masked by the sheer volume of data that can be generated cheaply. In domains with slower simulators, the lack of experience replay would hurt more. - **Discrete actions with small action spaces (typically 4-18 actions):** The value-based methods, which constitute three of the four algorithms, only work with discrete actions. The continuous-action extension is demonstrated only for A3C on MuJoCo, and even there, the evaluation is qualitative ("found good solutions") without benchmarking against published results for DDPG, TRPO, or other continuous-control algorithms that existed in 2016. - **Dense, frequent reward signals:** Most Atari games provide score changes on nearly every frame. The n-step return mechanism accelerates credit assignment, but the experiments do not test environments with extremely sparse or delayed rewards (e.g., Montezuma's Revenge, where the agent must explore extensively before finding any reward, is a known hard case — A3C LSTM achieves only 41.0 on this game, lower than Gorila's 84.0, as shown in Supplementary Table S3). - **Visual input with consistent structure:** The Atari preprocessing pipeline (grayscale, 84×84 resize, frame stacking) has been heavily tuned. Performance on domains with different input modalities (natural language, raw sensor streams, graph-structured state) is completely unexplored. **The consequence.** A practitioner working on a non-Atari problem — robotics, dialogue, recommendation systems, real-world control — has limited evidence that the asynchronous framework will transfer. The MuJoCo results suggest A3C can handle continuous control, but without quantitative baselines, a practitioner cannot estimate whether A3C would be competitive with purpose-built continuous-control algorithms like DDPG or TRPO on their specific task. The Labyrinth results demonstrate visual navigation in 3D, but the lack of any baseline makes the "approximately 50" score uninterpretable — is this near-optimal, mediocre, or achievable by random search? The paper's claim that A3C is "the most general and successful reinforcement learning agent to date" (Section 1) extrapolates well beyond the experimental evidence, which is heavily concentrated on a single benchmark family (Atari) with diminishing rigor as domains diverge from that benchmark. **What evidence exists in the paper.** The breadth of domains tested (Atari, TORCS, MuJoCo, Labyrinth) is a strength, but the depth of evaluation varies dramatically. Atari has 57 games with human-normalized scores, comparisons to 5 prior methods, and statistical robustness analysis (50 runs per game for hyperparameter sensitivity). TORCS has four configurations with all four methods and a human baseline, but no comparison to published TORCS results. MuJoCo has "a set of rigid body physics domains" with convergence curves and scatter plots, but no tables of final performance, no comparison to DDPG/TRPO, and the paper notes that "the rewards and thus performance are not comparable for most of the tasks due to changes made by the developers of Mujoco which altered the contact model" — making even approximate comparison impossible. Labyrinth has a single number (~50 average score) with no baselines or ablations. **Mitigation status.** The paper does not acknowledge this as a limitation. It presents the diversity of domains as evidence of generality without addressing the uneven rigor of evaluation across those domains. The claim of being "the most general" agent is asserted rather than tested — a claim of generality would require demonstrating that A3C matches or exceeds the best known method on each domain tested, which is not done for MuJoCo or Labyrinth. The paper does not discuss what properties of a domain make it suitable or unsuitable for the asynchronous framework. ### The LSTM Performance Is Not Well-Characterized **The assumption or constraint.** The paper introduces an LSTM variant of A3C that achieves the highest reported scores (623.0% mean human-normalized across 57 Atari games, Table 1), and the Labyrinth experiment uses an LSTM agent. However, the LSTM variant's behavior is not systematically analyzed — no ablation comparing LSTM vs. feedforward across different game types, no analysis of what the LSTM learns to remember, no measurement of how the LSTM affects training stability or sensitivity to hyperparameters, and no investigation of whether the LSTM's benefit is due to better handling of partial observability or some other factor. The paper notes only that the recurrent agent has "an additional 256 LSTM cells after the final hidden layer" (Section 5.1) and provides per-game scores in Supplementary Table S3. There is no discussion of how the LSTM is trained (truncated backpropagation through time over the n-step unrolling? full episode backpropagation?), how the hidden state is managed across thread synchronization steps, or whether the LSTM's memory interacts with the asynchronous update scheme in non-obvious ways. **The consequence.** A practitioner who wants to use A3C with recurrence — which is necessary for any domain with partial observability, time delays, or the need for memory — has almost no guidance from this paper. They do not know: whether the 256 LSTM cell size is tuned or arbitrary; whether the LSTM helps across all game types or only specific ones; how the training dynamics differ from the feedforward case (is Shared RMSProp still optimal? are different learning rates needed?); whether the LSTM's hidden state should be reset between episodes, between update intervals, or never; and whether the benefit of recurrence justifies the additional computational cost and implementation complexity. The per-game scores in Supplementary Table S3 reveal large variance: the LSTM dramatically outperforms feedforward A3C on Assault (14,497.9 vs. 5,474.9), Asteroids (5,093.1 vs. 4,474.5), Chopper Command (10,150.0 vs. 7,021.0), and Video Pinball (470,310.5 vs. 331,628.1), but underperforms feedforward on Amidar (173.0 vs. 263.9), Bank Heist (932.8 vs. 970.1), Boxing (37.3 vs. 59.8), and Centipede (1,997.0 vs. 3,755.8). This pattern — large gains on some games, noticeable regressions on others — is unexplained. The reader cannot predict whether their target domain would benefit from or be harmed by adding an LSTM. **What evidence exists in the paper.** Table 1 provides aggregate LSTM results (mean 623.0%, median 112.6%), and Supplementary Table S3 provides per-game raw scores. There is no LSTM-specific analysis beyond these numbers: no learning curves comparing LSTM and feedforward variants during training, no ablation of LSTM size or architecture, no investigation of whether the LSTM helps more on games requiring memory (e.g., games where critical objects leave the screen, games with delayed rewards, games with long-term dependencies). The Labyrinth experiment uses an LSTM but does not include a feedforward baseline, so the LSTM's contribution to maze exploration cannot be assessed. **Mitigation status.** The paper treats the LSTM as a straightforward architectural extension and reports its final performance without analysis. The unexplained performance regressions on some games (Amidar, Bank Heist, Boxing, Centipede, Robotank, Seaquest) are not discussed — the paper's narrative focuses on the mean score improvement, not the per-game volatility. The LSTM experiments do not appear in the robustness analysis (Figure 2 uses feedforward A3C only), the scalability analysis (Table 2, Figures 3-4 use feedforward methods only), or the optimizer comparison (Supplementary Figure S5 uses feedforward agents only). A reader interested in recurrent A3C must extrapolate from feedforward results with no evidence that the findings transfer. ### Target Network Update Timing and Gradient Accumulation Are Unexplored Hyperparameters **The assumption or constraint.** The paper fixes two critical temporal hyperparameters without systematic investigation: the target network update frequency (`I_target = 40,000` global frames for value-based methods) and the gradient accumulation interval (`t_max = 5` and `I_AsyncUpdate = 5` for all methods). These values are carried over from DQN (which used a target network update every 10,000 minibatch updates, roughly corresponding to 40,000 frames with its training regime) and chosen as a "reasonable" default, but their interaction with the asynchronous training dynamics is unexplored. The target network update frequency matters because it controls the staleness of the Q-learning targets. In the asynchronous setting with 16 threads, 40,000 global frames corresponds to only 2,500 frames per thread on average — much less experience than DQN's 40,000 frames from a single agent. Each thread therefore sees only a fraction of the data that DQN saw between target network updates, potentially making the target network update more frequent relative to per-thread experience. The optimal frequency might differ substantially from the DQN-derived value. The gradient accumulation interval `t_max = 5` controls the tradeoff between update frequency and gradient quality. Larger values reduce write conflicts in the Hogwild! scheme and provide more accurate n-step returns (longer horizons reduce bootstrap bias), but increase the staleness of the parameters used to generate the trajectory. With `t_max = 5` and an action repeat of 4, each thread generates 20 frames of gameplay between updates — roughly 0.33 seconds at 60 FPS, which is a very short horizon. Whether longer horizons (e.g., `t_max = 20` or `t_max = 50`) would improve performance through better credit assignment or harm it through increased staleness is unknown. **The consequence.** A practitioner applying these methods to a new domain has no basis for choosing these hyperparameters. Should `t_max` scale with the typical timescale of rewards in the environment? Should `I_target` scale with the number of threads? The paper provides only the fixed values and their performance, not the sensitivity of results to these choices. If performance degrades in a new domain, the practitioner cannot diagnose whether the problem is the algorithm itself or a poor choice of these temporal hyperparameters. The fact that all experiments use `t_max = 5` also means the maximum credit assignment horizon is very short — rewards more than 5 steps (20 frames) away from an action must propagate through the value function bootstrap rather than directly through the n-step return, potentially slowing learning in environments with long action-reward delays. **What evidence exists in the paper.** The paper provides no experiments varying `t_max`, `I_AsyncUpdate`, or `I_target`. The values are stated as fixed parameters in the experimental setup (Section 8 of Supplementary) and are not discussed in any ablation or sensitivity analysis. The n-step Q-learning and A3C algorithms both use `t_max` as the maximum n-step horizon, making this parameter central to their credit assignment mechanism, yet its effect on learning speed or final performance is never measured. The one exception is the MuJoCo experiments, where `t_max` is implicitly set to the full episode length (since the paper states episodes are batched into single updates without bootstrapping) — but this change is not analyzed as a hyperparameter choice; it is a consequence of short episodes. **Mitigation status.** The paper does not acknowledge these as unexplored dimensions. The values are presented as part of the standard experimental setup without discussion of their role or sensitivity. This is a common practice in empirical RL papers (inheriting hyperparameters from prior work without re-tuning), but it means the reported results are conditional on these specific choices, and the robustness of the framework to alternative temporal settings is unknown. Future work could investigate whether the framework's performance is sensitive to these parameters, and whether the optimal settings depend on environment characteristics (reward density, episode length, action-repeat factor) or the number of parallel threads. ## 7. Implications and Future Directions ### How This Work Changes the Landscape This paper caused a **paradigm shift** in deep reinforcement learning by demonstrating that experience replay — which the field had come to view as a *necessary* ingredient for stable training of deep neural network controllers — is actually just one mechanism among several for achieving the fundamental requirement of decorrelated training data. The shift was not incremental: it unlocked an entire family of algorithms (on-policy methods) that had been effectively off-limits for deep RL, and it reconfigured the hardware assumptions of the field from GPU-centric to CPU-possible. Before this paper, the dominant mental model in deep RL was a causal chain: stability requires decorrelated data → decorrelation requires a replay buffer → replay buffers require off-policy algorithms → therefore, deep RL equals off-policy Q-learning with experience replay. Every major result in the preceding three years — DQN, Double DQN, Dueling DQN, Prioritized Replay, Gorila — accepted this chain and innovated within it. The breakthroughs were about *better replay* (prioritization), *better Q-value estimation* (double, dueling), or *faster replay* (distributed architectures). None questioned the premise that replay was necessary. This paper broke the chain at the second link. It showed that decorrelation can come from **spatial diversity across parallel actors** rather than temporal diversity from stored transitions. This is a reframing of the problem at the level of mechanism rather than implementation: the agent doesn't need to remember its past to decorrelate its present; it just needs enough simultaneous experience in different states. The theoretical justification draws on Tsitsiklis (1994) and Recht et al. (2011), but the empirical demonstration — that this alternative mechanism actually works for training deep convolutional networks on 57 Atari games — is what changed the field's assumptions. The paradigm shift manifested in several concrete ways: **On-policy methods became viable for deep RL overnight.** The paper demonstrated that one-step Sarsa (on-policy value-based) trains stably and competitively with one-step Q-learning (off-policy) on Atari (Figure 1), and that A3C (on-policy actor-critic) outperforms all prior DQN variants including the heavily engineered Prioritized DQN (Table 1: 496.8% vs. 463.6% mean human-normalized). This was previously thought impossible — Huang et al.'s later work on self-correction would find analogous results, but in 2016 the consensus was that on-policy deep RL was too unstable to work. The paper didn't just claim this; it demonstrated it across multiple algorithms, multiple games, and 50-run robustness analyses showing that the methods don't collapse. **The hardware bottleneck shifted from "you need a GPU" to "a multi-core CPU can suffice."** DQN required 8-10 days on an Nvidia K40 GPU. Gorila required 130 machines. This paper showed that 16 CPU cores could achieve better results in less wall-clock time (4 days for A3C FF to reach 496.8%, versus Prioritized DQN's 8 GPU-days to reach 463.6%). This democratized deep RL research — any academic lab with a multi-core workstation could now train state-of-the-art agents, dramatically lowering the barrier to entry and accelerating the research cycle. The fact that the LSTM A3C variant achieved 623.0% mean human-normalized — 34% higher than the previous best published result — on CPU hardware made the practical implications impossible to ignore. **The exploration-exploitation tradeoff was recast as a parallel rather than sequential problem.** Traditional RL forces a single agent to balance exploration and exploitation over time (anneal ε from high to low). The asynchronous framework sidesteps this by running agents with different ε values simultaneously: highly exploratory agents discover new strategies while exploitative agents refine known ones, and the shared model integrates both types of experience in every update. This is a qualitatively different exploration paradigm that hadn't been systematically explored before. The superlinear speedups for one-step methods with more threads (24.1× for 16 threads in Table 2) suggest that parallel exploration heterogeneity provides benefits beyond pure computational throughput — the model actually learns more efficiently per frame with more diverse parallel actors. **The optimizer became recognized as part of the stabilization infrastructure.** The paper's finding that Shared RMSProp substantially outperforms both per-thread RMSProp and Momentum SGD (Supplementary Figure S5) revealed that in asynchronous training, the optimizer's internal state acts as an additional stabilization mechanism by aggregating gradient statistics across threads. This was not obvious a priori — the field treated optimizer choice as primarily about convergence speed, not stability. The paper showed that sharing the RMSProp statistics vector g across threads (updated without locks, Hogwild!-style) provides a form of implicit ensemble that smooths out per-thread gradient noise. This finding influenced subsequent work on large-scale distributed training beyond RL, contributing to the broader adoption of shared-statistics optimizers in asynchronous deep learning. **The paper reconciled contradictory intuitions about deep RL stability.** Before this work, there was a tension in the field: theory papers (Tsitsiklis, 1994) suggested asynchronous Q-learning should converge, but practical attempts to do online deep RL without replay buffers had largely failed. The paper resolved this by identifying *what* made those attempts fail — insufficient decorrelation in single-agent online training — and providing a mechanism (parallelism) that achieved sufficient decorrelation without a replay buffer. This explained both why DQN's replay buffer worked (it decorrelated data) and why parallelism could substitute for it (it also decorrelates data, through a different mechanism). The conceptual unification — decorrelation is the fundamental requirement, replay and parallelism are alternative implementations — provided a more principled understanding of deep RL stability than the field had before. However, the shift was not total. The paper did not render experience replay obsolete, and the authors explicitly said so in the conclusion. Replay buffers improve sample efficiency by reusing data, and the asynchronous framework throws away each transition after one use. The two mechanisms are complementary: parallelism provides decorrelation for stability; replay provides data reuse for efficiency. The paper opened the door to combining them, and subsequent work (such as ACER and Reactor) would do exactly that. The landscape shift was therefore not "replay is dead" but rather "replay is not the only path to stability, and removing it enables a much broader class of algorithms." ### Follow-Up Research This Work Enables **Combining asynchronous parallelism with experience replay for data-efficient on-policy learning.** The paper explicitly identifies this as the most natural next step: using parallel actors for stabilization and decorrelation, but retaining a replay buffer to reuse old data and improve sample efficiency. The tension is that on-policy methods (Sarsa, A3C) require current-policy data for their updates, so replaying old transitions from a different policy introduces off-policy bias. A strong follow-up would implement importance sampling corrections for the policy gradient when replaying off-policy data — computing the ratio π_current(a|s) / π_old(a|s) for replayed actions — and measure whether the combination of parallel decorrelation and corrected replay yields better sample efficiency than either mechanism alone. The experiment would compare four conditions on a set of Atari games and MuJoCo tasks: (1) A3C baseline (this paper), (2) A3C with uniform replay + importance sampling, (3) A3C with prioritized replay + importance sampling, and (4) the same architectures with double the number of parallel actors but no replay. The key metrics would be total environment frames to reach reference scores, not just wall-clock time — this paper's scalability analysis (Table 2, Figure 3) already showed that one-step methods achieve superlinear data efficiency scaling with more threads, and the question is whether replay can provide similar data efficiency gains for policy-based methods without destabilizing them. The negative result — importance sampling variance explodes and makes replay-assisted A3C worse than pure A3C — would be equally informative, clarifying that the on-policy/parallelism combination has a fundamental sample-efficiency floor that replay cannot easily overcome. **Distributed A3C across multiple machines using parameter servers.** The paper deliberately constrains itself to a single machine to exploit shared memory and avoid communication overhead, but this caps the number of parallel actors at the CPU core count (16 in the experiments, with A3C already showing sublinear 12.5× scaling). A distributed variant would replace Hogwild! shared-memory updates with asynchronous gradient transmission to parameter servers, similar to Gorila's architecture but without the per-actor replay buffers. The key research question is whether the stabilization benefits of parallelism survive the increased staleness and communication delays of a distributed setting. The experiment would scale A3C from 16 actors (single machine, as in this paper) to 100, 500, and 1000 actors distributed across multiple machines, measuring both wall-clock time to reach reference scores and total environment frames required. The hypothesis from this paper is that more actors provide more decorrelation (spatial diversity), which might partially compensate for increased staleness. But the sublinear scaling at 16 threads (12.5× speedup rather than 16×) suggests diminishing returns, and distributed communication overhead might accelerate the taper. The negative result — performance collapses beyond some threshold number of distributed actors — would define the practical limits of parallelism as a stabilizer and motivate hybrid approaches (e.g., local replay buffers within each actor, or synchronized parameter updates at intervals). The positive result — linear or near-linear scaling to hundreds of actors — would make A3C competitive with batch RL methods like PPO that later came to dominate distributed RL. **Adaptive difficulty-aware exploration via thread-specific policy parameterization.** The paper uses a simple mechanism for exploration diversity: value-based threads sample ε from a fixed discrete distribution, and A3C relies on entropy regularization with a fixed β = 0.01. A more principled approach would dynamically adjust each thread's exploration parameters based on its recent performance or the shared model's uncertainty. For example, threads that are achieving high episode returns could automatically reduce their entropy bonus (or ε) to exploit more, while threads with low returns could increase exploration. Alternatively, threads could be assigned different intrinsic motivation objectives — curiosity-driven exploration (maximizing prediction error of a learned dynamics model), count-based exploration (bonus rewards for rarely visited states), or skill diversity objectives — creating a heterogeneous population of specialized explorers that collectively cover the state space more efficiently. The experiment would compare fixed-exploration A3C (this paper's baseline) against adaptive-exploration variants on hard-exploration Atari games like Montezuma's Revenge, Pitfall, and Private Eye, where A3C performs relatively poorly (Supplementary Table S3: A3C LSTM scores 41.0, -135.7, and 421.1 respectively, far below human performance). The key metric would be whether adaptive exploration enables A3C to solve games that the fixed-exploration baseline cannot — this paper established that A3C works, but not that it explores effectively in sparse-reward environments, and adaptive exploration policies could close that gap. **Systematic ablation of the parallel stabilization hypothesis.** The paper's central causal claim — that parallel actor-learners provide a stabilizing effect that substitutes for experience replay — conflates parallelism with optimizer choice (Shared RMSProp), exploration diversity (different ε per thread), gradient accumulation (t_max = 5), and target networks (for value-based methods). A rigorous follow-up would isolate the contribution of each component through a factorial experiment: {single-thread, multi-thread (16)} × {Shared RMSProp, per-thread RMSProp, Adam} × {experience replay, no replay} × {diverse exploration, uniform exploration}, all on the five Atari games from Figure 1 with 50-run robustness sweeps. The critical conditions are: (a) single-thread + Shared RMSProp + no replay (does the optimizer alone provide enough stabilization?), (b) multi-thread + per-thread RMSProp + no replay (does parallelism stabilize without shared optimizer statistics?), and (c) single-thread + Shared RMSProp + replay (does combining optimizer and replay work better than either alone?). This experiment would answer whether the paper should be remembered for "parallelism stabilizes RL" or "Shared RMSProp stabilizes RL and parallelism accelerates it" — two very different takeaways with different implications for how practitioners should design their systems. The negative results matter as much as the positive ones: if single-thread + Shared RMSProp + no replay diverges on most runs, parallelism is indeed necessary for stability; if it works, the paper's framing overstates the role of parallelism and the optimizer deserves more credit. **Extension to hierarchical and goal-conditioned RL with asynchronous sub-policies.** The paper demonstrates that A3C can learn a general maze-exploration strategy in Labyrinth (Section 5.4), but the agent learns a single flat policy for the entire task. The asynchronous framework's natural support for multiple parallel actors with different exploration behaviors makes it well-suited for hierarchical RL, where different threads could learn different sub-policies (options, skills, or goal-conditioned behaviors) and a higher-level controller could learn to sequence them. For example, in the Labyrinth task, one thread could specialize in corridor following, another in room exploration, and another in portal-seeking behavior, with the shared model integrating all of these into a reusable skill library. The experiment would extend A3C with a hierarchical architecture where the policy outputs both a sub-policy selection and primitive actions, trained with the same asynchronous framework but with different threads biased toward different sub-policies through auxiliary objectives (e.g., maximizing visitation of different regions of the state space). The Labyrinth environment is ideal for this because the randomly generated mazes require transferable exploration skills — a hierarchical agent that learns reusable "explore room," "follow corridor," "approach apple" sub-policies should outperform a flat agent by transferring these behaviors across maze layouts. The baseline would be the flat A3C LSTM from this paper (~50 average score); the hypothesis is that hierarchical A3C achieves higher scores and generalizes to larger mazes that the flat agent cannot handle. **Application to domains with expensive simulators: combining asynchronous actors with model-based planning.** The paper identifies that the asynchronous framework is sample-inefficient by design — each transition is used once and discarded. For domains like TORCS (Section 5.2) or MuJoCo (Section 5.3) where simulation is more expensive than Atari, this becomes a bottleneck. A promising direction is to combine asynchronous data collection with learned environment models: each actor thread maintains or shares a learned dynamics model, uses it for planning or generating synthetic rollouts, and the shared policy is updated from both real and simulated experience. The asynchronous framework is well-suited for this because the parallel actors already provide diverse real-world data for training the dynamics model, and the model's rollouts can further decorrelate the policy updates (by generating on-policy synthetic trajectories branching from diverse starting states). The experiment would compare pure A3C against model-augmented A3C on the TORCS benchmark from this paper, measuring both final performance and sample efficiency (environment frames to reach human performance). The paper already notes that TORCS training takes 12-40 hours on CPU; a model-based variant that reduces this to, say, 2-4 hours would be a substantial practical improvement. The risk — that learned model errors compound and hurt policy learning — is real and would need to be addressed through uncertainty-aware planning or short-horizon model rollouts. ### Practical Applications and Downstream Use Cases **On-device and edge deployment of RL agents without GPU hardware.** This paper's most immediate practical implication is that reinforcement learning training no longer requires specialized GPU hardware. A standard multi-core CPU workstation can train state-of-the-art agents on complex visual tasks — A3C FF achieved 496.8% mean human-normalized on 57 Atari games in 4 days on 16 CPU cores (Table 1). For organizations deploying RL in settings where GPUs are unavailable or impractical — embedded systems, robotics with onboard computation, field-deployed devices with only CPU resources — the asynchronous framework provides a training recipe that works on commodity hardware. The LSTM variant's success on Labyrinth (Section 5.4) with purely visual input (84×84 RGB images) suggests that visual navigation policies can be learned on-CPU for real-world applications like warehouse robots or drone navigation, where the training can happen on a multi-core server and the resulting policy can run on modest embedded hardware. **Accelerated research iteration cycles for RL algorithm development.** The one-day A3C result (344.1% mean, Table 1) matching Dueling Double DQN's final performance after 8 days of GPU training represents an approximately 8× reduction in training time for comparable results, on cheaper hardware. For RL researchers, this means testing a new algorithmic idea on a full 57-game Atari benchmark drops from "wait a week" to "wait a day" — a difference that transforms the feasible experiment throughput of a research lab. The robustness analysis (Figure 2) showing that A3C tolerates a wide range of learning rates (LogUniform(10^(-4), 10^(-2))) without catastrophic failure further reduces the burden of hyperparameter tuning, meaning researchers can run fewer experiments per hypothesis and still get reliable signals. The fact that the framework works across value-based, policy-based, on-policy, and off-policy methods (Figure 1) means researchers can prototype new algorithms in any of these families without building separate stabilization infrastructure for each one. **Continuous control for robotics with visual input on CPU clusters.** The MuJoCo results (Section 5.3, Supplementary Figures S7, S8) demonstrate that A3C can learn continuous motor control policies from both low-dimensional state and pixel input, typically in under 24 hours on CPU. For robotics labs that do not have access to GPU clusters but do have multi-core CPU servers, this provides a practical path to training visuomotor policies — controlling robot arms, grippers, or locomotion from camera input — without specialized hardware. The paper shows that for tasks like pendulum, pointmass2D, and gripper control from pixels, A3C finds solutions within 24 hours on CPU (Supplementary Section 9). While this is slower than GPU-based training, it makes visuomotor policy learning accessible to groups with only CPU infrastructure, which was not previously possible with DQN-style methods (which required GPUs for timely training). **General-purpose agent training for game AI and simulation environments.** The paper's demonstration that A3C handles discrete actions (Atari), continuous actions (MuJoCo), visual input (Atari, TORCS, Labyrinth), recurrent memory (LSTM variant), and both 2D and 3D environments makes it a compelling single-algorithm solution for training agents across diverse game and simulation platforms. For game development studios or simulation companies needing AI for multiple different game types or simulation scenarios, A3C provides a unified training recipe that does not require per-domain algorithm selection or specialized hardware. The TORCS results (Section 5.2, Supplementary Figure S6) show that A3C reaches 75-90% of human performance across different car configurations (slow/fast, with/without opponents) from visual input alone, suggesting it can handle the visual complexity and continuous control aspects of driving simulations without domain-specific engineering. The Labyrinth results (Section 5.4) extend this to 3D maze navigation with random layouts, demonstrating transfer to novel environment configurations — a key requirement for simulation-based training where the agent must generalize to unseen scenarios. ### When to Prefer This Method The paper explicitly positions the asynchronous framework against two alternatives: **GPU-based DQN with experience replay** (Mnih et al., 2015 and its variants) and **distributed architectures like Gorila** (Nair et al., 2015). Based on the experimental evidence and the paper's own discussion, the decision criteria are: **Prefer asynchronous A3C over GPU-based DQN variants when:** - Hardware is limited to multi-core CPUs (no GPU available, or GPU resources are scarce/expensive). The 4-day CPU result (496.8% mean) vs. 8-day GPU result (463.6% for Prioritized DQN) demonstrates that CPU training can be both faster and better when wall-clock time and hardware cost are the relevant metrics (Table 1). - The problem requires continuous actions (MuJoCo tasks, robotics). Value-based methods like DQN cannot handle continuous action spaces without a separate optimization over actions at each step; A3C handles them natively by outputting distribution parameters (Section 5.3, Supplementary Section 9). - On-policy learning is preferred for theoretical or practical reasons (safer exploration in stochastic environments, compatibility with entropy regularization for exploration, or domain constraints that make off-policy corrections high-variance). The paper demonstrates that one-step Sarsa performs comparably to one-step Q-learning (Figure 1), and A3C outperforms all value-based methods, establishing that on-policy methods are not at a disadvantage in the asynchronous framework. - Training time is the bottleneck and rapid iteration matters. The one-day A3C result (344.1% mean, matching Dueling Double DQN's 8-day performance) shows an 8× reduction in the time to get competitive results (Table 1). **Prefer asynchronous A3C over distributed architectures (Gorila) when:** - Infrastructure is limited to a single machine (no cluster available, or distributed system overhead is unacceptable). The paper achieves better results than Gorila (A3C FF: 496.8% mean, 116.6% median vs. Gorila: 215.2% mean, 71.3% median; Table 1) on a single 16-core CPU machine rather than 130 machines, representing a dramatic reduction in operational complexity. - Communication overhead of distributed training is a concern. The Hogwild! shared-memory approach eliminates network communication, gradient serialization, and parameter server management, simplifying implementation and debugging substantially. - The number of parallel actors needed fits within a single machine's core count (the paper tests up to 16 threads; scaling beyond this is unexplored and may not work without rearchitecting). **Prefer experience-replay-based methods (DQN variants) over asynchronous methods when:** - Sample efficiency is critical — environment interactions are expensive (real-world robotics, expensive simulators) and data reuse matters. The asynchronous framework uses each transition once; experience replay reuses transitions many times. The paper explicitly notes this tradeoff: "Incorporating experience replay into the asynchronous reinforcement learning framework could substantially improve the data efficiency of these methods by reusing old data" (Section 6). No head-to-head sample efficiency comparison is provided, but the mechanism strongly favors replay when data is scarce. - The environment has sparse or delayed rewards where the n-step return horizon (t_max = 5 in all experiments) may be insufficient. Prioritized experience replay can resample rare reward events, while the asynchronous framework's forward-view n-step returns are limited to the gradient accumulation window. - Off-policy learning is specifically required (e.g., learning from demonstration data, batch RL from fixed datasets, or offline RL settings where the agent cannot interact with the environment during training). The asynchronous framework generates all training data online from the current policy. **Prefer distributed architectures (Gorila-style) over single-machine asynchronous when:** - The required number of parallel actors exceeds what fits on one machine (e.g., hundreds or thousands of environments needed for adequate throughput). The paper's sublinear scaling for A3C at 16 threads (12.5× speedup; Table 2) suggests diminishing returns within a single machine, but distributed architectures can scale to more actors at the cost of communication overhead. The paper does not test scaling beyond 16 threads, so this boundary is undefined and would need empirical validation. - The environment simulation is the bottleneck (not the neural network updates), and many parallel simulators are needed across multiple machines to generate sufficient experience. The single-machine approach is limited by the CPU's simulation throughput.