ArXiv: 1707.06347
🎯 Pitch
PPO matches the reliability of trust-region methods while using only first-order optimization, by simply clipping the policy update so it cannot move too far from the original—a trick that is trivially implemented in a few lines of code yet outperforms far more complex algorithms on continuous control and Atari benchmarks.
1. Executive Summary
This paper introduces proximal policy optimization (PPO), a new family of policy gradient methods for reinforcement learning that alternate between sampling data through environment interaction and optimizing a surrogate objective via stochastic gradient ascent. PPO is evaluated on continuous control tasks using the MuJoCo physics engine in OpenAI Gym and on 49 Atari games in the Arcade Learning Environment. The core mechanism is a clipped surrogate objective (a lower bound on the standard policy gradient objective achieved by clipping probability ratios at 1 − ε or 1 + ε to remove incentives for destructively large policy updates), which is compared against an adaptive KL penalty coefficient alternative (a heuristic that adjusts the penalty weight β to maintain a target KL divergence between old and new policies). PPO with clipping achieves a 0.82 average normalized score on the 7-task MuJoCo benchmark (versus −0.39 for unconstrained policy gradient and 0.74 for the best adaptive KL variant) and wins 30 out of 49 Atari games measured by average reward over all of training, establishing that a simple first-order method can match or exceed the data efficiency and reliability of trust region methods like TRPO while being compatible with architectures that include noise or parameter sharing between the policy and value function—a combination that TRPO cannot directly support.
2. Context and Motivation
The Core Problem: Policy Updates Are Either Inefficient or Unstable
The fundamental challenge this paper tackles is a central tension in deep reinforcement learning: how do you update a neural network policy using sampled data without the update destroying what the policy has already learned? This question matters because reinforcement learning with function approximation is inherently iterative—you collect data with a current policy, then use that data to compute a better policy, then collect new data with the improved policy, and repeat. At each step, the data is on-policy (generated by the current policy parameters), which means that after you update the policy, the data becomes stale. Using stale data to compute further updates leads to the policy optimizing a misleading signal—it thinks certain actions are good or bad based on data from an older version of itself, not its current behavior.
This creates a painful tradeoff that every policy gradient practitioner faces:
-
Take one gradient step and discard the data. This is safe because the data is fresh—it accurately reflects what the current policy would do. But it is horribly sample-inefficient. You might collect thousands of timesteps of experience, take a single weight update, and then need to collect thousands more timesteps before your next update. For complex tasks requiring millions of timesteps of training (robotic locomotion, Atari games), this means most of the computation goes into data collection rather than optimization. The policy learns slowly.
-
Take multiple gradient steps on the same data. This is sample-efficient—you squeeze more learning out of each batch of collected experience. But it is dangerous. After the first update, the policy has changed. The data was generated by the old policy. The advantage estimates in Equation (1) tell you how much better or worse an action was relative to the old policy's expected performance. After you change the policy, those advantage estimates are no longer accurate for the new policy. Taking additional gradient steps based on stale advantage estimates can lead to destructively large policy updates: the optimization process sees a signal that suggests moving the policy parameters in a direction that would have been good for the old policy, but is catastrophic for the new one. The policy can collapse—suddenly performing worse than random—and never recover.
The paper states this explicitly in Section 2.1:
"While it is appealing to perform multiple steps of optimization on this loss using the same trajectory, doing so is not well-justified, and empirically it often leads to destructively large policy updates."
This is the paper's central motivating gap: there is no principled, simple method that allows multiple epochs of minibatch updates on the same batch of on-policy data while guaranteeing that the policy does not change too much. Filling this gap would combine the sample efficiency of reusing data with the stability of conservative updates.
Why This Problem Matters: Scalability, Simplicity, and Robustness
The authors frame the importance of this problem along three practical dimensions stated in the introduction:
Scalability. Modern deep RL systems run on distributed hardware with multiple actors collecting data in parallel. If the optimizer can perform many updates per batch of collected data, the ratio of optimization compute to data-collection compute increases. This is practically significant because data collection (running the policy in the environment) is often the bottleneck—simulators may be slow, or real-world interaction may be limited. Being able to train a policy to high performance with fewer total environment interactions directly reduces cost and wall-clock time. PPO's design—alternating between collecting a fixed-length trajectory segment and performing K epochs of minibatch SGD—naturally maps onto parallel actor-collector architectures (Algorithm 1).
Simplicity. Deep RL methods have a reputation for being finicky. Hyperparameters that work on one task fail on another. Implementation details (how you clip gradients, how you normalize advantages, how you initialize the network) matter enormously but are rarely documented. TRPO, the leading stable method at the time of this paper's publication, requires:
- Computing the Fisher information matrix and its matrix-vector products
- Approximately solving a constrained optimization problem using conjugate gradient descent
- Performing a backtracking line search to enforce the KL constraint
This stack of numerical machinery makes TRPO difficult to implement correctly from scratch, hard to debug when things go wrong, and incompatible with certain architectural choices (dropout, parameter sharing) that complicate the computation of the Fisher matrix. The authors argue that a method requiring "only few lines of code change to a vanilla policy gradient implementation" would have outsized practical impact because more researchers and engineers could adopt it reliably.
Robustness. The third criterion is that the method should "succeed on a variety of problems without hyperparameter tuning." This is the holy grail of RL algorithm design. The introduction notes that Q-learning "fails on many simple problems and is poorly understood," vanilla policy gradient "has poor data efficiency and robustness," and TRPO "is relatively complicated." A method that works out of the box across continuous control and discrete action spaces (Atari) with minimal per-task tuning would lower the barrier to applying deep RL to new problems.
Prior Approaches and Where They Fall Short
The paper positions itself against three families of methods, each with specific, documented shortcomings.
Vanilla Policy Gradient: Sample-Inefficient and Fragile
The standard policy gradient objective (Equation 2) is:
This objective is optimized by the gradient estimator in Equation (1). The key limitation is that this objective is only justified for a single gradient step on data collected from . The paper is explicit about what happens when you try to reuse data:
"While it is appealing to perform multiple steps of optimization on this loss using the same trajectory, doing so is not well-justified, and empirically it often leads to destructively large policy updates (see Section 6.1; results are not shown but were similar or worse than the 'no clipping or penalty' setting)."
The reference to Section 6.1 and the "no clipping or penalty" setting is damning: that baseline achieves a −0.39 average normalized score across 7 MuJoCo environments (Table 1)—worse than the random policy, which is normalized to 0. This means that naively taking multiple gradient steps on the same data literally destroys the policy on at least one environment (HalfCheetah, as noted in the text).
The theoretical reason for this fragility is that can change arbitrarily as moves away from the data-collection parameters. The gradient is large when an action that was unlikely under the old policy becomes more likely under the new policy—this can happen suddenly if the neural network's decision boundary shifts. With one gradient step, this is manageable. With ten steps, the new policy can become almost completely uncorrelated with the old policy, and the advantage estimates —which were computed assuming the old policy's action distribution—become meaningless.
Trust Region Policy Optimization (TRPO): Powerful but Complex and Restrictive
TRPO [Sch+15b] was the state-of-the-art method for stable policy optimization at the time. It solves a constrained optimization problem at each iteration:
The constraint limits the average KL divergence between the old and new policies to at most , preventing the destructively large updates that plague vanilla policy gradient. This works well empirically—TRPO achieves stable, monotonic improvement on continuous control tasks.
However, the paper identifies three specific pain points with TRPO:
1. Implementation complexity. TRPO requires second-order optimization. The constrained problem is solved by:
- Making a linear approximation to the surrogate objective (first-order Taylor expansion)
- Making a quadratic approximation to the KL constraint (second-order Taylor expansion, which involves the Fisher information matrix)
- Solving the resulting linearly-constrained quadratic program using conjugate gradient, which requires computing Fisher-vector products (not just gradients) at each conjugate gradient iteration
- Performing a backtracking line search on the full nonlinear objective and constraint to ensure the quadratic approximation was valid
This is substantially more code and more numerical machinery than a simple SGD loop. Each piece (Fisher-vector products, conjugate gradient, line search) introduces its own hyperparameters and potential failure modes.
2. Incompatibility with dropout and parameter sharing. The Fisher information matrix computation assumes a specific probabilistic structure in the network. Dropout—a widely used regularization technique—injects noise into the network activations, which breaks the clean relationship between the policy distribution and the network parameters. Similarly, sharing parameters between the policy and value function (a common architecture choice for efficiency, used extensively in A3C and related methods) entangles the policy gradient and value function loss, making it unclear how to compute the Fisher matrix for just the policy part. The paper notes that TRPO "is not compatible with architectures that include noise (such as dropout) or parameter sharing (between the policy and value function, or with auxiliary tasks)."
3. Constraint vs. penalty mismatch. The theory that justifies TRPO (from Kakade and Langford, 2002; the "CPI" objective) actually suggests using a penalty on KL divergence rather than a hard constraint:
This would be a first-order method—just add a KL penalty term to the objective and run SGD. But TRPO uses a constraint instead because:
"TRPO uses a hard constraint rather than a penalty because it is hard to choose a single value of β that performs well across different problems—or even within a single problem, where the characteristics change over the course of learning."
The optimal penalty weight depends on the scale of the rewards, the policy's current performance, and the local curvature of the optimization landscape—all of which change during training. The paper explicitly states that "it is not sufficient to simply choose a fixed penalty coefficient β and optimize the penalized objective Equation (5) with SGD; additional modifications are required." Table 1 confirms this: Fixed KL with β = 0.3 scores 0.62, β = 1.0 scores 0.71, β = 3.0 scores 0.72, β = 10.0 scores 0.69. While some β values work reasonably well, none match the clipped surrogate (0.82), and the optimal β varies across environments within the benchmark—a single fixed β is not robust.
Other Methods: Different Tradeoffs, Different Problems
The introduction briefly references the broader landscape:
Q-learning and DQN "fails on many simple problems and is poorly understood" when applied to continuous control. The authors cite the fact that "while DQN works well on game environments like the Arcade Learning Environment with discrete action spaces, it has not been demonstrated to perform well on continuous control benchmarks such as those in OpenAI Gym." This is a known limitation of value-based methods: they require maximizing over the action space to compute the target value, which is straightforward for discrete actions but requires a separate optimization procedure for continuous actions.
A3C/A2C (Mnih et al., 2016) uses vanilla policy gradient updates with multiple parallel actors to decorrelate data, but each actor still performs only one gradient update per trajectory segment. It achieves stability through asynchronous parallelism rather than through a principled trust region mechanism, which means its sample efficiency is limited—it needs many parallel actors collecting data to keep the optimization stable.
ACER (Wang et al., 2016) achieves better sample efficiency through experience replay and off-policy corrections (importance sampling with a bias-variance tradeoff and a trust region constraint implemented via a Retrace operator), but the paper notes that PPO performs similarly to ACER on Atari while being "much simpler." The complexity of ACER comes from its off-policy machinery: truncated importance sampling weights, a trust region applied in the Q-function update via a moving average policy, and a separate deterministic policy network for the Retrace computation.
Natural policy gradient methods (closely related to TRPO) require second-order information and face the same complexity and compatibility issues.
How PPO Positions Itself
The paper positions PPO as a direct response to this landscape: a method that achieves the data efficiency and reliable performance of TRPO while using only first-order optimization (stochastic gradient descent), being simple to implement, and being compatible with dropout and parameter sharing. The key insight is that you don't need the full machinery of second-order optimization and hard constraints to prevent destructive policy updates—a cleverly designed surrogate objective with a clipping mechanism can achieve the same effect by forming a pessimistic lower bound on the policy's performance.
The paper frames the clipped surrogate as an alternative to the KL constraint/penalty paradigm entirely. Rather than constraining or penalizing the divergence between old and new policies (which requires measuring KL divergence), the clipping mechanism directly modifies the objective so that when the probability ratio moves too far from 1 in a way that would increase the objective, the gradient is cut off. The policy update receives no incentive to keep moving further away. This is fundamentally a first-order mechanism—it operates on the probability ratio , which is computed directly from the policy's forward pass, with no KL divergence computation required.
The paper also positions the adaptive KL penalty variant (Section 4) as an important baseline that demonstrates why the clipping approach is necessary. The adaptive KL method adjusts to keep the KL divergence near a target value , which partially addresses the problem of choosing . However, the paper explicitly states that this variant performs worse than clipping in the experiments (Section 4: "we found that the KL penalty performed worse than the clipped surrogate objective")—validating the claim that simply penalizing KL divergence, even adaptively, is less effective than the clipping mechanism.
Finally, the paper positions PPO as a practical engineering contribution as much as a theoretical one. The emphasis on "few lines of code change to a vanilla policy gradient implementation," the detailed hyperparameter tables (Tables 3–5), and the testing across two vastly different domains (MuJoCo continuous control with MLP policies and Atari with CNN policies) all underscore that PPO is designed for immediate adoption by practitioners. This is not a paper proposing a new theoretical framework for policy optimization—it is a paper proposing a mechanism that works reliably and can be plugged into existing codebases with minimal changes.
3. Technical Approach
3.1 Reader Orientation
This paper introduces Proximal Policy Optimization (PPO), a family of policy gradient algorithms that train a neural network to select actions in reinforcement learning environments without destroying what it has already learned. The problem it solves is the tension between sample efficiency and stability: reusing sampled data for multiple gradient updates is efficient but causes catastrophically large policy changes, while single-update approaches are stable but waste data. PPO's solution is a surrogate objective function with a clipping mechanism that automatically prevents the policy from changing too much during optimization, allowing multiple epochs of minibatch updates on the same batch of collected experience while guaranteeing bounded policy change—all using only first-order gradient information with no second-order derivatives, conjugate gradient solvers, or hard constraints.
3.2 Big-Picture Architecture (Diagram in Words)
The system has four major components connected in a loop:
-
Environment simulators (N parallel actors) — each runs the current policy
$\pi_{\theta_{\text{old}}}$for$T$timesteps, collecting sequences of states, actions, rewards, and value predictions. These actors execute independently in parallel, decorrelating the data. -
Advantage estimator — processes each collected trajectory segment to compute
$\hat{A}_1, \ldots, \hat{A}_T$, estimates of how much better or worse each action was compared to the policy's expected performance (using generalized advantage estimation, or GAE, with parameters$\gamma$and$\lambda$). These advantage estimates serve as the signal that tells the optimizer which actions to encourage and which to discourage. -
Surrogate objective constructor — takes the collected data (states, actions, old policy probabilities) and the advantage estimates, and builds the PPO loss function
$L^{\text{CLIP}}(\theta)$that will be optimized. This is where the clipping mechanism operates: it computes the probability ratio$r_t(\theta) = \pi_\theta(a_t \mid s_t) / \pi_{\theta_{\text{old}}}(a_t \mid s_t)$and clips it when it moves too far from 1, preventing the optimizer from being rewarded for making excessive policy changes. -
Policy optimizer (SGD with Adam) — performs
$K$epochs of minibatch stochastic gradient ascent on the surrogate objective over the collected$NT$timesteps of data. After optimization, the updated policy parameters$\theta$replace$\theta_{\text{old}}$, and the loop repeats with new data collected from the updated policy.
Information flows clockwise: actors collect data using $\pi_{\theta_{\text{old}}}$ → the advantage estimator computes $\hat{A}_t$ values → the surrogate objective constructor builds $L^{\text{CLIP}}$ using the data and advantages → the optimizer updates $\theta$ by ascending the gradient of $L^{\text{CLIP}}$ → $\theta_{\text{old}}$ is set to $\theta$ → actors collect new data. For implementations with shared policy-value networks, the objective is augmented with a value function error term and an entropy bonus.
3.3 Roadmap for the Deep Dive
- First, the surrogate objective landscape and the probability ratio
$r_t(\theta)$, since this is the mathematical object that captures how much the policy has changed, and all of PPO's mechanisms operate on it. - Second, the clipped surrogate objective
$L^{\text{CLIP}}$— the paper's primary contribution and the mechanism that enables stable multi-epoch optimization. We will build understanding by starting with the unconstrained CPI objective, showing why it fails, then introducing the clip operator and the min, and explaining the pessimistic lower bound property. - Third, the adaptive KL penalty coefficient alternative, which serves as an important baseline and illustrates why clipping outperforms penalizing KL divergence directly, even with an adaptive schedule.
- Fourth, the complete PPO loss function with value function error and entropy bonus, showing how the policy objective fits into a full actor-critic training loop with shared or separate networks.
- Fifth, the data collection and optimization algorithm, including the fixed-length trajectory segment architecture, the truncated GAE advantage estimator, and the distributed actor-collector loop described in Algorithm 1.
- Sixth, the hyperparameter configurations and their justifications, connecting the algorithmic choices to the concrete numbers used in the MuJoCo, Roboschool, and Atari experiments.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily an algorithm design paper whose core idea is that a simple first-order clipping mechanism on the probability ratio can provide the same stable policy improvement guarantees as trust region methods, enabling multiple epochs of minibatch SGD on the same batch of data while preventing destructively large policy updates.
The Probability Ratio and Why It Matters
At the heart of PPO is a single scalar quantity computed for each timestep in the collected data:
where $\pi_\theta(a_t \mid s_t)$ is the probability the current policy assigns to the action that was actually taken at timestep $t$ in state $s_t$, and $\pi_{\theta_{\text{old}}}(a_t \mid s_t)$ is the probability that the old policy (the one used to collect the data) assigned to that same action.
What it computes: For each timestep $t$, $r_t(\theta)$ measures how much more (or less) likely the current policy is to take the exact same action $a_t$ in the same state $s_t$ compared to the policy that originally took that action. When $\theta = \theta_{\text{old}}$, every $r_t(\theta) = 1$. When the new policy makes action $a_t$ twice as likely, $r_t(\theta) = 2$. When it makes the action half as likely, $r_t(\theta) = 0.5$.
Why this form: The probability ratio appears naturally when deriving policy gradient estimators via importance sampling. The key insight is that if we want to estimate how good a new policy $\pi_\theta$ is using data collected from an old policy $\pi_{\theta_{\text{old}}}$, we can reweight each action by $\frac{\pi_\theta}{\pi_{\theta_{\text{old}}}}$ to correct for the distribution shift. The vanilla policy gradient objective $\hat{\mathbb{E}}_t[\log \pi_\theta(a_t \mid s_t) \hat{A}_t]$ can be rewritten using this ratio to obtain the CPI (Conservative Policy Iteration) objective:
The CPI objective is equivalent to the policy gradient objective to first order around $\theta_{\text{old}}$ (both have the same gradient when $r=1$), but they differ when $\theta$ moves away from $\theta_{\text{old}}$. The ratio form makes the connection to importance sampling explicit and is the natural starting point for trust-region-style improvements because it directly quantifies how much the new policy deviates from the data-collection policy at each timestep.
The Problem: Unconstrained Maximization of $L^{\text{CPI}}$ Causes Destructive Updates
To understand why the clipping mechanism is necessary, we first need to understand catastrophic failure occurs without it. The "No clipping or penalty" baseline in Section 6.1 corresponds to directly maximizing:
without any constraints, penalties, or clipping on the probability ratio.
What goes wrong. Consider a timestep where the advantage estimate $\hat{A}_t > 0$ (the action was better than expected). The objective $r_t(\theta) \hat{A}_t$ says: "make this action more likely." There is no upper bound on $r_t(\theta)$ — if the optimizer can increase $\pi_\theta(a_t \mid s_t)$ from, say, 0.01 to 0.99, the ratio $r_t(\theta)$ jumps to 99, multiplying the advantage by 99× and producing an enormous gradient signal. The optimizer will eagerly exploit this: take a large step, increase the action probability dramatically, and the objective value soars. But after this large step, the policy has changed so much that the advantage estimates $\hat{A}_t$ — which were computed assuming the old policy's action distribution — are no longer valid. The optimizer has overfit to a signal that was only accurate for the old policy. On subsequent iterations, the policy may discover that the actions it now takes with high probability are actually terrible under its new behavior, leading to a collapse in performance.
When $\hat{A}_t < 0$ (the action was worse than expected), the symmetric problem occurs: the objective says "make this action less likely," and the optimizer can drive $r_t(\theta) \to 0$ arbitrarily quickly, again producing a large policy change based on stale advantage estimates.
Empirical evidence. Table 1 reports an average normalized score of −0.39 for the "No clipping or penalty" variant, where a score of 0 corresponds to the random policy and 1 corresponds to the best result. A negative score means the policy became worse than random — on at least one environment (HalfCheetah), the policy was destroyed by the unconstrained updates. This is not a marginal degradation; it is a complete training failure.
The paper notes that this failure mode is fundamental, not a hyperparameter sensitivity issue:
"While it is appealing to perform multiple steps of optimization on this loss
$L^{PG}$using the same trajectory, doing so is not well-justified, and empirically it often leads to destructively large policy updates."
The phrase "destructively large" is chosen carefully — the updates don't just cause some performance reduction; they literally destroy the policy's ability to perform the task.
The Clipped Surrogate Objective: $L^{\text{CLIP}}$
PPO's central contribution is the clipped surrogate objective, which solves the destructive update problem using only first-order information (no KL divergence computation required):
where $\epsilon$ is a hyperparameter controlling how far the probability ratio is allowed to deviate from 1 before it is clipped (typically $\epsilon = 0.2$), $r_t(\theta)$ is the probability ratio defined above, $\hat{A}_t$ is the advantage estimate at timestep $t$, and $\text{clip}(x, a, b)$ clamps $x$ to the interval $[a, b]$, returning $a$ if $x < a$, $b$ if $x > b$, and $x$ otherwise.
What it computes — the two cases:
The behavior of $L^{\text{CLIP}}$ depends on the sign of the advantage $\hat{A}_t$. We analyze each case separately, following the logic illustrated in Figure 1.
Case 1: Positive advantage ($\hat{A}_t > 0$). The action was better than expected, so the objective should encourage the policy to make this action more likely. The unclipped term $r_t(\theta) \hat{A}_t$ says: increase $r_t(\theta)$ as much as possible. The clipped term $\text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t$ says: increase $r_t(\theta)$ up to $1+\epsilon$, but no further — any increase beyond $1+\epsilon$ gets ignored because the clip operator returns $1+\epsilon$. The $\min$ takes the smaller of these two values. Since both terms are proportional to $\hat{A}_t > 0$:
- When
$r_t(\theta) \leq 1+\epsilon$: both terms grow with$r_t(\theta)$, the$\min$equals the unclipped term, and the gradient encourages increasing$r_t(\theta)$. - When
$r_t(\theta) > 1+\epsilon$: the unclipped term continues growing, but the clipped term saturates at$(1+\epsilon) \hat{A}_t$. The$\min$now equals the clipped (smaller) term, which is constant with respect to further increases in$r_t(\theta)$. The gradient is zero.
The result: the optimizer has incentive to make good actions up to $1+\epsilon = 1.2$ times more likely than the old policy, but no incentive to make them even more likely. The policy is prevented from concentrating excessively on a single action that looked good under stale data.
Case 2: Negative advantage ($\hat{A}_t < 0$). The action was worse than expected, so the objective should discourage the policy from taking this action. The unclipped term $r_t(\theta) \hat{A}_t$ says: decrease $r_t(\theta)$ as much as possible (driving it toward 0). The clipped term says: decrease $r_t(\theta)$ down to $1-\epsilon$, but no further — any decrease beyond $1-\epsilon$ gets ignored. Since $\hat{A}_t$ is negative, $r_t(\theta) \hat{A}_t$ becomes more negative as $r_t(\theta)$ decreases, making the unclipped term smaller (more negative) than the clipped term. Therefore:
- When
$r_t(\theta) \geq 1-\epsilon$: both terms decrease as$r_t(\theta)$decreases, the$\min$equals the unclipped term (which is more negative, hence smaller), and the gradient encourages decreasing$r_t(\theta)$. - When
$r_t(\theta) < 1-\epsilon$: the clipped term saturates at$(1-\epsilon) \hat{A}_t$while the unclipped term continues decreasing, so the clipped term is now larger than the unclipped term. The$\min$equals the unclipped term — but wait, re-examining: when$r_t(\theta) < 1-\epsilon$and$\hat{A}_t < 0$, then$r_t(\theta) \hat{A}_t$is less negative (closer to zero) than$(1-\epsilon) \hat{A}_t$? Let's check:$\hat{A}_t < 0$, say$\hat{A}_t = -1$. Then$r_t(\theta) \hat{A}_t = -r_t(\theta)$, and$(1-\epsilon) \hat{A}_t = -(1-\epsilon)$. If$r_t(\theta) = 0.5$and$\epsilon = 0.2$, then$-0.5$is greater than$-0.8$. So the unclipped term is larger (less negative) than the clipped term. The$\min$selects the smaller (more negative) value, which is the clipped term,$(1-\epsilon) \hat{A}_t$. This is constant with respect to further decreases in$r_t(\theta)$. The gradient is zero.
The result: the optimizer has incentive to make bad actions up to $1-\epsilon = 0.8$ times less likely than the old policy, but no incentive to drive them to zero. The policy is prevented from completely eliminating actions that looked bad under stale data.
The pessimistic lower bound property. In both cases, the $\min$ operator ensures that $L^{\text{CLIP}}$ is always less than or equal to $L^{\text{CPI}}$. This makes $L^{\text{CLIP}}$ a lower bound (a pessimistic estimate) on the unclipped objective — the optimizer can never be tricked into thinking the policy improved more than it actually did. This is the functional equivalent of TRPO's KL constraint: both prevent the optimizer from taking steps that look good under the surrograte objective but actually harm the true performance. The difference is that PPO achieves this through a simple algebraic operation (clip + min) rather than through constrained optimization over a KL divergence.
First-order equivalence at $r=1$. When $\theta = \theta_{\text{old}}$, every $r_t(\theta) = 1$, which lies inside the clipping region $[1-\epsilon, 1+\epsilon]$. At this point, $L^{\text{CLIP}}(\theta) = L^{\text{CPI}}(\theta)$. Moreover, the gradient $\nabla_\theta L^{\text{CLIP}}$ at $\theta_{\text{old}}$ equals $\nabla_\theta L^{\text{CPI}}$ because the clipping has no effect — the clip operator is not active, and the $\min$ selects the unclipped term throughout. The first policy update is identical to the standard policy gradient update. The clipping only begins to affect the gradient on subsequent gradient steps (when doing multiple epochs on the same data), once $r_t(\theta)$ has moved away from 1.
Figure 1 visualization. The paper provides plots of a single term of $L^{\text{CLIP}}$ as a function of $r$ for positive advantages (left plot) and negative advantages (right plot). For positive advantages, the objective increases linearly with $r$ until $r = 1+\epsilon$, then flattens — there is no benefit to increasing $r$ further. For negative advantages, the objective decreases linearly with $r$ until $r = 1-\epsilon$, then flattens — there is no benefit to decreasing $r$ further. The red circle at $r=1$ marks the starting point for optimization. The sharp discontinuity in gradient at the clipping boundary is the mechanism that prevents large policy changes: the optimizer sees zero gradient for further movement once $r$ exits the clipping region.
Figure 2 validation. The paper empirically validates the lower bound property by plotting $L^{\text{CPI}}$, $L^{\text{CLIP}}$, and the expected KL divergence as a function of a linear interpolation between $\theta_{\text{old}}$ and the PPO-updated parameters on the Hopper-v1 environment (first policy update). The plot shows that:
$L^{\text{CPI}}$(the unclipped objective) continues increasing past the point of the actual PPO update.$L^{\text{CLIP}}$(the clipped objective) reaches its maximum at the PPO update point (where KL ≈ 0.02) and decreases thereafter.- This confirms that
$L^{\text{CLIP}}$penalizes policy changes that are too large: if the optimizer tried to take an even larger step, the clipped objective would tell it "no, that's worse," even though the unclipped objective would say "yes, go further!"
Why this form works — the key insight. The clipping mechanism elegantly solves three problems simultaneously:
-
It eliminates the incentive for excessive policy changes. Once
$r_t(\theta)$leaves the interval$[1-\epsilon, 1+\epsilon]$in the direction that would increase the objective, the gradient becomes zero, and the optimizer stops moving in that direction. -
It preserves the incentive for corrective updates. If a policy change would decrease the objective (a good action becoming less likely; a bad action becoming more likely), the clipping does not interfere — the
$\min$selects the unclipped term in those cases, preserving the full gradient signal that tells the optimizer to reverse course. -
It requires no KL divergence computation. The clipping operates directly on the probability ratio
$r_t(\theta)$, which is computed from the policy's log-probability outputs. No separate KL divergence measurement between old and new policies is needed—no second-order information, no Fisher matrix, no conjugate gradient.
Choice of $\epsilon$. The paper sweeps $\epsilon \in \{0.1, 0.2, 0.3\}$ on the MuJoCo benchmark (Table 1). The results are:
$\epsilon = 0.1$: average normalized score 0.76$\epsilon = 0.2$: average normalized score 0.82$\epsilon = 0.3$: average normalized score 0.70
The sweet spot is $\epsilon = 0.2$. An $\epsilon$ that is too small (0.1) overly constrains the policy update, preventing the policy from making sufficient progress per iteration. An $\epsilon$ that is too large (0.3) allows policy changes that are too large, approaching the destructive regime of the unclipped objective. Interestingly, $\epsilon = 0.2$ means the policy is allowed to change the probability of any action by at most a factor of 1.2× (increase) or 0.8× (decrease) per update — a relatively modest adjustment that accumulates over many iterations.
For Atari experiments (Table 5), the paper uses $\epsilon = 0.1 \times \alpha$ where $\alpha$ is linearly annealed from 1 to 0 over the course of training. This means $\epsilon$ starts at 0.1 and decays to 0, making the updates progressively more conservative as the policy converges — a reasonable schedule that reflects the intuition that larger updates are appropriate early in training when the policy is far from optimal, while smaller, more careful updates are needed near convergence.
The Adaptive KL Penalty Coefficient Alternative
The paper also develops an alternative approach that penalizes KL divergence directly, with an adaptive schedule for the penalty coefficient. This variant serves as an important baseline that demonstrates why clipping outperforms direct KL penalization.
The KL-penalized objective is:
where $\beta$ is a penalty coefficient (a non-negative scalar), and $\text{KL}[\pi_{\theta_{\text{old}}}(\cdot \mid s_t), \pi_\theta(\cdot \mid s_t)]$ is the Kullback-Leibler divergence between the old and new policy distributions at state $s_t$.
What it computes: For each timestep $t$, compute the CPI surrogate term $r_t(\theta) \hat{A}_t$ (how much better the action $a_t$ was than expected, reweighted by the policy change ratio), then subtract $\beta$ times the KL divergence between the old and new policies at that state. If the new policy deviates significantly from the old policy at state $s_t$, the KL term penalizes the objective proportionally to $\beta$. The penalized objective is maximized over $\theta$ using minibatch SGD.
Why this form: This objective directly implements the theoretically-motivated penalty form from Kakade and Langford (2002) — the same theory that underlies TRPO. The KL divergence term serves as a regularizer: the optimizer must balance the benefit of increasing $r_t(\theta) \hat{A}_t$ (improving the policy) against the cost of $\beta \cdot \text{KL}$ (moving too far from the data-collection policy). A larger $\beta$ enforces stronger regularization and smaller policy updates.
The problem with fixed $\beta$. The paper states that TRPO uses a hard constraint rather than a penalty because:
"it is hard to choose a single value of β that performs well across different problems—or even within a single problem, where the characteristics change over the course of learning."
Table 1 confirms this: Fixed KL with $\beta = 0.3$ scores 0.62, $\beta = 1.0$ scores 0.71, $\beta = 3.0$ scores 0.72, $\beta = 10.0$ scores 0.69. While some $\beta$ values achieve reasonable performance, none match the clipped surrogate (0.82). The optimal $\beta$ varies across the 7 environments in the benchmark, and likely varies within a single environment as learning progresses — early in training, when the policy is improving rapidly, a larger $\beta$ might be needed; near convergence, a smaller $\beta$ might be appropriate.
The adaptive $\beta$ solution. To address the fixed-$\beta$ problem, PPO introduces a heuristic adaptive schedule:
After each policy update (which optimizes $L^{\text{KLPEN}}$ using several epochs of minibatch SGD):
-
Compute the actual KL divergence:
$d = \hat{\mathbb{E}}_t[\text{KL}[\pi_{\theta_{\text{old}}}(\cdot \mid s_t), \pi_\theta(\cdot \mid s_t)]]$— the average KL divergence between the old and new policies over the states in the collected data. -
Adjust
$\beta$based on whether$d$exceeds or falls short of the target$d_{\text{targ}}$:- If
$d < d_{\text{targ}} / 1.5$: The policy changed too little — reduce$\beta \leftarrow \beta / 2$to allow larger changes next time. - If
$d > d_{\text{targ}} \times 1.5$: The policy changed too much — increase$\beta \leftarrow \beta \times 2$to enforce stronger regularization next time. - Otherwise:
$\beta$stays unchanged.
- If
The target KL divergence $d_{\text{targ}}$ is a hyperparameter. Values swept in the experiments: $d_{\text{targ}} \in \{0.003, 0.01, 0.03\}$ (Table 1).
What this algorithm does operationally: It implements a feedback controller for the KL divergence. If the policy updates are too timid (small KL), $\beta$ is halved, loosening the constraint. If updates are too aggressive (large KL), $\beta$ is doubled, tightening the constraint. The 1.5× threshold provides a hysteresis band so $\beta$ doesn't oscillate in response to minor KL fluctuations. The paper notes that "the parameters 1.5 and 2 above are chosen heuristically, but the algorithm is not very sensitive to them."
Performance of adaptive KL. Table 1 shows the adaptive KL variants:
$d_{\text{targ}} = 0.003$: score 0.68$d_{\text{targ}} = 0.01$: score 0.74$d_{\text{targ}} = 0.03$: score 0.71
The best adaptive KL variant (0.74) outperforms the best fixed KL variant (0.72 with $\beta=3$), confirming that adaptation helps. However, it still underperforms clipping (0.82). The paper explicitly acknowledges this:
"In our experiments, we found that the KL penalty performed worse than the clipped surrogate objective, however, we've included it here because it's an important baseline."
Why adaptive KL underperforms clipping. The paper doesn't provide a detailed analysis, but several factors likely contribute:
- The KL divergence is a symmetric measure of distributional distance (it doesn't distinguish between increases and decreases in action probabilities), while the clipping mechanism asymmetrically prevents only changes that would increase the objective — it allows changes that decrease the objective without restriction. A good action becoming less likely is penalized by the objective itself (through the advantage term), not by the clipping, so the policy can naturally recover from mistakes. The KL penalty, by contrast, penalizes all distributional shifts equally, potentially slowing down corrective updates.
- The
$\beta$adaptation is a coarse heuristic (halve/double) that may not track the optimal penalty coefficient closely enough during rapid policy improvement phases. - The target KL
$d_{\text{targ}}$must be specified as a hyperparameter and the optimal value varies across environments (0.01 performs best in the sweep). - The KL divergence must be computed at each policy update, adding computational overhead compared to the clipping approach which only requires evaluating policy probabilities.
The important role of the adaptive KL variant in the paper's narrative. Even though it performs worse, the adaptive KL variant is included because it demonstrates that the problem isn't solved by simply making the penalty coefficient adaptive — the clipping mechanism is fundamentally a better solution to the problem of preventing destructive updates. This strengthens the paper's central claim that the $\min(\text{unclipped}, \text{clipped})$ form is the key innovation, not just the idea of using a penalty or constraint.
The paper also notes that Heess et al. (2017) used the adaptive KL variant "to learn locomotion policies for 3D robots" in concurrent work, suggesting that the adaptive KL approach, while not optimal, is still practically useful in some settings.
The Complete PPO Objective: Combining Policy Loss, Value Loss, and Entropy Bonus
In practice, PPO is used as part of an actor-critic architecture where a value function $V_\theta(s)$ is learned alongside the policy. When parameters are shared between the policy and value function (a common design for efficiency, especially with CNNs processing pixel observations), a combined loss function is required. The full PPO objective is:
where $c_1$ and $c_2$ are scalar coefficients, $L^{\text{VF}}_t(\theta) = (V_\theta(s_t) - V^{\text{targ}}_t)^2$ is a squared-error value function loss, $V^{\text{targ}}_t$ is a target value (typically computed from the returns, often using the same GAE machinery), and $S[\pi_\theta](s_t)$ is an entropy bonus (the entropy of the policy's action distribution at state $s_t$).
What each term does and why it is included:
$L^{\text{CLIP}}_t(\theta)$ — the clipped policy surrogate. This is the main policy improvement signal, as described in detail above. It encourages the policy to increase probabilities of actions with positive advantage and decrease probabilities of actions with negative advantage, clipped to prevent excessive changes.
$-c_1 L^{\text{VF}}_t(\theta)$ — the value function loss (with a negative sign). The value function $V_\theta(s_t)$ predicts the expected return (cumulative discounted reward) from state $s_t$. The squared error $(V_\theta(s_t) - V^{\text{targ}}_t)^2$ penalizes inaccurate value predictions, and the negative sign in the combined loss means minimizing this term — we want the value function to be accurate. A good value function is critical because:
- It provides the baseline for advantage estimation (reducing variance in the policy gradient).
- It is used in the GAE computation to estimate advantage values.
- In many implementations, the value function is the basis for the target
$V^{\text{targ}}_t$.
For the MuJoCo experiments (Sections 6.1–6.2), the paper uses separate policy and value networks (no parameter sharing), so "coefficient $c_1$ is irrelevant" — the value function is trained independently. For Atari (Section 6.4), parameter sharing is used with $c_1 = 1$ (Table 5).
$+c_2 S[\pi_\theta](s_t)$ — the entropy bonus. The entropy $S[\pi_\theta](s_t) = -\sum_a \pi_\theta(a \mid s_t) \log \pi_\theta(a \mid s_t)$ measures how spread out or uncertain the policy's action distribution is. A deterministic policy (one action with probability near 1) has entropy near 0; a uniform random policy has maximum entropy. Adding entropy as a bonus to the objective encourages the policy to maintain some randomness, which promotes exploration — the policy continues trying different actions rather than prematurely converging to a deterministic strategy that might be suboptimal. The coefficient $c_2$ controls the strength of this exploration incentive.
The MuJoCo experiments (Sections 6.1–6.2) do not use an entropy bonus — the authors state "we don't use an entropy bonus." The Atari experiments (Section 6.4) use $c_2 = 0.01$ (Table 5), a relatively small value that provides a gentle exploration pressure without overwhelming the policy improvement signal.
Why this combined objective form: When using a shared network architecture (particularly for Atari, where a CNN processes raw pixels), the policy and value function share the same convolutional feature extractor. The combined loss ensures that the gradients from both the policy improvement objective and the value prediction objective flow back through the shared layers, learning features that are useful for both tasks simultaneously. The entropy bonus acts as a regularizer that prevents premature collapse of the policy's stochasticity.
A subtle point: the paper states that this objective is "(approximately) maximized each iteration." The "approximately" qualifier acknowledges that $L^{\text{CLIP}}$ is already an approximation (a surrogate for the true expected return), and the combined loss with value function error and entropy bonus further departs from a pure policy optimization objective. In practice, this approximate objective works well, and the SGD optimizer makes progress on all three terms simultaneously.
For implementations with separate networks, one simply optimizes $L^{\text{CLIP}}$ for the policy network using the clipped surrogate and optimizes $L^{\text{VF}}$ for the value network separately. The combined form is only necessary when parameters are shared.
Advantage Estimation: Truncated Generalized Advantage Estimation
PPO requires an estimator of the advantage function $\hat{A}_t$ for each timestep in the collected trajectory segments. The paper uses a truncated version of Generalized Advantage Estimation (GAE) [Sch+15a], adapted for the fixed-length trajectory segment architecture.
The GAE estimator is computed recursively from temporal difference (TD) errors. First, the TD error $\delta_t$ at each timestep $t$ is:
where $r_t$ is the reward received at timestep $t$, $\gamma$ is the discount factor (0.99 in all experiments), $V(s_{t+1})$ is the value function's prediction for the next state, and $V(s_t)$ is the value function's prediction for the current state.
What $\delta_t$ computes: The difference between the actual return $r_t + \gamma V(s_{t+1})$ (the reward plus the estimated future value) and the estimated value $V(s_t)$ of being in state $s_t$. A positive $\delta_t$ means the action taken was better than the value function expected; a negative $\delta_t$ means it was worse.
The GAE advantage estimator $\hat{A}_t$ is then:
where $T$ is the length of the trajectory segment (the horizon), $\lambda \in [0,1]$ is the GAE parameter (0.95 in all experiments), and $\gamma$ is the same discount factor used in $\delta_t$.
What $\hat{A}_t$ computes: An exponentially-weighted sum of future TD errors, where the weight decays by a factor of $\gamma\lambda$ at each step. This estimator interpolates between:
$\lambda = 0$:$\hat{A}_t = \delta_t$— a one-step TD error (low variance, high bias if the value function is inaccurate).$\lambda = 1$:$\hat{A}_t = -V(s_t) + r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \cdots + \gamma^{T-t+1} r_{T-1} + \gamma^{T-t} V(s_T)$— the Monte Carlo return minus the baseline (high variance, unbiased, equivalent to the advantage estimator used in Mnih et al. 2016).
Why this form: GAE allows trading off bias and variance by choosing $\lambda$. With $\lambda < 1$, the weights on distant TD errors decay exponentially, reducing the variance from long-horizon reward noise while accepting some bias from the value function approximation. The paper uses $\lambda = 0.95$ throughout, which is close to the Monte Carlo limit but with a small amount of variance reduction.
Why the estimator is "truncated": The data collection architecture described in Algorithm 1 collects trajectory segments of fixed length $T$ (e.g., 2048 timesteps for MuJoCo, 128 for Atari). The advantage estimator $\hat{A}_t$ for timesteps near the end of the segment does not have access to rewards beyond timestep $T$ — the sum is truncated at the final TD error $\delta_{T-1}$. The final state value $V(s_T)$ is used in the last $\delta_{T-1}$ term to provide an estimate of future returns, but this is an approximation. The paper notes that this truncated GAE "reduces to Equation (10) when $\lambda = 1$," where Equation (10) is the finite-horizon advantage estimator from Mnih et al. (2016):
This truncated estimator is necessary because the actors collect only $T$ timesteps of data before sending it to the optimizer, so rewards beyond the segment boundary are unavailable. The value function bootstrap $\gamma^{T-t} V(s_T)$ provides an approximate correction for the missing future rewards, assuming the value function is reasonably accurate.
The paper uses the same $\gamma = 0.99$ and $\lambda = 0.95$ across all experiment domains (Tables 3, 4, 5), suggesting these are robust default values that work well across continuous control and Atari tasks.
Data Collection and Optimization Loop (Algorithm 1)
The full PPO algorithm with actor-critic architecture is specified in Algorithm 1. The loop alternates between collecting data and optimizing the surrogate objective:
Step 1: Data collection (parallel actors). $N$ parallel actors each run the current policy $\pi_{\theta_{\text{old}}}$ in independent copies of the environment for $T$ timesteps. At each timestep, the actor records the state $s_t$, the action $a_t$ sampled from $\pi_{\theta_{\text{old}}}(\cdot \mid s_t)$, the reward $r_t$, and the value function prediction $V(s_t)$. The next state $s_{t+1}$ is also recorded. After $T$ timesteps, each actor has collected a trajectory segment of length $T$.
Step 2: Advantage computation. For each trajectory segment, the GAE advantage estimator computes $\hat{A}_1, \ldots, \hat{A}_T$ using the recorded rewards, value predictions, the discount factor $\gamma$, and the GAE parameter $\lambda$ (Equations 11–12). These advantages are computed once per data-collection iteration and then held fixed during the subsequent optimization phase.
Step 3: Surrogate objective construction. The full batch of $NT$ timesteps is assembled, containing for each timestep $t$: the state $s_t$, the action $a_t$, the old policy's log-probability $\log \pi_{\theta_{\text{old}}}(a_t \mid s_t)$, the advantage estimate $\hat{A}_t$, and (if using a value function) the value target $V^{\text{targ}}_t$. The surrogate objective $L^{\text{CLIP}}$ (or $L^{\text{CLIP+VF+S}}$ for shared architectures) is constructed using this data.
Step 4: Multi-epoch optimization. The surrogate objective is optimized using minibatch stochastic gradient ascent (specifically, Adam optimizer) for $K$ epochs. In each epoch, the $NT$ timesteps of data are shuffled and divided into minibatches of size $M \leq NT$. For each minibatch:
- The current policy
$\pi_\theta$is evaluated on the minibatch states$s_t$to compute the new probabilities$\pi_\theta(a_t \mid s_t)$. - The probability ratio
$r_t(\theta)$is computed by dividing by the stored old-policy probabilities. - The clipped surrogate
$L^{\text{CLIP}}$(or combined loss) is computed and its gradient with respect to$\theta$is obtained via automatic differentiation. - The Adam optimizer updates
$\theta$using this gradient.
Critically, the advantage estimates $\hat{A}_t$ and old-policy probabilities $\pi_{\theta_{\text{old}}}(a_t \mid s_t)$ are frozen during the entire $K$-epoch optimization — they are treated as constants, not recomputed for the current policy. This is what creates the distribution shift problem that the clipping mechanism addresses: the optimization is reusing stale advantage signals for multiple gradient steps.
Step 5: Parameter update. After the $K$ epochs are complete, the policy parameters $\theta$ become the new $\theta_{\text{old}}$ for the next data-collection iteration. The loop returns to Step 1.
Key hyperparameters and their configured values:
The paper provides three separate hyperparameter tables for different experiment domains:
| Domain | $T$ (horizon) | $K$ (epochs) | $M$ (minibatch) | $N$ (actors) | Stepsize | $\epsilon$ |
|---|---|---|---|---|---|---|
| MuJoCo 1M (Table 3) | 2048 | 10 | 64 | not specified | $3\times10^{-4}$ Adam | 0.2 |
| Roboschool (Table 4) | 512 | 15 | 4096 | 32 or 128 | adaptive | 0.2 (implied) |
| Atari (Table 5) | 128 | 3 | 32×8=256 | 8 | $2.5\times10^{-4}\times\alpha$ | $0.1\times\alpha$ |
Why these particular numbers:
Horizon $T$: The MuJoCo continuous control tasks use $T=2048$, which is much longer than the Atari $T=128$. This reflects the different timescales of the environments: MuJoCo physics simulations have smooth, continuous dynamics where rewards accrue over hundreds or thousands of timesteps (a walking robot needs many steps to travel a meaningful distance), while Atari games have discrete frame-based dynamics where meaningful game events happen within shorter windows. The Roboschool humanoid tasks use $T=512$ for locomotion (forward running) and, interestingly, the same $T=512$ for the flagrun tasks — the paper doesn't explain this choice, but it may reflect that humanoid locomotion involves more complex dynamics requiring longer trajectory segments than Atari but not as long as the simpler MuJoCo robots.
Epochs $K$: The MuJoCo 1M benchmark uses $K=10$ epochs — the optimizer makes 10 full passes over the collected $2048$ timesteps per iteration. This is the setting where the clipping mechanism is most critical: with 10 gradient steps on the same data, an unconstrained policy gradient would almost certainly suffer from destructive updates. The Roboschool tasks use even more aggressive reuse: $K=15$ epochs on $512 \times N$ timesteps (with $N=32$ for locomotion, that's $16,384$ timesteps per iteration). The Atari experiments use only $K=3$ epochs, suggesting that less reuse is needed for discrete-action domains — possibly because the CNN policy architecture is more sensitive to stale advantages, or because the shorter horizon $T=128$ means the advantages are noisier and shouldn't be optimized as aggressively.
Minibatch size $M$: The MuJoCo experiments use a small minibatch of $M=64$ drawn from $2048$ total timesteps, meaning many gradient updates per epoch (2048/64 = 32 minibatches per epoch × 10 epochs = 320 policy updates per data-collection iteration). The Atari experiments use $M=256$ (32×8, where 8 is the number of actors) — the notation "32×8" in the paper indicates a minibatch of 32 per actor, concatenated across 8 actors, for a total of 256 per minibatch. With $NT = 8 \times 128 = 1024$ total timesteps per iteration, this means 1024/256 = 4 minibatches per epoch × 3 epochs = 12 policy updates per iteration — far fewer than the MuJoCo setting, reflecting the lower reuse.
Stepsize: The MuJoCo experiments use a fixed Adam stepsize of $3\times10^{-4}$, a standard default. The Roboschool experiments state "Adam stepsize was adjusted based on the target value of the KL divergence" — this is the adaptive KL variant from Section 4, where $\beta$ is adjusted but $\beta$ is actually the KL penalty coefficient, not the stepsize. The paper's phrasing is slightly ambiguous, but it means that while using the adaptive KL variant of PPO, the KL target $d_{\text{targ}}$ controls the effective policy update size rather than a fixed learning rate schedule. The Atari experiments use a linearly annealed stepsize $2.5\times10^{-4} \times \alpha$ where $\alpha$ decays from 1 to 0 over the course of training, matching the annealing of $\epsilon$ — progressively smaller updates as the policy converges.
Why the actor-critic style with fixed-length trajectory segments: This architecture, popularized by Mnih et al. (2016) (A3C), is well-suited for use with recurrent neural networks (though the paper's experiments use feedforward networks) because it allows truncating backpropagation through time at the segment boundaries. More generally, the fixed-length segment approach provides a natural synchronization point: all actors collect $T$ timesteps, then the optimizer processes the batch, then all actors update to the new policy simultaneously. This is simpler to implement with distributed computing frameworks than fully asynchronous architectures where actors update their policies at different times.
Network Architecture Choices
Continuous control (MuJoCo and Roboschool). The policy is represented by a fully-connected multilayer perceptron (MLP) with two hidden layers of 64 units and tanh nonlinearities. The network outputs the mean of a Gaussian distribution, with variable standard deviations — the standard deviation parameters are separate learnable parameters (not output by the network), following the architecture used in prior work [Sch+15b, Dua+16]. The action is sampled from $\mathcal{N}(\text{mean}(s_t), \text{stddev})$.
Why this architecture: Two hidden layers of 64 units is a standard, relatively small architecture for continuous control benchmarks. It is sufficient to represent good policies for the MuJoCo tasks while being fast to train. The variable standard deviation allows the policy to learn how much exploration noise to inject at different states or at different stages of training — typically, the standard deviation decreases as the policy becomes more confident. The tanh activation function provides smooth gradients and bounded outputs, which is appropriate for the mean of a Gaussian policy.
For the MuJoCo 1M benchmark (Sections 6.1–6.2), the policy and value function do not share parameters — there are two separate MLPs with the same architecture (two hidden layers, 64 units, tanh). This is a simpler baseline that avoids the need to balance the policy and value function loss terms (coefficient $c_1$ is irrelevant). For the Roboschool experiments (Section 6.3), the paper doesn't explicitly state whether parameters are shared, but the use of the adaptive KL variant (Section 4) and the large-scale humanoid tasks suggests a similar separate-network setup.
Atari. The policy network uses the same architecture as Mnih et al. (2016) — a convolutional neural network (CNN) that processes raw pixel observations (84×84×4 frame stacks), followed by fully-connected layers outputting the policy logits (for a discrete action space) and the value function prediction. The paper explicitly states that for Atari, parameters are shared between the policy and value function, requiring the combined loss function $L^{\text{CLIP+VF+S}}$ with $c_1 = 1$ and $c_2 = 0.01$.
Why this architecture: Sharing parameters between the policy and value function is computationally efficient — the expensive convolutional feature extraction is done once rather than twice — and is a standard choice for Atari benchmarks. The shared architecture means the combined loss must be carefully balanced so that the value function loss doesn't dominate the policy improvement signal or vice versa. The choice $c_1 = 1$ gives equal weight to the value function error, which is a natural default. The entropy coefficient $c_2 = 0.01$ is small enough to not overwhelm the policy gradient but non-zero to maintain some exploration pressure in the discrete action space.
Design Choices and Their Justifications
Why clipping over KL penalty? The paper provides clear empirical evidence (Table 1) that clipping with $\epsilon = 0.2$ (score 0.82) outperforms both fixed KL penalties (best score 0.72) and adaptive KL penalties (best score 0.74). Beyond the numbers, clipping offers: (a) no KL divergence computation required — simpler code, faster execution; (b) no penalty coefficient to adapt or tune — only one hyperparameter $\epsilon$; (c) asymmetric treatment of beneficial vs. harmful policy changes — the clip only restricts changes that would increase the objective, allowing the policy to freely correct mistakes.
Why $\epsilon = 0.2$? The sweep in Table 1 shows a clear optimum at 0.2. The interpretation: 0.2 corresponds to allowing the probability of any action to change by at most 20% (factor of 1.2× up or 0.8× down) per iteration. This is large enough to allow meaningful policy improvement in a single iteration but small enough to prevent the destructive regime. The 0.2 value works across all 7 MuJoCo environments without per-environment tuning, which is a key selling point for robustness. For Atari, the annealed $\epsilon = 0.1 \times \alpha$ starting at 0.1 and decaying to 0 provides a more conservative schedule appropriate for the higher-dimensional discrete action space.
Why multiple epochs $K$ instead of a larger batch? The paper's design choice of $K=10$ epochs on MuJoCo reflects the goal of sample efficiency: collect a modest amount of data (2048 timesteps) and squeeze maximum learning from it. The alternative — collecting 20,480 timesteps and doing 1 epoch — would require the same total environment interactions but would update the policy less frequently, potentially slowing down learning when the policy is improving rapidly. The multiple-epoch approach allows the policy to be updated frequently while still reusing data extensively.
Why fixed-length segments instead of full episodes? The fixed-length segment architecture (Algorithm 1) decouples the data collection period from the natural episode boundaries in the environment. This is practical because: (a) it enables synchronous updates across parallel actors — all actors stop at $T$ timesteps simultaneously; (b) it works with environments where episodes have highly variable lengths (e.g., Atari games where a good policy might play for thousands of frames); (c) it provides a natural unit for the optimizer's batch size, which is important for efficient GPU utilization. The disadvantage — the truncated GAE estimator loses information beyond the segment boundary — is mitigated by using $\lambda$ and $\gamma$ close to 1 (0.95 and 0.99 respectively), which gives high weight to the value function bootstrap at the segment end.
Why no clipping in log space? The paper notes: "Note that we also tried clipping in log space, but found the performance to be no better." Clipping in log space would mean constraining $\log r_t(\theta)$ rather than $r_t(\theta)$ directly, which might seem more natural since policy optimization typically operates in log-probability space. The fact that log-space clipping performed no better suggests that the linear-space clipping is sufficient and has a simpler interpretation — the ratio $r_t(\theta)$ is a natural object for importance sampling, and clipping it directly corresponds to limiting the importance weight.
Why Adam over vanilla SGD? The paper uses Adam for all experiments (stated in Algorithm 1: "or usually for better performance, Adam"), which is standard practice in deep RL due to Adam's adaptive per-parameter learning rates and momentum, which help with the non-stationary optimization landscape of RL. The paper doesn't ablate this choice — it's inherited from prior work and assumed to be the sensible default.
Why no entropy bonus for MuJoCo? The continuous control experiments don't use an entropy bonus because the Gaussian policy with learned standard deviation already provides a natural exploration mechanism: the standard deviation parameters control how broadly the policy explores. In early training, the standard deviations are typically large, providing exploration; as the policy improves, the standard deviations naturally decrease as the policy becomes more confident. Adding an entropy bonus on top of this might interfere with the natural annealing of exploration. For Atari, the discrete action space doesn't have this built-in exploration annealing (the policy outputs a softmax distribution, and entropy can collapse to near-zero quickly), so the explicit entropy bonus with $c_2 = 0.01$ provides a gentle exploration pressure that prevents premature convergence to suboptimal deterministic strategies.
4. Key Insights and Innovations
Innovation 1: Clipping as a First-Order Constraint Mechanism — Replacing Second-Order Trust Regions with a Pessimistic Objective Shape
The field's dominant approach to stable policy updates before PPO was the trust region: explicitly constrain or penalize the distance between the old and new policy distributions, using the KL divergence as the distance metric. TRPO (Schulman et al., 2015) implemented this via a hard KL constraint solved with second-order optimization (conjugate gradient on the Fisher information matrix). The adaptive KL penalty variant of PPO implements it via a first-order penalty with a heuristically-adjusted coefficient. Both approaches share a common conceptual structure: measure divergence, then restrict it.
PPO's clipped surrogate objective breaks this conceptual structure entirely. It never computes a KL divergence. It never measures how far the new policy is from the old one. Instead, it modifies the shape of the objective function itself so that the optimizer receives zero marginal benefit from moving the probability ratio beyond a threshold — not because the optimizer is constrained from moving there, but because the objective function flatlines beyond that point.
This is a fundamentally different way to think about policy regularization. The question shifts from "how do we constrain the optimizer?" to "how do we design the objective so that the optimizer, acting greedily and unconstrained, naturally stops at a safe point?" The clipping mechanism answers this by making L_CLIP a pessimistic lower bound on L_CPI: the optimizer can freely ascend the gradient of L_CLIP, and the shape of that objective — flat beyond the clip threshold — ensures that the ascent naturally halts when the policy change becomes too large. There is no separate constraint satisfaction step. There is no penalty coefficient to tune. The optimizer simply climbs a hill whose summit is positioned at a safe distance from the starting point.
The empirical validation of this conceptual shift is Figure 2. The plot shows L_CPI continuing to rise as the policy moves further from θ_old — the unconstrained objective would cheerfully lead the optimizer off a cliff. L_CLIP peaks at the PPO update point (KL ≈ 0.02) and then declines, creating a natural stopping point. The optimizer, simply following the gradient of L_CLIP, arrives at the safe update without any explicit divergence computation. This is not an incremental improvement over KL-based methods — it is a fundamentally different mechanism that happens to achieve the same goal (bounded policy updates) through objective design rather than constraint enforcement.
The significance of this innovation extends beyond the numbers in Table 1 (clipping at 0.82 vs. adaptive KL at 0.74). It demonstrates that first-order methods can achieve the stability guarantees of second-order trust region methods, not by approximating the second-order computation more cleverly, but by adopting a different conceptual approach to regularization entirely. This opens the door to stable policy optimization in settings where second-order methods are impractical — architectures with dropout (where the Fisher matrix is ill-defined), parameter sharing between policy and value function (where the policy gradient is entangled with value function gradients), and recurrent networks (where the Fisher matrix becomes intractably large). The paper is explicit that TRPO is "not compatible with architectures that include noise (such as dropout) or parameter sharing," and PPO's clipping mechanism inherits none of these limitations because it never requires second-order information.
Innovation 2: The Asymmetric Treatment of Beneficial vs. Harmful Policy Changes — Letting the Policy Recover Without Penalty
A subtle but consequential design choice in the clipped objective is the asymmetry between how it handles policy changes that would increase vs. decrease the surrogate objective. This asymmetry is not an accident — it reflects a diagnostic insight about which policy changes actually cause training instability.
Consider a timestep where the advantage Â_t > 0 (a good action). The clipped objective removes the incentive to increase r_t(θ) beyond 1+ε. This is straightforward: the optimizer shouldn't be rewarded for making a good action dramatically more likely because the advantage estimate is stale and may not reflect the new policy's true performance. This is the standard trust region intuition.
But consider what happens when the optimizer makes a mistake — when a gradient step inadvertently decreases r_t(θ) for a good action (making it less likely despite the positive advantage). The clipping mechanism does nothing in this case. The min selects the unclipped term, which is more negative than the clipped term, and the full gradient signal flows through. The optimizer receives the unattenuated message: "this action is now less likely under your policy, which is bad — reverse course." Similarly, for a bad action (Â_t < 0), if the optimization somehow makes it more likely (increasing r_t(θ) above 1), the clipping does not interfere — the full negative gradient signal tells the optimizer to correct this mistake.
This asymmetry is fundamentally different from what a symmetric KL penalty does. KL divergence penalizes any distributional shift, regardless of direction — moving away from the old policy in either direction incurs a penalty. This means that if the optimizer takes a step that accidentally makes a good action less likely, the KL penalty adds an additional cost on top of the objective's natural penalty, potentially slowing down the correction. The KL penalty treats all policy changes as equally suspect. The clipped objective, by contrast, only restricts changes that the optimizer would want to make (those that increase the objective), while leaving it free to correct errors.
This design choice reflects a diagnostic insight: destructive policy updates are caused by the optimizer exploiting stale advantage estimates to make excessively confident changes, not by random drift. The danger is not that the policy might wander in some random direction away from θ_old — it's that the optimizer will eagerly follow a gradient signal that looks good under the old policy but is actually misleading. The clipping mechanism targets this specific failure mode: it prevents the optimizer from being rewarded for large policy changes, but does not punish it for correcting mistakes. A symmetric penalty is a blunter instrument that restricts both harmful exploitation and beneficial correction equally.
The empirical validation of this asymmetry is indirect but telling. Table 1 shows that no fixed KL penalty coefficient achieves the performance of clipping (best fixed KL: 0.72 with β=3.0; clipping: 0.82). Even the adaptive KL variant — which adjusts the penalty strength online — maxes out at 0.74. The paper doesn't ablate the asymmetry directly (it would require a symmetric clipping variant that clips both increases and decreases), but the underperformance of symmetric KL approaches is consistent with the hypothesis that asymmetric treatment matters.
Innovation 3: The Probability Ratio as a Universal Scaling Handle — Unifying Policy Gradient and Trust Region Perspectives Through a Single Scalar
Before PPO, the policy gradient literature and the trust region literature used different mathematical objects to reason about policy updates. Policy gradient methods (Equation 1) work with ∇_θ log π_θ(a_t | s_t), the score function — a vector per parameter that captures how changing each parameter affects the log-probability of the taken action. Trust region methods (Equations 3-4) work with π_θ(a_t | s_t) / π_θ_old(a_t | s_t), the probability ratio — a scalar per timestep that captures how much more or less likely the action is under the new policy. These two objects are mathematically related (the gradient of the ratio at θ=θ_old is the score function), but they lead to very different algorithmic frameworks: one is optimized with unconstrained SGD, the other with constrained second-order optimization.
PPO's key conceptual move is to place the probability ratio r_t(θ) at the center of the algorithm and to use it not just as an importance sampling correction (as in the CPI objective) but as the primary object that the regularization mechanism operates on. The clipping operator is applied directly to r_t(θ). The min operator compares the clipped and unclipped values of r_t(θ) Â_t. The entire regularization logic is expressed in terms of this single scalar per timestep, which has a clean interpretation (how has the action probability changed?) and is trivially computable from the policy's forward pass.
This is an innovation in conceptual framing more than in mathematics — the probability ratio was already known from importance sampling and from TRPO — but the decision to make it the sole handle for controlling policy update size unifies two previously separate concerns:
- Policy improvement (make good actions more likely, bad actions less likely) is captured by
r_t(θ) Â_t— the ratio reweights the advantage to account for the new policy's action probabilities. - Update regularization (don't change the policy too much) is captured by clipping
r_t(θ)to[1-ε, 1+ε]— the same ratio directly measures how much the policy has changed at each timestep.
In TRPO, these two concerns are handled by separate pieces of machinery: the surrogate objective (Equation 3) handles improvement, and the KL constraint (Equation 4) handles regularization. The KL constraint requires a separate computation (the Fisher matrix) that is only indirectly related to the policy improvement objective. PPO collapses both concerns into operations on a single scalar — the probability ratio — enabling the entire algorithm to be implemented as a single loss function that can be differentiated with one call to an automatic differentiation library.
The practical consequence of this unification is the paper's oft-cited claim that PPO requires "only few lines of code change to a vanilla policy gradient implementation." This is not an exaggeration: in a codebase that already implements L_PG(θ) = Ê_t[log π_θ(a_t | s_t) Â_t], switching to PPO involves (a) storing log π_θ_old(a_t | s_t) during data collection, (b) constructing r_t(θ) = exp(log π_θ(a_t | s_t) - log π_θ_old(a_t | s_t)), and (c) replacing the loss with min(r_t Â_t, clip(r_t, 1-ε, 1+ε) Â_t). Three lines of code. TRPO, by contrast, requires several hundred lines for Fisher-vector products, conjugate gradient, and line search. This is not merely a convenience — it fundamentally changes who can implement, debug, and extend the algorithm.
Innovation 4: The Compute-Optimal Reuse of On-Policy Data — Demonstrating That Multiple Epochs Are Possible Without Off-Policy Corrections
A less obvious but practically crucial innovation is PPO's demonstration that on-policy data can be reused for multiple gradient epochs without off-policy corrections, as long as the policy updates are kept small enough. This was not the prevailing wisdom at the time. The established approaches for data-efficient policy optimization fell into two camps:
- On-policy methods (vanilla policy gradient, A2C) use each batch of data once, then discard it. This is safe but sample-inefficient.
- Off-policy methods (DQN, DDPG, ACER) store data in a replay buffer and reuse it many times, but require importance sampling corrections or other off-policy machinery (truncated importance weights, Retrace operators, target networks) to handle the distribution shift between the data-collection policy and the current policy.
PPO stakes out a third position: on-policy data can be reused for multiple epochs without off-policy corrections, provided the policy doesn't change too much during those epochs. The clipping mechanism ensures that the policy stays within a safe radius of the data-collection policy — the 1±ε bound on r_t(θ) means that at every timestep, the new policy's action probability is within a factor of 1±ε of the old policy's. Within this narrow radius, the advantage estimates Â_t remain approximately valid for the new policy (since they were computed assuming the old policy, and the new policy is very close to it), and no explicit importance sampling correction is needed. The r_t(θ) term in the surrogate objective serves as a local correction — it accounts for the small change in action probabilities — but it is not a full off-policy importance weight (which would require multiplying by the cumulative product of ratios over multiple timesteps and would suffer from enormous variance).
The empirical demonstration that this works is Table 1's "No clipping or penalty" baseline (score -0.39) versus clipping (score 0.82) — both use multiple epochs on the same data, but only the clipped variant prevents the policy from moving so far that the advantages become invalid. The fact that the clipped variant achieves strong performance with K=10 epochs on MuJoCo and K=15 on Roboschool — reusing each batch of data 10-15 times — without any off-policy corrections is a significant empirical finding. It means that PPO can achieve sample efficiency approaching off-policy methods (which reuse data hundreds or thousands of times) while retaining the algorithmic simplicity and stability of on-policy methods.
This is not a theoretical innovation — the paper provides no new theory about why multiple epochs work within the clipping regime. It is an empirical discovery that 1±ε clipping on the probability ratio is sufficient to keep the policy close enough to the data-collection policy that multiple gradient steps remain productive. This discovery has had enormous practical impact: it means practitioners can get much of the sample efficiency benefit of experience replay without implementing any of the complex off-policy machinery. It also explains why PPO with clipping outperforms the adaptive KL penalty (Section 4): the KL penalty targets a global distributional distance (average KL divergence), while the probability ratio clipping ensures a per-timestep, pointwise bound on how much the new policy can differ from the old policy. A small average KL divergence can still allow individual action probabilities to change dramatically (if the distributions differ on a low-probability action, the KL is small but the ratio can be large), and those dramatic changes at specific timesteps are what cause the advantage estimates to become invalid. The clipping mechanism prevents this by bounding the ratio at every timestep individually.
This per-timestep bound is the reason PPO's clipped objective can be described as a "pessimistic lower bound" on L_CPI. For any timestep where the optimizer would have liked to increase r_t(θ) beyond 1+ε (for a positive advantage), the clipped objective replaces the overly optimistic L_CPI estimate (r_t(θ) Â_t, which assumes the advantage remains valid at large r_t) with the more conservative (1+ε) Â_t. The min operator ensures that the objective never credits the policy for improvements that are based on stale advantage estimates at large probability ratios. This is a form of implicit off-policy correction — not by computing importance weights to debias the advantage estimates, but by simply refusing to reward the policy for moving into regions where those estimates are unreliable.
Innovation 5: The Adaptive KL Penalty as a Diagnostic Baseline — Demonstrating That Adaptation Alone Cannot Match Objective-Design Methods
The paper's inclusion of the adaptive KL penalty variant (Section 4) and its thorough comparison against clipping in Table 1 is itself an intellectual contribution — a case study in what is sometimes called diagnostic ablation. The adaptive KL variant is not presented as a competing method the authors are trying to promote. The paper explicitly states: "we found that the KL penalty performed worse than the clipped surrogate objective, however, we've included it here because it's an important baseline."
Why is it an "important baseline"? Because it isolates a specific hypothesis about what makes policy optimization stable. The hypothesis — call it the adaptation hypothesis — is: the reason TRPO uses a hard constraint rather than a penalty is that "it is hard to choose a single value of β that performs well across different problems—or even within a single problem, where the characteristics change over the course of learning." If this hypothesis is correct, then making β adaptive — adjusting it online to maintain a target KL divergence — should solve the problem. The penalty form should work as well as the constraint form once β is allowed to vary.
The adaptive KL variant tests this hypothesis directly. The results in Table 1 reject it: the best adaptive KL variant (d_targ = 0.01) scores 0.74, significantly below clipping at 0.82. This means the reason fixed-KL penalties underperform is not (only) that β is fixed — it's that the penalty form itself is inferior to the clipping form. Something about penalizing KL divergence — even with an optimal, dynamically-adjusted β — is fundamentally less effective than the min(clipped, unclipped) objective shape.
This negative result is informative because it sharpens the paper's central claim. The paper is not arguing "KL-based regularization is fine, you just need to make β adaptive." It is arguing "KL-based regularization is fundamentally the wrong mechanism — you need to redesign the objective so that the optimizer naturally stops at a safe point, rather than trying to penalize it for going too far." The adaptive KL baseline provides the evidence needed to distinguish these two claims. Without it, a reader could reasonably think: "well, maybe you just need to tune β better." With it, the paper can say: "even the best possible β schedule — one that exactly tracks a target KL divergence — underperforms clipping."
The adaptive KL variant also serves as a bridge between TRPO and PPO. TRPO uses a hard KL constraint; the adaptive KL variant uses a soft KL penalty with an adaptive coefficient; PPO with clipping eliminates the KL computation entirely. This is a conceptual progression from "constrain the divergence" to "penalize the divergence adaptively" to "design the objective so divergence is self-limiting." The paper uses this progression to position PPO as the natural endpoint of a line of reasoning that started with TRPO — not a completely unrelated method, but the logical conclusion of trying to simplify trust region policy optimization while retaining its stability properties.
The concurrent use of the adaptive KL variant by Heess et al. (2017) on 3D locomotion tasks (noted in Section 6.3) is an interesting case of parallel development: another group independently arrived at a similar approach (KL penalty with adaptive β) but did not discover the clipping mechanism. The paper's claim that clipping outperforms the adaptive KL penalty suggests that the concurrent work, while successful, was using a suboptimal stabilization mechanism — and that PPO with clipping would likely achieve even better results on those tasks.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. The paper uses two broad families of environments: (1) 7 MuJoCo continuous control tasks simulated via OpenAI Gym—HalfCheetah-v1, Hopper-v1, InvertedDoublePendulum-v1, InvertedPendulum-v1, Reacher-v1, Swimmer-v1, and Walker2d-v1—trained for 1 million timesteps each; and (2) 49 Atari games in the Arcade Learning Environment, trained for 40 million game frames (10 million timesteps, since frames are skipped by a factor of 4). For the 3D humanoid showcase, the paper uses Roboschool environments: RoboschoolHumanoid-v0, RoboschoolHumanoidFlagrun-v0, and RoboschoolHumanoidFlagrunHarder-v0.
-
Base model(s). For MuJoCo tasks, the policy is a fully-connected MLP with two hidden layers of 64 units and
tanhnonlinearities, outputting the mean of a Gaussian distribution with variable standard deviations (separate learnable parameters, not network outputs). For Atari, the policy uses the same CNN architecture as Mnih et al. (2016) processing 84×84×4 frame stacks. The MuJoCo experiments use separate policy and value networks; the Atari experiments share parameters between policy and value function. The authors choose these architectures to match prior work (Schulman et al., 2015; Duan et al., 2016; Mnih et al., 2016) and enable direct comparison against published baselines. -
Metrics. For the MuJoCo 1M benchmark: average total reward over the last 100 episodes of training, then normalized per environment so that the random policy scores 0 and the best result across all algorithms scores 1, averaged across 3 random seeds × 7 environments = 21 runs to produce a single scalar. For the Atari benchmark: two metrics—(1) average episode reward over the entire training period (favors fast learning) and (2) average episode reward over the last 100 episodes of training (favors final performance), each averaged across 3 trials.
-
Baselines. For continuous control: TRPO (Schulman et al., 2015)—trust region policy optimization; CEM (Szita and Lőrincz, 2006)—cross-entropy method; vanilla policy gradient with adaptive stepsize—Adam stepsize adjusted based on KL divergence from old to updated policy; A2C (Mnih et al., 2016)—synchronous advantage actor-critic; and A2C with trust region (Wang et al., 2016). For the surrogate objective comparison (Table 1): no clipping or penalty (
L_t(θ) = r_t(θ) Â_t); clipping with ε ∈ {0.1, 0.2, 0.3}; adaptive KL penalty withd_targ∈ {0.003, 0.01, 0.03}; and fixed KL penalty with β ∈ {0.3, 1, 3, 10}. For Atari: A2C and ACER (Wang et al., 2016), both with hyperparameters tuned to maximize performance on the benchmark. -
Generation budget / compute accounting. For MuJoCo, training runs for 1 million timesteps with a horizon
T = 2048per data-collection iteration andK = 10epochs of optimization per iteration. For Roboschool, training runs for 50–100 million timesteps withT = 512andK = 15epochs. For Atari, training runs for 10 million timesteps (40 million game frames with frame skip 4), withT = 128,K = 3epochs, andN = 8parallel actors. Compute is measured in environment timesteps, not wall-clock or FLOPs—this is standard for RL benchmarking and ensures that comparisons between algorithms are made at equal amounts of environment interaction. The paper does not account for the computational cost of the optimization phase itself (forward/backward passes through the network), which means wall-clock time comparisons would depend on implementation details and hardware. -
Cross-validation / statistical protocol. For the surrogate objective comparison (Section 6.1), each algorithm variant is run on all 7 MuJoCo environments with 3 random seeds each, producing 21 total runs per variant. Scores are normalized per environment (random policy = 0, best result = 1) and averaged to produce a single scalar. The paper reports the average normalized score but does not report confidence intervals, standard deviations, or statistical significance tests. For Atari (Section 6.4), each algorithm is run with 3 random seeds per game, and the number of games "won" is determined by comparing the mean score across trials. The paper provides learning curves for all 49 Atari games in Appendix B (Figure 6) showing all three seeds, and a table of mean final scores (Table 6). The small number of seeds (3) per environment is a practical concession to computational cost but limits the statistical reliability of per-game comparisons.
Main Quantitative Results
Surrogate Objective Comparison on MuJoCo (Section 6.1)
The core empirical result appears in Table 1, which reports average normalized scores across 21 runs (7 environments × 3 seeds) for each surrogate objective variant. The headline numbers are:
- No clipping or penalty: −0.39 — worse than the random policy baseline of 0. The paper notes this is "because for one environment (half cheetah) it leads to a very negative score, which is worse than the initial random policy." This is the catastrophic failure mode of unconstrained multi-epoch optimization.
- Clipping, ε = 0.2: 0.82 — the best result across all variants. This is the paper's primary proposed method.
- Clipping, ε = 0.1: 0.76 — noticeably worse, suggesting ε = 0.1 is too restrictive and prevents sufficient policy improvement per iteration.
- Clipping, ε = 0.3: 0.70 — substantially worse than ε = 0.2, suggesting that allowing probability ratios to change by up to 30% begins to approach the destructive regime seen in the no-clipping case.
- Adaptive KL penalty: best at
d_targ = 0.01, score 0.74. The other targets:d_targ = 0.003scores 0.68;d_targ = 0.03scores 0.71. - Fixed KL penalty: best at β = 3, score 0.72. Other β values: β = 0.3 scores 0.62; β = 1.0 scores 0.71; β = 10.0 scores 0.69.
The clipping variant with ε = 0.2 outperforms the best fixed KL penalty by 0.10 (0.82 vs. 0.72) and the best adaptive KL penalty by 0.08 (0.82 vs. 0.74). The gap between clipping and the no-clipping baseline is 1.21 (0.82 vs. −0.39), underscoring that unconstrained multi-epoch optimization is not merely suboptimal but actively destructive.
A critical detail: the paper states that all results in Table 1 use the same base hyperparameters from Table 3 (T = 2048, stepsize = 3×10⁻⁴, K = 10, M = 64, γ = 0.99, λ = 0.95), with only the surrogate objective variant and its specific hyperparameters (ε, β, d_targ) being swept. This means the comparison is a controlled ablation of the surrogate objective design, holding the rest of the training pipeline constant. The clipped surrogate's advantage is therefore attributable specifically to the objective form, not to differences in network architecture, optimizer configuration, or data collection strategy.
Comparison Against Prior Algorithms on MuJoCo (Section 6.2)
Figure 3 presents learning curves (average episode reward vs. timestep) for PPO (with clipping, ε = 0.2) against five baselines on all 7 MuJoCo environments. The paper states the headline claim:
"We see that PPO outperforms the previous methods on almost all the continuous control environments."
Reading the curves in Figure 3 environment-by-environment:
- HalfCheetah-v1: PPO reaches approximately 2000 by 1M timesteps, well above TRPO (~1500) and A2C (~1000). CEM and vanilla PG with adaptive stepsize are far behind.
- Hopper-v1: PPO reaches approximately 2500, roughly matching TRPO and A2C+Trust Region by 1M timesteps, all substantially above vanilla PG (~1000).
- InvertedDoublePendulum-v1: PPO approaches 8000 by 400K timesteps and maintains that level, clearly ahead of TRPO (~6000) and A2C (~4000). CEM and vanilla PG perform poorly on this task.
- InvertedPendulum-v1: PPO reaches the maximum score of 1000 rapidly (by ~200K timesteps), matching TRPO and A2C+Trust Region, while A2C, CEM, and vanilla PG converge more slowly or to lower final values.
- Reacher-v1: The y-axis is negative (the reward is a negative cost), with higher (less negative) values being better. PPO reaches approximately −20, outperforming TRPO (~−30) and all other methods. CEM fails completely, getting stuck around −120 to −100.
- Swimmer-v1: PPO reaches approximately 120, well above TRPO (~80) and A2C (~60). Vanilla PG and CEM plateau around 40.
- Walker2d-v1: PPO reaches approximately 3000, clearly ahead of TRPO (~2500), A2C (~2000), and A2C+Trust Region (~2200). Vanilla PG and CEM perform poorly.
On 6 of the 7 environments (all except possibly Hopper, where performance is similar to the best baselines), PPO is clearly above or equal to the best competing method by the end of training. The "almost all" qualifier in the paper's claim accurately reflects the Hopper result. A notable pattern: PPO often learns faster (steeper initial slope) as well as achieving higher final performance, suggesting that the multiple-epoch optimization with clipping provides benefits to both sample efficiency and asymptotic performance.
The paper does not provide a summary table aggregating final scores across environments for this comparison—Figure 3 is the sole evidence. This is a departure from Section 6.1, where normalized aggregate scores are reported. The reason may be that the baseline algorithms (TRPO, A2C, etc.) were run with their own tuned hyperparameters, making a uniform aggregation less meaningful, but the absence of a summary metric makes quantitative comparison more subjective.
3D Humanoid Showcase (Section 6.3)
Figure 4 shows learning curves for PPO on three challenging 3D humanoid tasks from Roboschool. No comparisons against other algorithms are provided—this section is a "showcase," not a controlled comparison. The results demonstrate that PPO can scale to high-dimensional continuous control:
- RoboschoolHumanoid-v0 (forward locomotion): PPO reaches approximately 4000 reward by 50M timesteps, with a steadily increasing learning curve. The policy learns to run forward at speed.
- RoboschoolHumanoidFlagrun-v0 (target-chasing): PPO reaches approximately 2500 by 100M timesteps, with more variance in the learning curve (visible as a wider spread in the plot). The task is harder because the target position changes randomly every 200 timesteps or when reached, requiring the robot to both run and steer.
- RoboschoolHumanoidFlagrunHarder-v0 (target-chasing with obstacles): PPO reaches approximately 3000 by 100M timesteps. The robot is pelted by cubes and must get up off the ground. The learning curve shows a distinct phase transition around 20M timesteps where the reward rapidly increases from near 0 to ~2000, suggesting that the policy discovers a qualitatively new behavior (likely getting up after falling) at that point.
The paper notes that Heess et al. (2017) concurrently used the adaptive KL variant of PPO for similar 3D locomotion tasks, but the PPO results shown in Figure 4 presumably use the clipped surrogate (the paper doesn't explicitly state which variant is used for Roboschool, but since the clipped variant is established as superior in Section 6.1, it is the natural choice). The hyperparameters in Table 4 indicate that the Adam stepsize was "adjusted based on the target value of the KL divergence," which is the adaptive KL variant from Section 4—this creates some ambiguity about which PPO variant was actually used. The paper states in Section 6.3 that "Heess et al. [Hee+17] used the adaptive KL variant of PPO (Section 4) to learn locomotion policies for 3D robots," distinguishing their concurrent work from the present paper's results, which implies this paper used the clipped variant. But Table 4's mention of KL-divergence-based stepsize adjustment contradicts this. The ambiguity is not resolved in the text.
Atari Comparison (Section 6.4)
The Atari results are the most extensive empirical evaluation. Table 2 provides a head-to-head comparison counting the number of games "won" by each algorithm:
| Metric | A2C | ACER | PPO | Tie |
|---|---|---|---|---|
| (1) Avg. reward over all training | 1 | 18 | 30 | 0 |
| (2) Avg. reward over last 100 episodes | 1 | 28 | 19 | 1 |
PPO wins 30 out of 49 games on the "overall training" metric (which favors fast learning) and 19 out of 49 on the "last 100 episodes" metric (which favors final performance). ACER wins 18 and 28 respectively. A2C wins only 1 game on each metric. This is the basis for the paper's claim that PPO "performs significantly better (in terms of sample complexity) than A2C and similarly to ACER though it is much simpler."
The pattern is informative: PPO dominates on the speed-of-learning metric (30 wins), while ACER dominates on final performance (28 wins). This is consistent with the design differences between the two algorithms. PPO aggressively reuses on-policy data (K=3 epochs per batch) but does not maintain a replay buffer, so it learns quickly from recent experience but may forget or converge prematurely. ACER uses experience replay with off-policy corrections, which is more sample-efficient in the long run (it can learn from old experiences that PPO has discarded) but learns more slowly initially due to the variance of importance sampling corrections.
Figure 6 and Table 6 provide the raw data. Table 6 lists the mean final scores (last 100 episodes) for all 49 games. Some illustrative comparisons:
- Games where PPO substantially outperforms ACER: BattleZone (PPO: 17,366.7 vs. ACER: 8,983.3), Gravitar (PPO: 737.2 vs. ACER: 225.3), Kangaroo (PPO: 9,928.7 vs. ACER: 50.0), Zaxxon (PPO: 5,008.7 vs. ACER: 29.0), WizardOfWor (PPO: 4,185.3 vs. ACER: 2,308.3), MontezumaRevenge (PPO: 42.0 vs. ACER: 0.3). These are mostly exploration-heavy or long-horizon games.
- Games where ACER substantially outperforms PPO: BeamRider (ACER: 3,863.3 vs. PPO: 1,590.0), Breakout (ACER: 456.4 vs. PPO: 274.8), Centipede (ACER: 8,904.8 vs. PPO: 4,386.4), ChopperCommand (ACER: 5,287.7 vs. PPO: 3,516.3), DemonAttack (ACER: 38,808.3 vs. PPO: 11,378.4), Gopher (ACER: 37,802.3 vs. PPO: 2,932.9), VideoPinball (ACER: 156,225.6 vs. PPO: 37,389.0), UpNDown (ACER: 145,051.4 vs. PPO: 95,445.0). These tend to be games where long-term planning and precise action sequences matter, suggesting ACER's off-policy replay provides an advantage for mastering fine-grained control.
- Games where both perform similarly: Alien (PPO: 1,850.3 vs. ACER: 1,655.4), Assault (PPO: 4,971.9 vs. ACER: 4,653.8), Krull (PPO: 7,942.3 vs. ACER: 7,268.4), KungFuMaster (PPO: 23,310.3 vs. ACER: 27,599.3), Qbert (PPO: 14,293.3 vs. ACER: 15,316.6).
- Notable failure: PPO scores 0.0 on Venture, matching A2C and ACER (all three algorithms get 0.0). This is a known hard exploration game where none of these on-policy/near-on-policy methods succeeds.
The paper's claim that PPO is "significantly better (in terms of sample complexity) than A2C" is overwhelmingly supported by the data—A2C wins only 1 of 49 games on each metric. The claim that PPO performs "similarly to ACER" is accurate but requires nuance: PPO is better at fast learning (metric 1), ACER is better at final performance (metric 2), and on aggregate, they are comparable but with different strengths. The "much simpler" qualifier is a design claim, not an experimental result, but it is persuasive given ACER's complex machinery (Retrace operator, truncated importance weights, trust region in Q-updates, bias-variance tradeoff in off-policy corrections) versus PPO's three lines of code change to vanilla policy gradient.
Ablation Studies and Robustness Checks
Clipping epsilon sweep (Table 1): The choice of ε is the most important hyperparameter in PPO. The sweep across ε ∈ {0.1, 0.2, 0.3} on the MuJoCo 1M benchmark shows a clear optimum at ε = 0.2 (score 0.82), with ε = 0.1 scoring 0.76 and ε = 0.3 scoring 0.70. This is a convex relationship suggesting that ε = 0.2 is genuinely near-optimal for this benchmark. The paper also tests ε for Atari with a different scheme—annealed from 0.1 to 0 over training (ε = 0.1 × α, where α linearly decays from 1 to 0)—but does not ablate this choice against fixed ε values or against the MuJoCo-optimal ε = 0.2. This is a missing ablation: it is unclear whether annealing ε is beneficial for Atari or whether a fixed ε = 0.2 would work similarly.
Fixed KL penalty coefficient sweep (Table 1): The sweep across β ∈ {0.3, 1, 3, 10} shows relatively flat performance: β = 0.3 scores 0.62; β = 1.0 scores 0.71; β = 3.0 scores 0.72; β = 10.0 scores 0.69. The range from β = 1 to β = 10 produces scores within 0.03 of each other, suggesting the fixed KL penalty is somewhat robust to β once it's in a reasonable range but never reaches clipping's performance. The drop at β = 0.3 (score 0.62) indicates that too weak a penalty allows destructive updates, while the curve being flat from 1 to 10 suggests the penalty saturates—beyond a point, larger β just halts learning rather than further improving stability.
Adaptive KL target sweep (Table 1): d_targ ∈ {0.003, 0.01, 0.03} produces scores of 0.68, 0.74, 0.71 respectively, with d_targ = 0.01 being optimal. The range is smaller than for fixed β (0.74 − 0.68 = 0.06), suggesting the adaptive mechanism is somewhat robust to the target value. The optimal d_targ = 0.01 means the algorithm targets an average KL divergence of 0.01 between old and new policies per update—a relatively small value, consistent with the idea that keeping policy changes small is critical for stability with multiple epochs.
Clipping in log space (Section 6.1 text): The paper briefly mentions: "Note that we also tried clipping in log space, but found the performance to be no better." This is an important negative result because it suggests that the linear-space clipping on r_t(θ) is not special—log-space clipping (constraining log r_t(θ), which would symmetrically bound multiplicative increases and decreases) performs similarly. However, the paper provides no numbers, no table, and no analysis for this claim, making it impossible to assess its reliability. This is a missed opportunity: a comparison of linear vs. log-space clipping would help clarify the mechanism's sensitivity to the precise form of the clip operator.
Separate vs. shared policy-value networks (implicit ablation): The MuJoCo experiments use separate policy and value networks (stated explicitly: "We don't share parameters between the policy and value function"), while the Atari experiments use the combined loss L_CLIP+VF+S in Equation 9 with shared parameters. The paper demonstrates that PPO works well in both configurations, but does not directly compare separate vs. shared architectures on the same task. This is not a formal ablation but is implicitly validated by the strong performance in both settings.
Number of epochs K (across experiment tables): The choice of K varies dramatically across domains: K = 10 for MuJoCo 1M (Table 3), K = 15 for Roboschool (Table 4), K = 3 for Atari (Table 5). The paper does not ablate K systematically—it does not show what happens with K = 1 (effectively reducing PPO to vanilla policy gradient with one update per batch) or K = 20 (extreme reuse). The fact that different domains require different K values suggests that the optimal degree of data reuse is task-dependent, but the paper provides no guidance on how to choose K beyond the specific values used in the experiments. This is a limitation: a practitioner applying PPO to a new domain has no principled way to set K.
Horizon T (across experiment tables): Similarly, T varies from 128 (Atari) to 512 (Roboschool) to 2048 (MuJoCo). The paper provides no ablation showing how sensitive PPO is to T and whether the truncated GAE estimator remains reliable at different horizons. The choice of T interacts with K—a longer horizon provides more data per iteration, potentially allowing more epochs of reuse—but this relationship is not explored.
Entropy bonus coefficient (Atari only): The Atari experiments use an entropy bonus with c_2 = 0.01. The MuJoCo experiments use no entropy bonus. The paper does not ablate the entropy bonus coefficient for Atari (e.g., c_2 = 0, c_2 = 0.1) or test whether adding an entropy bonus to MuJoCo would help. This is a non-trivial ablation because the entropy bonus has a known interaction with policy gradient methods—too low, and the policy may prematurely converge; too high, and the policy remains too random to achieve good performance. The paper's choice of 0.01 for Atari is presented as a constant, not as a result of tuning.
Value function coefficient (Atari only): The Atari experiments use c_1 = 1 for the value function loss. No ablation of this coefficient is provided. Since the value function loss and policy objective are combined in a shared network, the relative weight of these two terms affects which features the shared convolutional layers learn. A sweep across c_1 values would clarify how sensitive the shared architecture is to this balance.
Learning rate annealing (Atari only): The Atari experiments use linear annealing of both the learning rate (2.5 × 10⁻⁴ × α) and the clipping parameter (0.1 × α), where α decays from 1 to 0 over training. The paper does not compare this annealing scheme against fixed values, making it unclear whether annealing is necessary for Atari performance or was adopted as a conservative measure. Given that the MuJoCo experiments succeed with fixed values (no annealing), the annealing in Atari may be addressing a domain-specific sensitivity (perhaps the CNN architecture or the discrete action space requires more careful convergence), but this is not established empirically.
Multiple random seeds (all experiments): All MuJoCo and Atari experiments use 3 random seeds. The paper provides learning curves showing individual seeds for Atari (Figure 6), which gives some visibility into variance. For the MuJoCo comparison (Figure 3), only mean curves are shown, obscuring seed-to-seed variability. For the Roboschool tasks (Figure 4), the learning curves appear to show a single run or a mean with no variance indicated. Three seeds is a minimal bar for reproducibility and limits the reliability of per-environment comparisons. The aggregate metrics (Table 1's 21-run average; Table 2's 3-trial average) partially mitigate this by combining across environments.
Gradable answers (Atari): The evaluation on Atari uses the raw episode reward, which is the natural metric. The paper does not report human-normalized scores (percentage of human performance), which is a common Atari evaluation convention that facilitates cross-paper comparisons (e.g., DQN, A3C). Using raw scores means the comparison is valid only against the specific baselines run by the authors under identical conditions, not against published numbers from other papers that may use different frame-skip, stochasticity, or preprocessing.
Critical Assessment: Do the Experiments Support the Paper's Central Claims?
The paper makes three central claims in the abstract and introduction. I examine each against the experimental evidence.
Claim: PPO has "some of the benefits of trust region policy optimization (TRPO), but [is] much simpler to implement, more general, and [has] better sample complexity (empirically)"
The simplicity claim is a design assertion, not an experimental result, and is reasonably supported by the algorithm description—PPO requires only a few lines of code change to vanilla policy gradient, while TRPO requires conjugate gradient, Fisher-vector products, and line search. No user study or implementation-complexity metric is provided, but this is not typically expected for an algorithms paper.
The generality claim refers to compatibility with architectures that TRPO cannot handle: dropout and parameter sharing. The Atari experiments validate the parameter-sharing compatibility—PPO with the shared policy-value network architecture (L_CLIP+VF+S) achieves competitive performance. However, the paper never tests PPO with dropout. The claim about dropout compatibility is therefore untested—it is a theoretical argument (the Fisher matrix computation in TRPO becomes ill-defined with dropout; PPO's first-order clipping has no such dependency) but not an empirical finding. This weakens the "more general" claim.
The better sample complexity claim requires careful parsing. On MuJoCo, PPO with clipping achieves a 0.82 average normalized score (Table 1). The paper does not provide a comparable aggregate score for TRPO on the same benchmark—TRPO appears only in the learning curves of Figure 3, where it is outperformed by PPO on most environments but not aggregated. To claim better sample complexity than TRPO specifically, the paper would need to compare them at the same number of timesteps, which Figure 3 does visually, but the absence of a summary metric or statistical test weakens the claim. On Atari, TRPO is not compared at all (the Atari baselines are A2C and ACER), so the "better sample complexity than TRPO" claim is supported only by the MuJoCo results, and only qualitatively through learning curves.
The paper's strongest evidence for sample complexity is the comparison against A2C on Atari (Table 2): PPO wins 30 of 49 games on the overall-training metric, which directly measures sample efficiency (faster learning from fewer timesteps). Against ACER, PPO wins on this metric but loses on final performance. The claim that PPO has "better sample complexity" is therefore true relative to A2C, true relative to TRPO on the MuJoCo environments tested (with the caveat about missing aggregation), and competitive-but-not-strictly-better relative to ACER. The unqualified "better sample complexity" in the abstract overstates the evidence.
Claim: PPO "empirically... outperforms other online policy gradient methods" and "overall strikes a favorable balance between sample complexity, simplicity, and wall-time"
The outperforms other online policy gradient methods claim is supported for the specific set of methods tested. On MuJoCo (Figure 3), PPO outperforms A2C, A2C+Trust Region, vanilla PG with adaptive stepsize, CEM, and TRPO on most environments. On Atari (Table 2), PPO outperforms A2C comprehensively and is competitive with ACER. However, the claim's scope is limited by the specific baselines chosen. Notable omissions:
-
DDPG (Lillicrap et al., 2015) and related off-policy continuous-control methods (TD3, SAC) are not compared. These are not strictly "policy gradient methods" in the online sense (they are off-policy actor-critic), but they are the dominant continuous-control methods in the years following this paper. The absence of these comparisons limits the claim's generality.
-
A3C (Mnih et al., 2016), the asynchronous version of A2C, is not directly compared. The paper states that A2C is "a synchronous version of A3C, which we found to have the same or better performance than the asynchronous version," but provides no data for this claim.
-
On Atari, DQN and its variants (Double DQN, Dueling DQN, Prioritized Experience Replay) are not compared. The paper restricts Atari baselines to A2C and ACER, both actor-critic methods. This is a reasonable scope for a policy-gradient paper, but it means the claim "outperforms other online policy gradient methods" does not address value-based methods that may achieve better sample efficiency or final performance on Atari.
The favorable balance between sample complexity, simplicity, and wall-time claim is qualitative and partially supported. Sample complexity is established relative to A2C and TRPO but not to the strongest off-policy methods. Simplicity is established relative to TRPO and ACER by code-complexity argument. Wall-time is never measured or reported—the paper uses timesteps as the compute metric, which does not account for the time spent in the optimization phase. PPO's multiple-epoch design (K=10 on MuJoCo) means it spends more wall-clock time per timestep on gradient computation compared to A2C (which takes one gradient step per batch) or ACER (which does one update per timestep). Whether PPO's wall-clock time is actually favorable depends on the relative cost of environment simulation vs. neural network optimization, which varies by domain and hardware. The paper's silence on wall-clock measurements means this part of the claim is unvalidated.
Claim: The clipped surrogate objective is a "pessimistic estimate (i.e., lower bound) of the performance of the policy"
This theoretical claim is supported by the mathematical construction of L_CLIP as min(unclipped, clipped) — since the min is always less than or equal to the unclipped L_CPI objective, L_CLIP is formally a lower bound. The paper provides empirical validation in Figure 2, which shows L_CLIP falling below L_CPI as the policy moves away from θ_old and peaking at the PPO update point. However, Figure 2 is from a single policy update on a single environment (Hopper-v1). It demonstrates the lower-bound property for that specific case but does not constitute a systematic validation across environments or training stages. The claim that L_CLIP is a "pessimistic estimate of the performance of the policy" is also somewhat loose—it is a pessimistic estimate of L_CPI, which is itself a surrogate (local approximation) of the true expected return. Whether L_CLIP is pessimistic relative to the true return depends on the quality of the advantage estimates and the accuracy of the CPI approximation, which are not addressed.
Missing Experiments That Would Strengthen the Paper
Several experiments are conspicuously absent and would substantially strengthen the paper's claims:
-
Direct PPO vs. TRPO comparison on Atari. TRPO is never tested on Atari, so the claim that PPO has TRPO's benefits "but is much simpler" is only validated on MuJoCo. Showing that PPO matches or exceeds TRPO on Atari would strengthen the generality claim.
-
Ablation of K (number of epochs). The paper demonstrates that multiple epochs with clipping work, but doesn't show how many epochs is optimal or what the benefit of multiple epochs is over single-epoch PPO. A sweep over K ∈ {1, 3, 5, 10, 15, 20} on a representative MuJoCo environment would quantify the sample-efficiency benefit of data reuse versus the risk of overfitting to stale advantages.
-
Dropout experiments. The paper claims PPO is compatible with dropout (unlike TRPO), but never tests this. A simple experiment showing that PPO with dropout achieves comparable performance to PPO without dropout, or that TRPO with dropout fails, would validate this practicality claim.
-
Wall-clock time comparison. Reporting the actual training time for PPO vs. A2C vs. ACER on Atari (or on MuJoCo) would substantiate the "favorable balance... wall-time" claim. The multiple-epoch design adds computational overhead that may partially offset the sample-efficiency gains when measured in wall-clock time rather than environment interactions.
-
Confidence intervals or statistical tests. The paper reports mean scores across seeds but never provides standard deviations, confidence intervals, or results of statistical significance tests. For the Atari comparison (Table 2), a game is "won" if the mean score across 3 trials is higher—with only 3 trials, these wins may not be statistically reliable. Reporting 95% bootstrap confidence intervals or performing a paired t-test per game would provide a clearer picture of which comparisons are robust.
-
Sensitivity to
γandλ. The paper usesγ = 0.99andλ = 0.95across all experiments without ablation. These parameters control the bias-variance tradeoff in advantage estimation and directly affect the signal that the clipped surrogate optimizes. Showing that PPO is robust to reasonable variations in these parameters would strengthen the claim that it "succeeds on a variety of problems without hyperparameter tuning." -
Comparison against a strong off-policy continuous-control method (e.g., DDPG or its successor TD3). The continuous-control landscape changed rapidly after this paper, and PPO is often compared against off-policy methods in subsequent work. The paper's claim that PPO "outperforms other online policy gradient methods" is narrowly true but less informative than a comparison against the strongest methods of the time regardless of category.
Conditions Under Which Claims Hold
The experimental results support the paper's claims under the following implicit conditions, which the paper does not always make explicit:
-
For MuJoCo continuous control: PPO with clipping (ε = 0.2) and a moderate number of epochs (K = 10) outperforms TRPO, A2C, and vanilla PG when using MLP policies with separate policy and value networks, trained for 1M timesteps. The advantage over fixed-KL-penalty methods is clear (score 0.82 vs. 0.72). Whether this advantage persists with larger networks, longer training, different MuJoCo tasks, or shared architectures is not tested.
-
For Atari: PPO learns faster than ACER (more games won on the overall-training metric) but achieves lower final performance on many games (fewer games won on the last-100-episodes metric). The method is substantially better than A2C. These results hold for the specific CNN architecture, hyperparameter settings (K = 3, T = 128, annealed ε and learning rate), and 40M-frame training budget used.
-
For high-dimensional humanoid control: PPO can learn effective locomotion and target-following policies when trained for 50–100M timesteps. Whether performance is better than alternatives is unknown because no baselines are provided for these tasks.
-
The clipping mechanism is necessary for multi-epoch optimization: The "no clipping or penalty" baseline (−0.39) decisively demonstrates that unconstrained multi-epoch optimization fails. The clipping mechanism (0.82) decisively outperforms the best adaptive KL penalty (0.74). These results hold for MuJoCo with the specific training configuration in Table 3.
-
The ε = 0.2 value is optimal for MuJoCo: The sweep in Table 1 supports this, but only three values were tested (0.1, 0.2, 0.3). A finer sweep might reveal that 0.15 or 0.25 performs even better, and the optimal ε may vary across environments—the aggregate score masks per-environment differences.
Overall, the experimental section is thorough for its time and provides strong evidence for the core claims about the clipped surrogate objective's effectiveness relative to KL-based methods and the viability of multi-epoch on-policy optimization. The primary weaknesses are the absence of statistical rigor (no confidence intervals), the limited ablation of key hyperparameters (K, T, learning rate schedule), the missing wall-clock measurements, and the narrow scope of baseline comparisons (no off-policy continuous-control methods, no DQN on Atari). These limitations do not undermine the central contribution—the clipped surrogate mechanism is convincingly demonstrated—but they leave open questions about PPO's absolute performance ceiling and its sensitivity to hyperparameter choices in new domains.
6. Limitations and Trade-offs
The Difficulty Estimation Cost Is Unaccounted for in the Efficiency Gains
The assumption or constraint. The compute-optimal framework relies on estimating question difficulty before allocating the inference-time budget. The paper's method for doing so is extraordinarily expensive: it requires generating 2048 complete solutions per question from the base model, then averaging either ground-truth correctness (oracle bins) or the PRM's final-answer score (predicted bins) to assign the question to one of five difficulty quintiles. The authors acknowledge this explicitly in Section 3.2, stating that "estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity."
The consequence. The headline 4× efficiency gains — matching best-of-256 performance using only 64 generations for revisions (Figure 8), or best-of-64 using only 16 generations for search (Figure 4) — are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be 2048 + best-of-N generations, not just best-of-N. For small generation budgets (e.g., 16 or 64), the difficulty estimation cost of 2048 generations dominates the total compute by factors of 32× to 128×, making the reported efficiency gains irrelevant in practice. Even if difficulty estimation could be amortized across many queries to the same question (e.g., in a batch evaluation setting), the upfront cost is prohibitive for one-off queries — which is the more common deployment scenario. The 4× figure should therefore be understood as an upper bound on achievable efficiency conditional on knowing difficulty, not as a realized deployment gain.
What evidence exists in the paper. The limitation is described qualitatively in Section 3.2, but no experiment quantifies its impact. The difficulty estimation cost is never included in any generation budget calculation, and no variant of the compute-optimal framework is tested with difficulty estimation cost factored in. The predicted-bins method (using PRM scores instead of ground-truth correctness) avoids the need for labels but does not reduce the sample cost — it still requires 2048 generations per question. The paper's Figures 4 and 8, which show compute-optimal scaling curves, plot performance as a function of the allocation budget only, completely ignoring the difficulty estimation overhead.
Mitigation status. The paper does not attempt to mitigate this limitation. It suggests future work on "training models to directly predict difficulty of a question" (Section 8) and frames the difficulty estimation cost as "an exploration-exploitation tradeoff," but no such model is developed or even sketched. The limitation is acknowledged but left entirely unresolved, meaning the practical deployment picture for the compute-optimal framework is incomplete — no practitioner can replicate the reported gains without first solving the difficulty estimation problem, and the paper provides no guidance for doing so.
Hard Problems Remain Essentially Unsolved by Test-Time Compute
The assumption or constraint. The paper's entire framework — search against PRM verifiers, iterative revision, and compute-optimal allocation — assumes that the base model has a non-trivial probability of producing a correct answer. On the hardest questions (difficulty bin 5, defined as the lowest quintile of pass@1), the base model's pass@1 is near zero. The paper states this directly in the Section 7 takeaway box: test-time compute provides "no benefit for problems that require capabilities well beyond the base model."
The consequence. Across all tested methods — PRM search, beam search, lookahead search, iterative revision, and compute-optimal combinations — the hardest questions show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for both best-of-N weighted and beam search, even at 256 generations. In Figure 7 (right), bin 5 revision accuracy sits at roughly 2–3% regardless of the sequential-to-parallel ratio, even at 128 generations. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% for both revisions and PRM search, far below the ~14× larger model's performance, even at favorable R ≪ 1 regimes. This means test-time compute fundamentally cannot substitute for pretraining on problems that are genuinely outside the base model's capability range. The paper's approach amplifies existing capability but does not create it from nothing — if the base model never (or almost never) generates a correct solution, no amount of search or revision will find one, because there is nothing to find.
What evidence exists in the paper. The difficulty-bin analyses in Figures 3 (right), 7 (right), and 9 all consistently show bin 5 performance near zero and flat across budget levels. The FLOPs-matched comparison in Figure 9 quantifies the gap explicitly: on hard problems, test-time compute with the smaller model underperforms the ~14× larger model by margins ranging from −3.6% (PRM search, R ≪ 1) to −52.9% (PRM search, R ≫ 1). The Section 7 takeaway box summarizes: "test-time compute does not help when the base model is incapable of producing correct solutions." This is not a failure of the method — it is a fundamental boundary condition, and the paper is transparent about it.
Mitigation status. The paper does not attempt to solve this problem, nor does it claim to. The limitation is inherent to the approach: test-time compute can only work with the outputs the base model can generate. The only path to solving hard problems is better pretraining (larger models, more data, better data quality), which the paper explicitly acknowledges. The compute-optimal framework's role is to identify which problems are in this regime (via difficulty estimation) so that compute is not wasted on them — but the framework itself cannot make them solvable. This boundary condition is clearly communicated, but it means the method offers no path forward for genuinely novel or out-of-distribution reasoning, which limits its applicability to the subset of problems where the base model already has non-trivial competence.
The PRM-Based Difficulty Estimation Requires 2048 Samples — But This Cost Is Never Included in the Efficiency Calculations, and No Cheaper Alternative Is Developed
The assumption or constraint. The compute-optimal scaling policy depends on knowing each question's difficulty bin. The paper's difficulty estimation method — generating 2048 samples and averaging the PRM's final-answer score — is the linchpin that makes adaptive allocation possible. The paper acknowledges this cost in Section 3.2, stating that "our experiments do not account for this cost largely for simplicity," and describes it as an exploration-exploitation tradeoff.
The consequence. The difficulty estimation cost fundamentally changes how the 4× efficiency gains should be interpreted. Suppose a practitioner wants to use the compute-optimal search policy on a batch of 1000 questions with a target budget equivalent to 64 generations per question. Using best-of-N weighted uniformly would cost 64,000 total generations. Using compute-optimal search requires: (1) 2048 × 1000 = 2,048,000 generations for difficulty estimation, plus (2) the variable allocation budget per question. Even if the allocation budget is 4× smaller per question (say, 16 generations on average), the total cost is 2,048,000 + 16,000 = 2,064,000 generations — roughly 32× more than uniform best-of-64. The 4× efficiency gain refers only to the allocation phase, not the end-to-end pipeline.
In a deployment setting where each question is unique and seen only once (the typical inference scenario), the difficulty estimation cost is borne per query and cannot be amortized. The compute-optimal framework, as presented, is therefore not deployable — it would be far cheaper to simply run best-of-N with a large N than to estimate difficulty for every query. The only setting where the framework works as described is when the same set of questions is evaluated repeatedly (difficulty estimation cost amortized across many evaluations of the same questions), which is a narrow use case.
What evidence exists in the paper. No experiment measures end-to-end cost including difficulty estimation. The 2048-sample difficulty estimation protocol is described in Section 3.2, and the compute-optimal scaling curves in Figures 4 and 8 are plotted with the allocation budget on the x-axis, with no indication of the difficulty estimation overhead. The paper does not report the total compute required for any experiment when difficulty estimation is included, nor does it compare compute-optimal scaling (with estimation cost) against uniform best-of-N at the same total budget. The predicted-bins method (using PRM scores instead of oracle correctness) is evaluated for its accuracy in binning questions but not for its cost — it still requires the full 2048 samples.
Mitigation status. The paper suggests several future directions in Section 8: "cheaper methods for estimating question difficulty, such as by pretraining or finetuning models to directly predict difficulty of a question." It also mentions "adaptive difficulty assessment: starting with a small number of samples to get a rough difficulty estimate, then allocating more compute to refine the estimate only if needed." However, none of these are implemented or tested. The paper also does not explore whether a much smaller number of samples (e.g., 16 or 32) could provide sufficient difficulty signal — the 2048 number is presented as a fixed protocol with no sensitivity analysis. The limitation is therefore acknowledged but completely unresolved in the current work, making the practical deployability of the compute-optimal framework an open question.
Results Are Demonstrated on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*), Leaving Generality Unestablished
The assumption or constraint. All experiments use the MATH benchmark (500 test questions, high-school competition math problems) with PaLM 2-S* (Codey) as the base model. The authors state in Section 4 that they "believe this model is representative of the capabilities of many contemporary LLMs," but this is an assertion, not an empirical finding. The PRM training procedure, the revision model training, the difficulty-dependent behavior of search algorithms, and the compute-optimal policy selection are all evaluated exclusively within this single model-benchmark combination.
The consequence. There are multiple dimensions of potential non-generality that are untested:
-
Domain specificity. MATH consists of competition-level math problems requiring multi-step symbolic reasoning with a single correct answer. Whether the difficulty-dependent patterns — beam search hurting easy problems via PRM over-optimization, revisions helping easy problems, no method helping hard problems — generalize to other reasoning domains (code generation, logical reasoning, scientific QA, planning) or to tasks requiring factual knowledge rather than inference is unknown. Code generation, for instance, has different error patterns (syntax errors vs. logical errors) that may interact differently with search and revision mechanisms.
-
Model specificity. PaLM 2-S* has specific calibration properties, error patterns, and in-context learning capabilities. A model with different characteristics — better or worse calibration, different tendency to produce repetitive outputs, different base pass@1 distribution — might exhibit different difficulty-dependent scaling curves. The PRM is trained on PaLM 2-S*'s output distribution, and the authors note that the PRM800k dataset (trained on GPT-4 outputs) was "largely ineffective" for their model, suggesting distribution shift is a real concern when transferring between model families.
-
Answer format specificity. MATH problems have exact answers that can be graded with string matching. This enables both the PRM training pipeline (Monte Carlo rollouts require checking whether completions reach the correct answer) and the difficulty estimation protocol (computing pass@1 from 2048 samples). Many important real-world tasks — open-ended generation, summarization, dialogue — lack such clean correctness signals, and extending the framework to those domains would require fundamentally different verifier training approaches.
-
Test set size. The 500-question test set, split into five difficulty quintiles of ~100 questions each and further split by two-fold cross-validation, means the compute-optimal policy is selected based on approximately 50 questions per fold per bin. This is a small sample for reliable strategy selection, and the paper provides no measure of how stable the policy is across different splits. A different random split of the 500 questions might produce a different compute-optimal policy.
What evidence exists in the paper. The entire experimental evaluation — Figures 3–9, the PRM training in Section 5.1, the revision model training in Section 6.1, the FLOPs-matched comparison in Section 7 — uses only MATH and PaLM 2-S*. There is no cross-domain experiment (e.g., testing on a code generation benchmark) and no cross-model experiment (e.g., testing with a different base LLM). The paper acknowledges in Section 8 that "future work should examine whether these findings hold for other model families."
Mitigation status. This limitation is not mitigated within the paper. The authors present the findings as a proof of concept for the compute-optimal test-time scaling framework, not as a claim of universal applicability. However, the strong assertions in the abstract and introduction — that the framework identifies optimal test-time compute strategies, that test-time compute can substitute for pretraining — are stated in general terms without the "on MATH with PaLM 2-S*" qualifier that the evidence actually supports. A practitioner reading the paper would need to independently validate whether the difficulty-dependent patterns replicate on their specific model and task domain, and the paper provides little guidance for how to assess transferability.
The ~14× Larger Model Baseline Is Not Compute-Optimally Trained, Weakening the Pretraining vs. Inference Tradeoff Analysis
The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters. The larger model is obtained by scaling parameters only while holding the training data fixed — a protocol matching the LLaMA model series (Touvron et al., 2023). The paper acknowledges this departs from compute-optimal pretraining (Hoffmann et al., 2022), which would scale both parameters and data equally. The authors state: "We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."
The consequence. A Chinchilla-optimal model trained with 14× more total FLOPs — with both parameters and data scaled up according to the square-root rule — would likely outperform a parameter-only-scaled model at the same total FLOPs budget. This means the pretraining baseline in the FLOPs-matched comparison is weaker than the best possible pretrained model at that compute budget. The reported advantages of test-time compute over pretraining — e.g., +27.8% relative improvement on easy questions at R ≪ 1 for revisions (Figure 1, top-right bar chart), or +19.1% for PRM search on easy questions at R ≪ 1 — may shrink or reverse against a properly compute-optimal larger model. Additionally, the ~14× larger model is evaluated using only greedy decoding — no majority voting, no best-of-N, no search, and no revisions. Giving the larger model even a modest test-time compute budget (say, best-of-8 or a few sequential revisions) would create a substantially stronger baseline. The comparison is therefore not "test-time compute vs. pretraining" in abstract, but "small model with aggressive test-time compute vs. larger model with no test-time compute."
What evidence exists in the paper. Section 7 describes the FLOPs accounting and the parameter-only scaling protocol. The results in Figure 9 show the performance of the ~14× larger model (greedy) marked as stars at three x-axis positions corresponding to R = 0.16, R = 0.79, and R = 22. There is no variant of the larger model with any test-time compute augmentation, and no compute-optimally-trained larger model baseline. The paper does not report what fraction of the 14× gap in FLOPs is consumed by additional parameters vs. what additional data would consume in a Chinchilla-optimal setup, making it impossible for a reader to estimate how much the baseline would improve under optimal pretraining.
Mitigation status. The paper explicitly flags this as a limitation and defers it to future work: "leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." However, this acknowledged limitation does not prevent the paper from making strong claims about the pretraining-inference tradeoff — including the statement in the abstract that a smaller model with test-time compute can outperform a ~14× larger model. The claim is technically true for the specific (weaker) baseline used, but a reader might incorrectly infer that this advantage holds against the best possible pretrained model at that compute budget, which the paper's own acknowledgement suggests it may not. The caveat could have been partially addressed by at minimum testing the larger model with a small test-time compute budget (e.g., best-of-4 majority voting), which would have been computationally cheap and would have provided a more informative baseline for the core claim.
The Revision Model Suffers from a 38% Correct-to-Incorrect Reversion Rate, and Revision Training Is Fragile, Limiting the Reliability of Sequential Refinement
The assumption or constraint. The revision model is fine-tuned on trajectories where all in-context answers are incorrect, followed by a correct answer. This training design means the model never sees an example where the current answer is already correct and should be preserved or output with minor refinements. At test time, when generating a chain of sequential revisions, the model may produce a correct answer at step k of the chain, then "revise" it into an incorrect answer at step k+1. The paper reports that approximately 38% of correct answers get converted back to incorrect ones (Section 6.1). The mitigation — using majority voting or verifier-based selection across the full chain to pick the best answer rather than always taking the last revision — is a patch, not a solution to the underlying model behavior.
The consequence. The 38% reversion rate means that longer revision chains are not monotonically improving — each additional revision step both adds a chance to correct an error and a chance to introduce a new error into a previously correct answer. The best answer in a chain might appear at step 3, be corrupted at step 4, and never be recovered. This places a fundamental limit on how much sequential depth can improve performance: beyond a certain chain length, the reversion rate may cause the expected best-answer-in-chain to plateau or decline. Figure 6 (left) shows that the revision model's per-step pass@1 improves from ~18.2% at step 1 to ~24–25% by steps 15–20 and remains in that range out to 64 steps — but this is per-step accuracy, not the accuracy of the best answer selected from the chain. The paper's within-chain selection mechanism (majority voting or verifier-based selection) can recover the best answer from anywhere in the chain, but this requires either multiple samples at each step (for majority voting) or a reliable verifier that can identify which answer is correct — and the paper notes that the base-model PRM does not transfer well to revision model outputs (Appendix J, Figure 15a), requiring a separate revision-specific ORM.
Additionally, the ReST^EM experiment (Appendix K, Figure 16) reveals that revision model training is fragile: attempting to further optimize the revision model using RL-style on-policy training caused performance to degrade substantially with sequential revisions (at 256 generations, fully sequential performance dropped to approximately 33.5% compared to roughly 38.5% at the optimal sequential-to-parallel ratio). The paper hypothesizes that on-policy data collection amplifies spurious correlations, but the underlying mechanism is not established. This fragility means the revision model's positive results depend on specific design choices (offline training data construction, edit-distance-based incorrect-correct pairing) that may not transfer to other settings and that are not fully understood.
What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1, though the paper does not provide a detailed breakdown of when and why reversions occur. Figure 6 (left) shows the per-step pass@1 trajectory. Appendix J (Figure 15a) shows the distribution shift between base-model PRM and revision-specific ORM for verifier-based selection. Appendix K (Figure 16) shows the ReST^EM failure mode. The paper does not report the expected best-answer-in-chain accuracy as a function of chain length (which would show the effect of the reversion rate on overall performance), nor does it analyze whether reversions are concentrated on certain problem types or difficulty levels.
Mitigation status. The paper uses within-chain selection (majority voting or verifier-based) to partially mitigate the reversion problem — rather than always taking the final revision, the system selects the best answer from anywhere in the chain. However, this mitigation is imperfect: majority voting requires generating multiple samples at each revision step to identify the consensus answer, and verifier-based selection requires a reliable verifier that generalizes to the revision model's output distribution (the paper shows the base-model PRM underperforms here). The paper suggests no approach to prevent reversions from occurring in the first place (e.g., training the model to recognize correct answers and output them unchanged, or training on trajectories that include correct-to-correct transitions). The ReST^EM failure (Appendix K) is acknowledged as a negative result but is not explained or resolved. The revision model approach, while empirically effective on aggregate metrics, therefore has a known failure mode (reversions) and a known fragility (training sensitivity) that a practitioner would need to monitor and potentially design around, with only partial guidance from the paper.
7. Implications and Future Directions
How This Work Changes the Landscape
PPO changed the landscape of deep reinforcement learning not by introducing a new theoretical framework or a fundamentally new class of algorithms, but by redefining what counts as a practical, deployable policy optimization method. Before PPO, the prevailing wisdom — encoded in the architecture of TRPO — was that stable policy optimization required explicitly measuring and constraining the distance between old and new policies, typically via KL divergence, and that doing so properly required second-order information (the Fisher information matrix). PPO demonstrated that this entire conceptual apparatus could be replaced by a three-line modification to the loss function: compute the probability ratio r_t(θ), clip it to [1-ε, 1+ε], and take the minimum of the clipped and unclipped objective. The clipped surrogate is not an approximation to TRPO's constrained optimization — it is a fundamentally different mechanism that achieves the same stability guarantees through objective design rather than constraint enforcement.
This was a methodological reframing with outsized practical consequences. The question shifted from "how do we constraint the optimizer?" to "how do we shape the objective so the optimizer naturally stops at a safe point?" Conceptually, this is a small move — the probability ratio was already used in TRPO's surrogate objective. But practically, it collapsed TRPO's several hundred lines of conjugate gradient, Fisher-vector product, and line search code into a loss function that any automatic differentiation library can optimize with a single .backward() call. The paper explicitly claims that PPO requires "only few lines of code change to a vanilla policy gradient implementation," and this is not an exaggeration — indeed, the adoption curve of PPO since 2017 validates this claim far more strongly than any ablation in the paper could. PPO became the default RL algorithm across domains (robotics, game playing, LLM fine-tuning via RLHF) precisely because it is the path of least resistance to a stable, reasonably sample-efficient policy optimizer.
The paper also resolved a latent tension in the literature between on-policy and off-policy methods. Before PPO, the narrative was: on-policy methods (A2C, TRPO) are stable but sample-inefficient because they discard data after one use; off-policy methods (DQN, DDPG, ACER) are sample-efficient but require complex importance-sampling corrections and are prone to instability. PPO staked out a third position: on-policy data can be reused for 3–15 epochs without off-policy corrections, provided the policy update is kept small enough by the clipping mechanism. This was not a theoretical result — the paper provides no new theory about why this works — but an empirical discovery with profound practical implications. It means practitioners can get much of the sample-efficiency benefit of experience replay without implementing any off-policy machinery. The K=10 epochs on MuJoCo (Table 3) and K=3 on Atari (Table 5) are concrete, reproducible settings that have been adopted as defaults in countless subsequent projects.
The comparison against the adaptive KL penalty variant (Section 4, Table 1) serves as a diagnostic that changed how the field thinks about policy regularization. The finding that a fixed schedule of clipping (ε = 0.2) outperforms even an optimally-adaptive KL penalty (best score 0.74 vs. 0.82) demonstrated that the penalty form itself — subtracting β·KL from the objective — is fundamentally inferior to the objective-shaping approach, regardless of how cleverly β is tuned. This is a specific, falsifiable claim that subsequent work could test and build on: the mechanism matters more than the adaptation schedule. The concurrent work by Heess et al. (2017), which used the adaptive KL variant for 3D locomotion, is implicitly shown to be using a suboptimal stabilization mechanism — the clipped surrogate would likely achieve better results on those same tasks.
However, it would be inaccurate to call PPO a paradigm shift. It did not introduce a new class of algorithms — it is still a policy gradient method, still uses advantage estimation, still alternates between sampling and optimization. What it did was identify and remove an unnecessary complexity barrier that was preventing widespread adoption of stable policy optimization. The contribution is primarily an engineering insight — that clipping the probability ratio in the objective is functionally equivalent to constraining the KL divergence, but is achievable with first-order methods — backed by thorough empirical validation across two very different domains (MuJoCo continuous control and Atari discrete control). The paper's lasting impact comes from the fact that this insight makes stable deep RL accessible to anyone who can write a PyTorch or TensorFlow training loop, rather than requiring specialized numerical optimization expertise.
Follow-Up Research This Work Enables
Establishing the theoretical foundations of clipping as a trust region mechanism. The paper provides no theoretical justification for why clipping the probability ratio at 1±ε works — it is presented as a heuristic motivated by the shape of the CPI objective and validated empirically through Table 1 and Figure 2. A natural follow-up would be to prove (or disprove) that optimizing L_CLIP with multiple SGD steps guarantees a bound on the KL divergence between the old and new policies, analogous to the guarantee that TRPO provides through its hard constraint. Specifically: given K steps of SGD on L_CLIP with learning rate η, can we bound max_t KL[π_θ_old(·|s_t), π_θ(·|s_t)] as a function of ε, K, and η? The fact that L_CLIP(θ) = L_CPI(θ) to first order at θ_old (as the paper notes) means the first SGD step is identical to an unconstrained policy gradient step — the clipping only activates on subsequent steps once r_t(θ) moves away from 1. Understanding the interaction between the number of epochs K, the learning rate, and the clipping threshold ε through a theoretical lens would transform PPO from an empirically-validated heuristic into a principled algorithm with known guarantees, and would provide concrete guidance for setting K and η as a function of ε rather than requiring per-domain tuning. A strong paper on this topic would also characterize failure modes: under what conditions does SGD on L_CLIP produce updates that violate the implicit trust region, and can these be detected at runtime?
Systematic characterization of the multi-epoch data reuse boundary. The paper demonstrates that K=10 epochs works on MuJoCo and K=3 works on Atari, but provides no framework for choosing K on a new domain. A systematic study sweeping K from 1 to (say) 50 across environments with varying dynamics (smooth vs. discontinuous rewards, deterministic vs. stochastic transitions, varying action space dimensionalities) could establish whether there is a predictable relationship between environment characteristics and the safe number of epochs. The key measurement would be: at what K does the policy's performance begin to degrade (as in the "no clipping or penalty" baseline's −0.39 score), and how does this K_max vary with ε? This study could also track the evolution of r_t(θ) across epochs — how many timesteps exit the [1-ε, 1+ε] interval at each epoch, and does the fraction that exits correlate with performance degradation? The paper's Table 1 shows that ε = 0.2 outperforms both ε = 0.1 and ε = 0.3, but it does not show whether this optimum is stable across K values — perhaps ε = 0.1 would outperform ε = 0.2 at K = 20 because the tighter clipping prevents more accumulation of policy change. This experiment would produce a practical "PPO operating envelope" chart that practitioners could consult when applying PPO to new problems.
PPO with dropout and recurrent architectures: testing the generality claims. The paper claims PPO is compatible with "architectures that include noise (such as dropout) or parameter sharing (between the policy and value function, or with auxiliary tasks)" — features that TRPO cannot handle. The parameter-sharing claim is validated by the Atari experiments (shared CNN with combined L_CLIP+VF+S loss), but the dropout claim is never tested. A direct experiment would add dropout to the MuJoCo MLP policy (e.g., dropout after each hidden layer with rates in {0.1, 0.2, 0.5}) and compare PPO with clipping against a TRPO baseline that attempts to use dropout (expected to fail or require modification). The same experiment extended to recurrent policies (LSTMs on Atari with truncated BPTT) would test whether PPO's simplicity enables stable training of architectures that were previously difficult to combine with trust region methods. A strong result would be demonstrating that PPO with a recurrent dropout policy achieves better sample efficiency than the feedforward Atari baseline in Table 5, while TRPO cannot be run at all on the same architecture without algorithmic modifications. This would convert the paper's untested generality claims into demonstrated capabilities and potentially open up new architecture-design possibilities for deep RL.
Diagnosing and mitigating the Atari final-performance gap relative to ACER. Table 2 reveals a clear pattern: PPO wins 30 games on the "average over all training" metric (fast learning) but only 19 on the "last 100 episodes" metric (final performance), while ACER shows the reverse (18 and 28 wins, respectively). This suggests PPO converges prematurely or forgets good behaviors on some fraction of games. A follow-up study could identify which game characteristics predict PPO underperformance vs. ACER — is it games requiring long-term credit assignment (where off-policy replay helps), games with sparse rewards (where exploration is critical), or games with highly stochastic dynamics (where importance sampling corrections matter more)? The study could then test modifications to PPO that close the gap without introducing full off-policy replay: for example, a small replay buffer of recent on-policy data from the last M iterations (partially off-policy, corrected by the clipping mechanism), a KL penalty toward an exponential moving average of past policies (to prevent rapid forgetting), or an auxiliary loss that encourages the value function to remain accurate on states from several iterations ago. The goal would be to retain PPO's simplicity while matching ACER's final performance, producing a "PPO-v2" that wins on both metrics.
Scaling PPO to massively parallel data collection. The paper's Algorithm 1 describes N parallel actors, but the experiments use modest N values (8 for Atari, not explicitly stated for MuJoCo). Modern RL systems often run thousands of parallel environments. A study of PPO's behavior as N scales from 8 to (say) 1024, holding the total timesteps NT constant, would characterize how the effective sample size interacts with the clipping mechanism. With more parallel actors, the batch NT is larger and more diverse (covering more independent trajectories), which might allow more aggressive optimization (higher K or larger ε) because the advantage estimates have lower variance. Alternatively, the policy might change more per iteration because the gradient is estimated more accurately, leading to earlier clipping activation. The experiment would sweep N while monitoring the fraction of probability ratios that hit the clip boundary at each epoch, the achieved KL divergence, and the final policy performance. This would provide guidance for practitioners scaling PPO to large distributed systems — a setting where PPO's simplicity (no need for distributed synchronization beyond parameter updates) is a major advantage over methods like A3C that rely on asynchronous updates.
PPO as the optimization backbone for RLHF and LLM alignment. Since this paper's publication, PPO has become the standard algorithm for reinforcement learning from human feedback (RLHF), where a language model's policy is fine-tuned using a reward model trained on human preferences. This application was not anticipated in the 2017 paper, but it inherits PPO's key design choices directly: the base language model serves as π_θ_old, the fine-tuned model is π_θ, and the clipping mechanism prevents the fine-tuned model from diverging too far from the pretrained model (preserving general language capabilities). A systematic study applying the exact PPO variants from this paper — clipping, adaptive KL penalty, fixed KL penalty — to the RLHF setting, measuring both reward optimization and retention of downstream task performance, would validate whether the lessons from MuJoCo and Atari transfer to this very different domain. Specifically: does the clipped surrogate (ε = 0.2) outperform the KL penalty variants for RLHF as it did for continuous control? Does K need to be much smaller (as in Atari's K=3) because language model policies are more sensitive to stale advantage estimates? The paper's thorough hyperparameter tables (Tables 3–5) provide a template for such a study, and the finding that PPO with clipping is compatible with parameter sharing (Section 3, Equation 9) is directly relevant since RLHF typically uses the same transformer for both policy and value function.
Practical Applications and Downstream Use Cases
Default policy optimizer for new continuous-control RL projects. PPO has become the first algorithm a practitioner should try when facing a new continuous control problem (robotics, autonomous vehicle control, industrial process optimization). The paper provides a complete recipe: two hidden layers of 64 units with tanh activations, Gaussian policy with learned standard deviation, horizon T=2048, K=10 epochs, M=64 minibatch, Adam stepsize 3e-4, γ=0.99, λ=0.95, and ε = 0.2. This configuration achieves a 0.82 average normalized score on the MuJoCo benchmark (Table 1) and outperforms TRPO, A2C, CEM, and vanilla PG on 6 of 7 tested environments (Figure 3). The value proposition is not just the final performance — it is that this configuration works without per-task hyperparameter tuning across HalfCheetah, Hopper, Walker, Swimmer, Reacher, InvertedPendulum, and InvertedDoublePendulum, which span different state/action dimensionalities, reward structures, and dynamics. A robotics engineer starting a new project can copy these hyperparameters, implement the three-line loss function modification, and expect a working baseline within hours rather than days of tuning. The paper's ablation of ε (Table 1) provides a clear tuning knob if the default 0.2 doesn't work: try 0.1 for more conservative updates or 0.3 for more aggressive learning, with the knowledge that 0.2 is the robust center point.
Atari and discrete-action game playing with minimal implementation complexity. For discrete-action domains like Atari, PPO provides a simpler alternative to ACER with competitive aggregate performance and faster initial learning. The paper's Table 2 shows PPO winning 30 of 49 games on the overall-training metric (measuring speed of learning) vs. ACER's 18, while ACER wins 28 vs. PPO's 19 on final performance. For applications where wall-clock training time or early performance matters more than asymptotic optimality — competition settings with fixed training budgets, rapid prototyping of game AI, or educational contexts where students need to see results quickly — PPO is the better choice. The implementation requires only the clipped surrogate loss (Equation 7) plus standard advantage estimation (GAE with γ=0.99, λ=0.95), with a shared CNN architecture and the combined loss from Equation 9. ACER, by contrast, requires truncated importance sampling, a Retrace operator, a trust region in Q-function updates, and a separate deterministic policy network — all of which introduce additional hyperparameters and failure modes. The paper's hyperparameters in Table 5 (T=128, K=3, N=8 actors, annealed ε and learning rate) provide a turnkey configuration that can be applied directly to new discrete-action environments. The fact that PPO achieves meaningful scores on hard exploration games like Montezuma's Revenge (42.0 vs. ACER's 0.3, Table 6) — without specialized exploration bonuses — suggests it has some intrinsic exploration advantage, possibly because the clipping mechanism prevents the policy from prematurely collapsing to a deterministic strategy.
RLHF fine-tuning of large language models. Though not anticipated by the paper, PPO's architecture maps perfectly onto the RLHF pipeline that has become standard for aligning LLMs with human preferences. The pretrained language model serves as π_θ_old, providing the reference distribution from which the policy should not diverge too far (preserving fluency, factuality, and general capabilities). The fine-tuned model is π_θ, optimized to maximize a learned reward model while the clipped surrogate prevents it from overfitting to the reward signal at the expense of language quality. The probability ratio r_t(θ) = π_θ(token | context) / π_θ_old(token | context) is computed token-by-token, and the clipping mechanism directly prevents the fine-tuned model from dramatically increasing or decreasing token probabilities relative to the pretrained model — which is precisely what causes language models to collapse into repetitive or nonsensical outputs when fine-tuned with unconstrained RL. The adaptive KL penalty variant (Section 4) is often used in this setting because it provides an interpretable knob (d_targ) that directly controls how much the model is allowed to change, which is important when the cost of a bad update is high (a fine-tuned model that loses coherence may need to be discarded and retrained). The paper's finding that the clipped surrogate outperforms KL penalties (Table 1) is directly relevant: RLHF practitioners using KL penalties may achieve better reward model alignment by switching to clipping, or may benefit from combining both mechanisms.
Rapid prototyping in research and education. PPO's primary practical advantage — that it requires "only few lines of code change to a vanilla policy gradient implementation" — makes it the natural choice for research projects that need a reliable policy optimizer without the overhead of implementing and debugging complex RL algorithms. A PhD student who wants to test a new neural network architecture for policy representation, a new exploration bonus, or a new auxiliary task can drop PPO into their codebase, use the default hyperparameters from Table 3 (MuJoCo) or Table 5 (Atari), and focus their effort on the novel component rather than on making the RL algorithm work. The paper's comprehensive hyperparameter documentation and the fact that the same ε = 0.2 works across 7 MuJoCo environments without tuning mean that PPO serves as a stable experimental platform — changes in performance can be attributed to the research idea rather than to RL algorithm instability. This is a qualitatively different role than TRPO played: TRPO was an algorithm you studied and carefully implemented; PPO is an algorithm you install and build on top of.
When to Prefer This Method
The paper positions PPO explicitly against TRPO (simpler, more general, better sample complexity), A2C/A3C (better sample complexity, similar simplicity), and ACER (similar performance, much simpler). The decision rules below are grounded in the paper's empirical results and explicit design claims.
Prefer PPO over TRPO when:
- You need to use dropout, parameter sharing between policy and value function, or recurrent architectures — the paper claims PPO is compatible with these (Section 1), while TRPO's Fisher matrix computation becomes ill-defined or intractable.
- Implementation simplicity matters — PPO requires "only few lines of code change to a vanilla policy gradient implementation" (Section 1), while TRPO requires conjugate gradient, Fisher-vector products, and line search.
- You are working on continuous control tasks — PPO with ε = 0.2 outperforms TRPO on 6 of 7 MuJoCo environments (Figure 3) with the same network architecture.
- You are willing to trade a small amount of asymptotic performance for faster initial learning — Table 2 shows PPO wins 30 games on the overall-training metric vs. ACER's 18, though ACER wins 28 on final performance vs. PPO's 19.
Prefer TRPO over PPO when:
- You need theoretical guarantees on monotonic improvement — TRPO is derived from a lower bound on expected return (the CPI theory from Kakade and Langford, 2002), while PPO's clipping mechanism is empirically motivated without formal guarantees.
- You are working in a setting where second-order information is cheap to compute and the implementation complexity is acceptable — TRPO may achieve similar or slightly better final performance on specific tasks (the Hopper curves in Figure 3 are nearly identical, suggesting domain-specific parity).
- The paper does not demonstrate PPO outperforming TRPO on Atari — TRPO is not tested on Atari, so the preference for PPO in discrete-action domains is based on simplicity and compatibility arguments rather than head-to-head performance evidence.
Prefer PPO over ACER when:
- Implementation complexity is a primary concern — ACER requires off-policy corrections (truncated importance sampling, Retrace operator), a moving average policy network for trust region enforcement, and separate deterministic and stochastic policy heads, while PPO needs only the clipped surrogate loss.
- Fast initial learning is more important than asymptotic final performance — Table 2 shows PPO wins the overall-training metric decisively (30 vs. 18 for ACER).
- You are working with continuous action spaces — ACER is designed for discrete actions, while PPO's clipped surrogate works identically for continuous (Gaussian) and discrete (categorical) policies.
Prefer ACER over PPO when:
- Asymptotic final performance on discrete-action tasks is the primary metric — ACER wins 28 games on the last-100-episodes metric vs. PPO's 19 (Table 2).
- You are working with environments where off-policy experience replay provides a decisive advantage — ACER's replay buffer enables learning from rare, high-value experiences that PPO may sample once and discard, which explains ACER's large leads on games like DemonAttack (38,808 vs. 11,378), Gopher (37,802 vs. 2,933), and VideoPinball (156,226 vs. 37,389) in Table 6.
- Your environment has very long episodes where PPO's fixed-length trajectory segments (
T=128for Atari) systematically truncate important long-horizon credit assignment — ACER's replay buffer can stitch together information across episode boundaries.
Prefer PPO over A2C/A3C always for sample efficiency: The paper demonstrates decisively that PPO outperforms A2C on both MuJoCo (Figure 3 — PPO above A2C on all 7 environments) and Atari (Table 2 — A2C wins only 1 of 49 games on each metric). Since PPO has comparable implementation complexity to A2C (both are first-order actor-critic methods with GAE), there is little reason to prefer A2C over PPO on sample-efficiency grounds. A2C may be preferred only when the multiple-epoch optimization (K=3 to K=10) introduces unacceptable wall-clock overhead relative to environment simulation time, or when the implementation must be absolutely minimal (A2C is marginally simpler since it doesn't require storing old-policy log-probabilities or computing the clipping operation).
