Proximal policy optimization (PPO) is a reinforcement learning (RL) algorithm for training an intelligent agent. Specifically, it is a policy gradient method, often used for deep RL when the policy network is very large. PPO was introduced in 2017 as an approximation of Trust Region Policy Optimization (TRPO), designed to improve training stability without the computational cost of computing second-order derivatives. Since 2018, PPO has been the default RL algorithm at OpenAI and has been applied to domains such as robotic arm control, playing Atari games, and beating professional players at Dota 2 through the OpenAI Five project.
PPO belongs to the family of policy gradient methods, which optimize a policy directly by estimating gradients of expected reward. Unlike value-based methods such as Deep Q-Networks (DQN), PPO is an on-policy algorithm, meaning it updates the policy using data collected from the current policy. It supports both discrete and continuous action spaces, making it versatile for a wide range of environments.
Background: Trust Region Policy Optimization
The predecessor to PPO, Trust Region Policy Optimization (TRPO), was published in 2015. TRPO addressed the instability issues of DQN by using a trust region method to limit the KL divergence between the old and new policies. This constraint ensures that the policy does not change too drastically during an update, which helps maintain stable learning. However, TRPO enforces this constraint by computing the Hessian matrix, a matrix of second-order derivatives, which is computationally expensive and inefficient for large-scale problems. This limitation motivated the development of PPO, which approximates TRPO's constraint without requiring the Hessian.
The PPO Algorithm
PPO simplifies TRPO by replacing the KL divergence constraint with a clipped objective function. The core idea is to limit the policy update by clipping the probability ratio between the new and old policies. This clipping mechanism prevents excessively large updates, which can destabilize training. The objective function is designed to provide a lower bound on the policy improvement, ensuring that updates are conservative but still effective.
The algorithm typically follows these steps:
- Collect trajectories by running the current policy in the environment.
- Compute rewards-to-go and advantage estimates (e.g., using generalized advantage estimation).
- Update the policy by maximizing the clipped surrogate objective, often using stochastic gradient ascent.
- Optionally, update a value function to improve advantage estimation.
PPO is known for its simplicity and ease of implementation compared to TRPO, while achieving comparable or better performance in many tasks. It has become a standard baseline in reinforcement learning research.
Applications and Impact
PPO has been widely adopted in both research and industry. At OpenAI, it became the default RL algorithm in 2018, used in projects such as OpenAI Five, which in 2019 defeated the reigning world champions in Dota 2. PPO has also been used for robotic control, including training a robotic arm to perform manipulation tasks, and for playing Atari games, where it achieved superhuman performance on many titles.
The algorithm's stability and sample efficiency have made it a popular choice for fine-tuning large language models, particularly in the context of reinforcement learning from human feedback (RLHF). Many modern large language models have been aligned using PPO or variants thereof, contributing to the development of generative AI systems.
Comparison with Other Methods
PPO is often compared with other policy gradient algorithms such as A2C (Advantage Actor-Critic) and DDPG (Deep Deterministic Policy Gradient). Unlike A2C, which uses multiple parallel environments, PPO can work with a single environment and uses importance sampling to reuse data. Compared to DDPG, which is off-policy and deterministic, PPO is on-policy and stochastic, making it more robust to hyperparameter variations. PPO's clipped objective also provides a simpler alternative to TRPO's trust region, reducing computational overhead while maintaining stability.
Limitations and Extensions
Despite its success, PPO has limitations. It can be sensitive to the clipping parameter and the choice of advantage estimation method. It also requires careful tuning of hyperparameters such as learning rate and mini-batch size. Researchers have proposed extensions such as PPO-λ, which incorporates generalized advantage estimation, and variants that adaptively adjust the clipping range. Additionally, PPO's on-policy nature can be sample-inefficient compared to off-policy methods, though this is often offset by its stability.