# Policy gradient

Policy gradient methods are a class of reinforcement learning algorithms that directly optimize a parameterized policy function via gradient ascent, without relying on a value function. They are a sub-class of policy optimization methods used in machine learning and artificial intelligence.

Policy gradient methods are a class of reinforcement learning algorithms and a sub-class of policy optimization methods. Unlike value-based methods, which learn a value function to derive a policy, policy optimization methods directly learn a policy function that selects actions without consulting a value function. For policy gradient to apply, the policy function is parameterized by a differentiable parameter, typically denoted as theta, and the goal is to maximize expected cumulative reward through gradient ascent on the policy parameters.

These methods are central to modern [reinforcement learning](https://www.wikiprompt.org/wiki/reinforcement-learning) and have been applied in areas such as robotics, game playing, and autonomous systems. They are particularly useful in environments with continuous action spaces, where value-based methods often struggle. Policy gradient methods are also studied under the title of "Monte Carlo gradient estimation" because they rely on stochastic sampling to estimate the gradient.

## Overview

In policy-based reinforcement learning, the actor is a parameterized policy function that maps states to a probability distribution over actions. For a given state, the policy outputs probabilities for each possible action, with the sum or integral over all actions equal to 1, depending on whether the action space is discrete or continuous. The goal is to find parameters that maximize the expected episodic reward, which is defined as the discounted sum of rewards over a time horizon, starting from an initial state.

The policy gradient is the gradient of this expected reward with respect to the policy parameters. Different policy gradient methods stochastically estimate this gradient in different ways, but all aim to iteratively improve the policy by ascending the gradient. The key challenge is obtaining an unbiased and low-variance estimate of the gradient, which has led to various techniques such as baselines and actor-critic architectures.

## REINFORCE

The REINFORCE algorithm, introduced by Ronald J. Williams in 1992, was the first policy gradient method. It is based on a fundamental identity that expresses the policy gradient as an expectation over trajectories of the product of the gradient of the log-policy and the total reward. A key improvement is the "causality trick," which weights each action only by rewards from that timestep onward, reducing variance without introducing bias. REINFORCE is a Monte Carlo method, meaning it uses complete episodes to estimate the gradient, which can lead to high variance but is simple to implement.

## Actor-Critic Methods

Actor-critic methods combine policy gradient with value function approximation to reduce variance. The actor is the policy network, while the critic estimates the value function, which is used to compute a baseline or an advantage function. This allows for more stable and sample-efficient learning compared to pure REINFORCE. Notable examples include A2C (Advantage Actor-Critic) and A3C (Asynchronous Advantage Actor-Critic), which have been widely used in deep reinforcement learning.

## Modern Variants

Modern policy gradient methods include Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO), which constrain the policy update to avoid destructive large steps. These methods have become standard in deep reinforcement learning due to their reliability and performance. They are used in training agents for games like Dota 2 and StarCraft II, as well as in robotics and autonomous driving research.

## Applications and Challenges

Policy gradient methods have been applied in various domains, including [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) research, [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) systems, and [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) frameworks. They are particularly effective for continuous control tasks, such as robotic manipulation and locomotion. However, they face challenges such as high sample complexity and sensitivity to hyperparameters. Research continues to address these issues, with advances in [neural-network](https://www.wikiprompt.org/wiki/neural-network) architectures and optimization techniques.

Policy gradient methods are also relevant to [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) training, where reinforcement learning from human feedback (RLHF) uses policy gradient-like updates to align models with human preferences. This connection highlights the broad applicability of these algorithms beyond traditional reinforcement learning settings.

---
Source: https://www.wikiprompt.org/wiki/policy-gradient
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-07T02:34:12.090376+00:00