Reinforcement learning from human feedback, commonly abbreviated RLHF, is a technique for shaping a model's behavior to match human preferences when those preferences are difficult to specify as an explicit, hand-coded objective. Rather than writing a formula for what makes a response "good," RLHF collects human judgments comparing pairs of model outputs, trains a separate reward model to predict those judgments, and then uses Reinforcement learning to fine-tune the original model so that it produces outputs the reward model scores highly.
Origins
The core idea, learning a reward function from human preference comparisons rather than a hand-specified reward, was developed by researchers including Paul Christiano in work on "deep reinforcement learning from human preferences" published in 2017, initially applied to control and game-playing agents rather than language. The technique's significance for AI broadened dramatically when OpenAI applied it to language models, most notably in the 2022 InstructGPT paper led by contributors that included John Schulman, which showed that a comparatively small amount of human feedback could make a language model far more helpful and aligned with user intent than simply scaling up next-token Pretraining alone. RLHF became the key technique that turned a raw base Large language model into the kind of conversational assistant popularized by ChatGPT in November 2022.
The three-step process
RLHF as applied to language models typically proceeds in three stages. First, a pretrained base model undergoes supervised Fine-tuning on a curated dataset of high-quality example responses, producing an initial instruction-following model. Second, human annotators are shown multiple outputs generated by the model for the same prompt and rank or compare them; these comparisons train a separate reward model to predict which output a human would prefer. Third, the language model is further fine-tuned using a reinforcement learning algorithm, most commonly Proximal Policy Optimization (PPO), where the model generates responses, the reward model scores them, and the model's parameters are updated to increase the likelihood of high-scoring responses, typically while a penalty term keeps the model from drifting too far from its original behavior.
Effects and limitations
RLHF has proven effective at making models more helpful, better at following instructions, and less likely to produce overtly harmful or offensive content, and it is a standard stage in training essentially every major commercial assistant, including Claude (AI model family) and Gemini. It is not without drawbacks. Because the reward model is only an approximation of true human preference, a language model optimized aggressively against it can learn to exploit quirks in the reward model rather than genuinely improving, a specific instance of Reward hacking; observed symptoms include models that produce longer responses than necessary because length is spuriously correlated with higher reward scores, or that adopt an overly agreeable, sycophantic tone because human raters tend to prefer responses that flatter or agree with them. Collecting the human comparison data needed for RLHF is also labor-intensive and expensive, and the resulting reward model can encode the particular biases or blind spots of whichever population of annotators, often contracted through firms such as Scale AI, provided the comparisons.
Alternatives
RLHF's complexity, requiring a separate reward model and an often unstable reinforcement learning training loop, motivated the development of simpler alternatives. Direct Preference Optimization, introduced in 2023, reformulates the same underlying preference-learning problem as a direct supervised loss on preference pairs, achieving comparable results without training a separate reward model or running reinforcement learning at all. Constitutional AI, developed by Anthropic, replaces much of the human feedback with AI-generated feedback guided by a written set of principles, reducing the volume of human labeling required. Despite these alternatives, RLHF, particularly the human preference-data collection step, remains a widely used component of modern alignment pipelines, whether combined with classic reinforcement learning or newer, more direct training objectives.