Direct Preference Optimization, commonly abbreviated DPO, is a method for training a language model to prefer certain outputs over others based on human preference data, introduced as a simpler alternative to RLHF. Rather than training a separate reward model and then running a reinforcement learning algorithm to optimize the language model against it, DPO reformulates the same preference-alignment problem mathematically so that the language model can be trained with a single, direct supervised loss computed straight from pairs of preferred and rejected responses, with no reward model and no reinforcement learning loop required.
Origins
DPO was introduced in a 2023 paper from Stanford researchers, who showed that the reinforcement learning objective typically used in RLHF has a closed-form optimal solution that can be expressed directly in terms of the language model's own output probabilities. This theoretical result meant that a model could be trained toward the same optimum RLHF pursues by simply increasing the probability it assigns to preferred responses relative to rejected ones, using a loss function derived from the same underlying preference model, the Bradley-Terry model of pairwise comparisons, that reward models in RLHF are trained with.
How it works
DPO training starts from the same kind of data RLHF's reward-modeling stage uses: a dataset of prompts, each paired with a "chosen" response that human annotators preferred and a "rejected" response that they did not. Rather than using this data to train a separate reward model, DPO trains the language model directly, computing a loss that increases the relative likelihood of the chosen response over the rejected one, measured against a reference version of the model, typically the model's own state before this training step, which acts as an implicit regularizer to prevent the model from drifting too far from sensible, fluent behavior. Because it eliminates the sampling and reward-scoring loop central to reinforcement learning, DPO training is considerably simpler to implement, more stable, and less computationally expensive than a full RLHF pipeline.
Relationship to RLHF
DPO and RLHF pursue the same underlying goal: aligning a language model's outputs with human preference judgments, typically as a stage following supervised Fine-tuning of a pretrained base model. RLHF does this indirectly, via an explicit reward model and a Reinforcement learning optimization loop, most often Proximal Policy Optimization. DPO collapses these two stages into one supervised objective, avoiding the instability, hyperparameter sensitivity, and infrastructure complexity that reinforcement learning training can introduce, at the cost of somewhat less flexibility, since DPO relies on the preference dataset itself rather than a reward model that could in principle be queried on responses beyond the original training data.
Adoption and impact
Following its introduction, DPO and closely related preference-optimization variants were adopted quickly across both industry and open research, in part because of the significant engineering simplification it offered: many teams without the infrastructure to run a stable reinforcement learning training loop could still perform preference alignment using standard supervised-learning tooling. It became a common technique in the training of open-weight models released by organizations such as Hugging Face and Mistral AI, and variants of preference optimization appear in the published training recipes of numerous models released from 2023 onward. Some derivative methods have since proposed refinements, such as removing the reference-model term or extending preference optimization to more than two candidate responses per comparison.
Limitations
Because DPO learns directly from a fixed, pre-collected preference dataset rather than an explicit reward model that can score arbitrary new outputs, it is generally considered less able to generalize to preference judgments about text distributions quite different from the training data, and it inherits RLHF's core dependency on the quality and consistency of the underlying human preference labels: annotator bias, inconsistency, or a narrow set of preference criteria will shape the resulting model's behavior just as directly as it would in a reward-model-based pipeline. As with RLHF, DPO does not by itself guarantee a model is truthful or safe; it optimizes toward whatever the preference data rewards, which remains an active area of research and debate within AI alignment work.