InstructGPT is a family of large language models developed by OpenAI, first introduced in January 2022. These models are designed to better follow user instructions and align with human intentions compared to earlier GPT-3 models. They employ a training technique called reinforcement learning from human feedback (RLHF), where human raters rank model outputs, and the model is guided to produce responses that are more helpful, truthful, and less harmful.
InstructGPT builds upon the foundation of the generative pre-trained transformer (GPT) architecture, a type of Transformer (architecture) model that originated in the 2017 introduction of the transformer architecture. The original GPT-3, released by OpenAI in 2020, was a decoder-only transformer with 175 billion parameters, but it could sometimes produce outputs that were off-task or misaligned with user intent. InstructGPT aimed to correct these issues by refining the model with human feedback.
Training and methodology
The InstructGPT models were created by fine-tuning GPT-3 with supervised learning and then applying reinforcement learning from human feedback. Human raters were asked to compare responses generated by different models, and their preferences were used to train a reward model. The policy (i.e., the model) was then updated to maximize this reward while also incorporating a penalty for deviations from the original GPT-3 distribution, to avoid excessive drift.
OpenAI reported that InstructGPT models were significantly better than GPT-3 in following instructions, generating fewer invented facts (a phenomenon known as hallucinations), and producing somewhat less toxic content, while also showing improved performance on certain tasks like summarization and Q&A.
Capabilities and limitations
InstructGPT inherited the core capabilities of GPT-3, including the ability to generate coherent text, answer questions, translate languages, and write code. It could also handle few-shot and zero-shot prompts. However, like its predecessors, it could still generate harmful or biased content when prompted with toxic inputs. OpenAI implemented content moderation tools for developers using the API, and as of January 2022, InstructGPT became the default model for the OpenAI API customers.
Comparison with predecessors
Compared to the original GPT-3 models, InstructGPT demonstrated notable improvements in user-intent alignment, as evidenced by human evaluations. For example, in machine-learning studies, evaluators judged InstructGPT outputs as more favorable than those from GPT-3 across a variety of prompts, including reducing unintended off-topic responses. Despite this, InstructGPT retained limitations common to large language models, such as sensitivity to phrasing and occasional factual errors.
The development of InstructGPT reflects broader trends in generative artificial intelligence, where models are trained to be more aligned with human ethics and safety standards. This approach has influenced subsequent models like ChatGPT, which use a similar alignment methodology.
Reception and impact
InstructGPT contributed to the growing ecosystem of AI models focused on helpful and safe AI. Researchers at other organizations, such as Anthropic and Google DeepMind, also explored similar alignment techniques. The model was a step toward more practical AI assistants, but it also raised discussions about the risks of AI-generated misinformation and the importance of transparency in AI systems. As of 2025, OpenAI continued to iterate on alignment techniques, building on the foundation established by InstructGPT.
See also
Arbitrary links can be made to related topics like Machine learning, Transformer (architecture), OpenAI, and Generative AI.