GPT-3 (Generative Pretrained Transformer 3) is a Large language model developed by OpenAI and introduced in a May 2020 paper, "Language Models are Few-Shot Learners," with broader API access following through the rest of 2020. Like its predecessor GPT-2, GPT-3 is built on the Transformer (architecture) architecture and trained with an autoregressive next-token objective, but scaled up dramatically to 175 billion parameters, more than 100 times larger than GPT-2's full model, trained on a mixture of Common Crawl, books, and Wikipedia text.
Few-shot learning
GPT-3's central research contribution was demonstrating that a sufficiently large language model could perform a wide range of tasks, including translation, question answering, arithmetic, and simple reasoning, given only a handful of examples in its prompt, without any gradient-based Fine-tuning. This capability, described by OpenAI researchers including lead author Tom Brown as Few-shot learning and more generally as In-context learning, suggested that scale alone could substitute for task-specific training, a finding that reinforced emerging beliefs about Scaling laws in language modeling and shaped subsequent research priorities across the field.
Commercial launch
Rather than releasing GPT-3's weights publicly, as it had eventually done with GPT-2, OpenAI made GPT-3 available exclusively through a paid API beginning in June 2020, initially through a private beta. This decision established a business model, later adopted widely across the industry, in which frontier language models are accessed as a metered cloud service rather than downloaded and run locally, a contrast with Open-weights models models released by competitors in later years. The API quickly attracted a wave of startups building products on top of GPT-3, including early AI writing assistants, chatbots, and code-generation tools, and Microsoft separately obtained an exclusive license to GPT-3's underlying model weights in 2020 as part of its broader investment in OpenAI.
Impact and limitations
GPT-3 was widely covered in mainstream and technology press as evidence that language models were approaching qualitatively new capabilities, and it directly informed the development of instruction-tuned variants trained with techniques including RLHF, work that culminated in the November 2022 release of ChatGPT, built on a GPT-3.5 variant. GPT-3 also drew substantial criticism for its tendency to produce fluent but factually incorrect output, an early prominent example of what the field later termed Hallucination (AI), as well as for reproducing biases present in its training data. Its successor, GPT-4, was released in March 2023 with substantially expanded capabilities including multimodal input.