Generative Pre-trained Transformer 3 (GPT-3) is a Large language model released by OpenAI in 2020 as part of the company's GPT series. It is a decoder-only Transformer (architecture) model of Deep learning neural network architecture, superseding recurrence and convolution-based designs with an attention mechanism that allows the model to focus selectively on relevant segments of input text. GPT-3 has 175 billion parameters, each stored with 16-bit precision, requiring 350GB of storage, and a context window of 2,048 tokens. It demonstrated strong zero-shot and few-shot learning abilities across many tasks.
On September 22, 2020, Microsoft announced it had licensed GPT-3 exclusively. Others could still receive output from its public API, but only Microsoft had access to the underlying model.
Background
Improvements in algorithms, more powerful computers, and increased digitized material fueled a revolution in Machine learning during the 2010s, leading to rapid improvements in tasks including language manipulation. Software models are trained using thousands or millions of examples in structures loosely based on the neural architecture of the brain. One architecture used in natural language processing is a neural network based on the transformer architecture, introduced in 2017. This enabled systems capable of processing, mining, organizing, connecting, and contrasting textual input, as well as answering questions.
On June 11, 2018, OpenAI researchers published a paper introducing the first generative pre-trained transformer (GPT), a type of generative large language model pre-trained on an enormous and diverse text corpus, followed by discriminative fine-tuning for specific tasks. Previously, the best-performing neural NLP models commonly used supervised learning from large amounts of manually labeled data, which was expensive and time-consuming. The first GPT model was followed by GPT-2 in February 2019, a direct scale-up with 1.5 billion parameters trained on 8 million web pages. In February 2020, Microsoft introduced Turing Natural Language Generation (T-NLG), claimed as the largest language model at 17 billion parameters.
Training and capabilities
On May 28, 2020, an arXiv preprint by 31 engineers and researchers at OpenAI described GPT-3, a third-generation state-of-the-art language model. The team increased capacity by over two orders of magnitude from GPT-2, making GPT-3 the largest non-sparse language model at that time. Its greater accuracy is attributed to increased capacity and parameters, being ten times larger than Microsoft's Turing NLG. Lambdalabs estimated a hypothetical cost of around $4.6 million and 355 years to train GPT-3 on a single GPU in 2020, with lower actual training time using parallel GPUs.
Sixty percent of the weighted pre-training dataset came from a filtered version of Common Crawl consisting of 410 billion byte-pair-encoded tokens. Fuzzy deduplication used Apache Spark's MinHashLSH. Other sources included 19 billion tokens from WebText2 (22% of weighted total), 12 billion tokens from Books1 (8%), 55 billion tokens from Books2 (8%), and 3 billion tokens from Wikipedia (3%). GPT-3 was trained on hundreds of billions of words and could also code in CSS, JSX, and Python.
Since training data was all-encompassing, GPT-3 did not require further training for distinct language tasks. The training data contained occasional toxic language, and GPT-3 occasionally generated toxic language by mimicking it. A University of Washington study found GPT-3 produced toxic language at a level comparable to GPT-2 and CTRL. OpenAI implemented strategies to limit toxic output, resulting in less toxic language than GPT-1, though more generations and higher toxicity than CTRL Wiki, a model trained entirely on Wikipedia data.
Public access and evolution
On June 11, 2020, OpenAI announced users could request access to its GPT-3 API, a machine learning toolset, to explore the strengths and limits of the technology. The API had a general-purpose text-in, text-out interface that could complete almost any English language task. One early user described GPT-3 as eerily good at writing coherent text with simple prompts. In an initial experiment, 80 US subjects judged short articles as human-written or GPT-3-generated, correctly identifying only 52% of the time, slightly better than random guessing.
On November 18, 2021, OpenAI announced enough safeguards had been implemented to make API access unrestricted, providing developers with a content moderation tool. On January 27, 2022, OpenAI announced its newest GPT-3 models, collectively referred to as InstructGPT, became the default on the API, producing content better aligned with user intentions, following instructions better, generating fewer made-up facts, and producing somewhat less toxic content.
Impact and concerns
Because GPT-3 could generate news articles that human evaluators had difficulty distinguishing from human-written ones, it had potential to advance both beneficial and harmful applications of language models. In the May 28, 2020 paper, researchers described potential harmful effects including misinformation, spam, phishing, abuse of legal and governmental processes, and fraudulent activities. The model's broad capabilities spurred discussions about the societal implications of Generative AI and the need for responsible deployment.
Legacy
GPT-3 marked a significant milestone in the development of Artificial intelligence, demonstrating that scaling transformer models could yield dramatic improvements in language understanding and generation. Its architecture and training approach influenced subsequent models, including successors within OpenAI and efforts by other organizations such as Anthropic and Google DeepMind. The exclusive licensing to Microsoft highlighted the commercial value of large language models and shaped the competitive landscape of cloud-computing services, with Microsoft Azure integrating GPT-3 into its offerings. GPT-3 also catalyzed research into alignment, safety, and evaluation of large models, contributing to the broader field of Neural network research.