Generative Pre-trained Transformer 3 (GPT-3) is a large language model released by OpenAI in 2020 as part of the company's GPT series of models. It is a decoder-only Transformer (architecture) model of a deep neural network, using an attention mechanism to focus selectively on relevant input segments. With 175 billion parameters, each stored in 16-bit precision, GPT-3 requires 350GB of storage and supports a context window of 2,048 tokens. It demonstrated strong zero-shot and few-shot learning across many tasks, and its release via an API was announced on June 11, 2020.
Background
According to The Economist, improved algorithms, more powerful computers, and a rising digitized corpus fueled a revolution in Machine learning. New techniques in the 2010s led to rapid improvements in language manipulation, relying on neural architectures loosely based on the brain. The Transformer (architecture) architecture, introduced in 2017, became foundational for Natural language processing systems. In June 2018, OpenAI published the first generative pre-trained transformer (GPT-1), a model trained on a large text corpus and then fine-tuned for specific tasks. GPT-2 followed in February 2019, scaling parameters and dataset size tenfold, reaching 1.5 billion parameters trained on 8 million web pages. In February 2020, Microsoft introduced Turing NLG, a 17-billion-parameter model, then considered the largest.
Training and capabilities
On May 28, 2020, an arXiv preprint by 31 OpenAI engineers and researchers described GPT-3's development, increasing model capacity by over two orders of magnitude from GPT-2, making it the largest non-sparse language model of that period. Its accuracy stemmed from increased capacity and parameters, being ten times larger than Microsoft's Turing NLG. Lambdalabs estimated training cost at $4.6 million and 355 years on a single GPU in 2020. Pre-training data included 410 billion byte-pair encoded tokens from filtered Common Crawl (60% of weighted data), plus WebText2, Books1, Books2, and Wikipedia (3%). GPT-3 also could code in Python, JSX, and CSS.
GPT-3 lacked task-specific fine-tuning, instead handling broad instructions. However, training data contained toxic language that the model could sometimes reproduce. A University of Washington study found GPT-3's toxicity comparable to GPT-2 and CTRL. OpenAI implemented mitigation strategies, reducing toxic outputs compared to GPT-1.
API access and evaluation
On June 11, 2020, OpenAI released a general-purpose API with a text-in text-out interface, enabling any English language task, rather than single-use cases. Early testers described GPT-3 as eerily good at coherent text generation. In an experiment with 80 US participants, only a 52% correct rate distinguished GPT-3 articles from human-authored ones, a slight edge over random guessing. By November 18, 2021, OpenAI broadened access to its API with a content moderation tool. On January 27, 2022, InstructGPT became the default API model, improving instruction following and reducing fabricated facts.
Licensing and impact
On September 22, 2020, Microsoft announced an exclusive license for GPT-3's underlying model. Others could use the public API but only Microsoft could access the code and weights. GPT-3 demonstrated potential for generating news articles hard to distinguish from human writing, prompting concerns about misinformation and fraud. The researchers described potential harmful effects including spam and legal abuse.