GPT-3, short for Generative Pre-trained Transformer 3, is a large language model released by OpenAI in 2020 as part of the company's GPT series. It is a decoder-only transformer model that uses an attention mechanism to focus on relevant parts of input text, superseding recurrence and convolution-based architectures. With 175 billion parameters, GPT-3 was the largest non-sparse language model at its release, requiring 350GB of storage due to 16-bit precision per parameter. It has a context window of 2,048 tokens and demonstrated strong zero-shot and few-shot learning abilities across many tasks.
The release of GPT-3 marked a milestone in generative AI, showcasing the potential of scaling up transformer-based neural networks. Its capabilities influenced subsequent developments in large language models and sparked discussions about the benefits and risks of such systems.
Background
The development of GPT-3 was rooted in advances in machine learning during the 2010s, driven by improved algorithms, more powerful computers, and increased digitized data. The transformer architecture, introduced in 2017, became a key foundation for natural language processing. OpenAI's first generative pre-trained transformer (GPT-1) was introduced in June 2018, followed by GPT-2 in February 2019, which scaled up parameters and dataset size by a factor of 10, reaching 1.5 billion parameters. In February 2020, Microsoft introduced Turing Natural Language Generation (T-NLG) with 17 billion parameters, which was then the largest language model.
Training and Capabilities
On May 28, 2020, an arXiv preprint by 31 OpenAI researchers described GPT-3, which increased capacity by over two orders of magnitude compared to GPT-2. The training dataset comprised 60% filtered Common Crawl (410 billion tokens), 22% WebText2, 8% Books1, 8% Books2, and 3% Wikipedia. GPT-3 was trained on hundreds of billions of words and could also code in languages like CSS, JSX, and Python. Lambda Labs estimated a training cost of around $4.6 million and 355 years on a single GPU, though parallelization reduced actual time.
GPT-3's all-encompassing training data meant it did not require fine-tuning for distinct tasks, but it could generate toxic language due to biases in the data. A University of Washington study found its toxicity comparable to GPT-2 and CTRL. OpenAI implemented safeguards, and by November 2021, access to the API became unrestricted. In January 2022, InstructGPT models became the default, better aligned with user intentions and producing less toxic content.
Impact and Reception
GPT-3's ability to generate coherent text that humans could not easily distinguish from human-written content raised both opportunities and concerns. In an experiment, 80 US subjects judged short articles correctly only 52% of the time, near random. The model's potential for misinformation, spam, and phishing was noted in the original paper. GPT-3 also influenced the broader field of Artificial intelligence, leading to further research in Large language models and Generative AI.
Commercialization and Licensing
On September 22, 2020, Microsoft announced an exclusive license to GPT-3, granting access to the underlying model while others could still use the public API. This deal highlighted the commercial value of advanced AI models and set a precedent for future partnerships in the industry. The licensing arrangement was part of Microsoft's broader investment in OpenAI and its integration into products like Azure.
Legacy
GPT-3's release accelerated progress in natural language processing and inspired subsequent models from various organizations, including Anthropic and Google DeepMind. Its architecture and training approach became benchmarks for later developments, such as InstructGPT and other successors. The model also spurred discussions on AI ethics, safety, and the need for content moderation, shaping the trajectory of Machine learning research and deployment.