# OpenAI GPT-3 Release

GPT-3 is a large language model released by OpenAI in 2020, with 175 billion parameters and strong few-shot learning, enabling text generation and task completion from minimal prompts.

Generative Pre-trained Transformer 3 (GPT-3) is a large language model released by OpenAI in 2020 as part of the company's GPT series. It is a decoder-only transformer model of deep neural network, superseding recurrence and convolution-based architectures with an attention mechanism that allows selective focus on relevant input text. GPT-3 has 175 billion parameters, each with 16-bit precision, requiring 350GB of storage, and a context window of 2,048 tokens, demonstrating strong zero-shot and few-shot learning abilities on many tasks.

## Background

The development of GPT-3 was part of a broader revolution in machine learning, driven by improved algorithms, more powerful computers, and increased digitized data. The transformer architecture, introduced in 2017, became a key foundation for natural language processing (NLP) systems. OpenAI researchers published a paper on June 11, 2018, introducing the first generative pre-trained transformer (GPT-1), a type of generative large language model pre-trained on a large text corpus and fine-tuned for specific tasks. GPT-2, released in February 2019, scaled up GPT-1 with 1.5 billion parameters and was trained on 8 million web pages. In February 2020, Microsoft introduced Turing Natural Language Generation (T-NLG) with 17 billion parameters, which was then the largest language model.

## Training and capabilities

On May 28, 2020, an arXiv preprint by 31 OpenAI engineers and researchers described GPT-3, a third-generation state-of-the-art language model. GPT-3's capacity was increased by over two orders of magnitude from GPT-2, making it the largest non-sparse language model at that time. Its accuracy is attributed to its increased capacity and number of parameters, being ten times larger than Microsoft's Turing NLG. Lambdalabs estimated a hypothetical training cost of around $4.6 million and 355 years on a single GPU, with lower actual time using parallel GPUs.

The pre-training dataset comprised 60% filtered Common Crawl (410 billion byte-pair-encoded tokens), 22% WebText2 (19 billion tokens), 8% Books1 (12 billion tokens), 8% Books2 (55 billion tokens), and 3% Wikipedia (3 billion tokens). Fuzzy deduplication used Apache Spark's MinHashLSH. GPT-3 was trained on hundreds of billions of words and could code in CSS, JSX, and Python.

Since training data was all-encompassing, GPT-3 required no further training for distinct tasks. However, it occasionally generated toxic language due to mimicking its training data. A University of Washington study found GPT-3's toxicity comparable to GPT-2 and CTRL. OpenAI implemented strategies to limit toxic output, resulting in less toxicity than GPT-1 but more than CTRL Wiki.

On June 11, 2020, OpenAI announced access to its GPT-3 API, a machine learning toolset with a text-in, text-out interface for any English language task. Early users found it "eerily good" at writing coherent text. In an experiment, 80 US subjects judged short articles as human or GPT-3-written, correctly identifying only 52% of the time, slightly better than random. On November 18, 2021, OpenAI made API access unrestricted with content moderation tools. On January 27, 2022, InstructGPT models became the default, producing better-aligned content with fewer made-up facts and less toxicity.

GPT-3's ability to generate news articles indistinguishable from human-written ones has potential for both beneficial and harmful applications, including misinformation, spam, phishing, and abuse of legal processes, as detailed in the May 28, 2020 paper.

## Commercial licensing

On September 22, 2020, Microsoft announced it had licensed GPT-3 exclusively. While others could still use the public API, only Microsoft had access to the underlying model. This exclusive licensing was part of Microsoft's broader partnership with OpenAI, which included investments and integration into Azure services.

## Impact and legacy

GPT-3 significantly influenced the field of artificial intelligence, demonstrating the power of scaling transformer models. Its few-shot learning capabilities reduced the need for task-specific fine-tuning, inspiring subsequent models like [InstructGPT](https://www.wikiprompt.org/wiki/instructgpt) and later [GPT-4](https://www.wikiprompt.org/wiki/gpt-4). The release also sparked discussions about the ethical implications of large language models, including bias, misinformation, and access control.

GPT-3's architecture and training approach became a benchmark for subsequent large language models, including those from [Anthropic](https://www.wikiprompt.org/wiki/anthropic) and [Google DeepMind](https://www.wikiprompt.org/wiki/google-deepmind). Its commercial success through the API and Microsoft licensing established a business model for AI research, influencing the broader [generative AI](https://www.wikiprompt.org/wiki/generative-ai) landscape.

## Technical details

GPT-3 uses a transformer architecture with multi-head attention and positional encoding. It employs a decoder-only design, unlike encoder-decoder models, and uses techniques like layer normalization and residual networks. Training involved gradient descent with optimizers like Adam and learning rate schedules. The model's parameters are stored with 16-bit precision, reducing memory requirements.

GPT-3's context window of 2,048 tokens limits the length of input it can process, but it can generate coherent text over multiple paragraphs. Its few-shot learning is achieved through prompting, where examples are provided in the input, without updating model weights. This capability was a key innovation, enabling broad applicability without fine-tuning.

## Reception and concerns

GPT-3 received widespread attention for its impressive text generation, but also raised concerns about potential misuse. Researchers highlighted risks such as generating fake news, spam, and phishing content. OpenAI's initial restricted access and later content moderation tools aimed to mitigate these risks. The model's tendency to produce toxic language, despite improvements, remained a concern, leading to ongoing research in AI safety and alignment.

The exclusive licensing to Microsoft was controversial, as it limited access to the underlying model for other researchers and companies. However, the public API allowed broader experimentation, contributing to the rapid advancement of language model applications.

GPT-3's release marked a milestone in [artificial intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence), demonstrating the potential of large-scale transformer models and shaping the future of [machine learning](https://www.wikiprompt.org/wiki/machine-learning) and [deep learning](https://www.wikiprompt.org/wiki/deep-learning). Its influence is seen in subsequent models and the growing field of [generative AI](https://www.wikiprompt.org/wiki/generative-ai).

---
Source: https://www.wikiprompt.org/wiki/openai-gpt-3-2020
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-13T03:50:53.344256+00:00