# GPT-3 Launch (2020)

GPT-3 is a large language model released by OpenAI in 2020, with 175 billion parameters, known for its strong few-shot learning and text generation capabilities. It marked a significant milestone in generative AI, leading to widespread API access and commercial licensing.

Generative Pre-trained Transformer 3 (GPT-3) is a [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) released by [openai](https://www.wikiprompt.org/wiki/openai) in 2020 as part of the company's GPT series. It is a decoder-only [transformer](https://www.wikiprompt.org/wiki/transformer) model of [deep-learning](https://www.wikiprompt.org/wiki/deep-learning), superseding recurrence and convolution-based architectures with an attention mechanism that allows the model to focus selectively on relevant input text. GPT-3 has 175 billion parameters, each with 16-bit precision, requiring 350GB of storage. It has a context window of 2,048 tokens and demonstrated strong zero-shot and few-shot learning abilities on many tasks.

## Background

According to The Economist, improved algorithms, more powerful computers, and increased digitized material fueled a revolution in [machine-learning](https://www.wikiprompt.org/wiki/machine-learning). New techniques in the 2010s led to rapid improvements in language manipulation tasks. Software models are trained using thousands or millions of examples in a structure loosely based on the neural architecture of the brain. A key architecture is the transformer, introduced in 2017, which became central to [natural-language-processing](https://www.wikiprompt.org/wiki/natural-language-processing) systems.

On June 11, 2018, OpenAI researchers published a paper introducing the first generative pre-trained transformer (GPT), a type of generative large language model pre-trained on a diverse text corpus, followed by discriminative fine-tuning. GPT-1 was followed by GPT-2 in February 2019, which scaled up parameters and dataset size by a factor of 10, reaching 1.5 billion parameters trained on 8 million web pages. In February 2020, Microsoft introduced Turing Natural Language Generation (T-NLG) with 17 billion parameters, then the largest published language model.

## Training and capabilities

On May 28, 2020, an arXiv preprint by 31 OpenAI engineers and researchers described GPT-3, a third-generation state-of-the-art language model. The team increased capacity by over two orders of magnitude from GPT-2, making GPT-3 the largest non-sparse language model at that time. Its accuracy is attributed to increased capacity and parameters, being ten times larger than Microsoft's Turing NLG. Lambdalabs estimated a hypothetical cost of around $4.6 million and 355 years to train on a single GPU in 2020, with lower actual time using parallel GPUs.

Sixty percent of the weighted pre-training dataset came from a filtered version of Common Crawl with 410 billion byte-pair-encoded tokens, using fuzzy deduplication with Apache Spark's MinHashLSH. Other sources included WebText2 (19 billion tokens, 22%), Books1 (12 billion tokens, 8%), Books2 (55 billion tokens, 8%), and Wikipedia (3 billion tokens, 3%). GPT-3 was trained on hundreds of billions of words and could code in CSS, JSX, and Python.

Since training data was all-encompassing, GPT-3 does not require further training for distinct tasks. However, it occasionally generates toxic language mimicking its training data. A University of Washington study found GPT-3 produced toxic language comparable to GPT-2 and CTRL. OpenAI implemented strategies to limit toxicity, resulting in less toxic output than GPT-1 but more than CTRL Wiki.

## API and access

On June 11, 2020, OpenAI announced users could request access to its GPT-3 API, a machine learning toolset with a general-purpose text-in, text-out interface. One early user described it as eerily good at writing coherent text with simple prompts. In an experiment, 80 US subjects judged short articles as human or GPT-3 written, correct only 52% of the time, slightly better than random.

On September 22, 2020, Microsoft announced it had licensed GPT-3 exclusively. Others could still receive output from the public API, but only Microsoft had access to the underlying model. On November 18, 2021, OpenAI announced unrestricted API access with a content moderation tool. On January 27, 2022, OpenAI made InstructGPT the default model, which followed instructions better and generated fewer made-up facts.

## Impact and concerns

GPT-3 can generate news articles that human evaluators struggle to distinguish from human-written ones, with potential for both beneficial and harmful applications. The May 2020 paper detailed potential harms including misinformation, spam, phishing, and abuse of legal and governmental processes. The model's release spurred further development in [generative-ai](https://www.wikiprompt.org/wiki/generative-ai) and influenced subsequent models from [anthropic](https://www.wikiprompt.org/wiki/anthropic) and [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind). Its architecture and training approach became foundational for later large language models, and its API model influenced commercial AI services on [azure](https://www.wikiprompt.org/wiki/azure) and [amazon-web-services](https://www.wikiprompt.org/wiki/amazon-web-services).

## Legacy

GPT-3 marked a turning point in public awareness of large language models, demonstrating that scaling up transformer models could yield emergent capabilities. Its 175-billion-parameter scale set a benchmark, though later models would exceed it. The exclusive licensing to Microsoft highlighted the commercial value of such models. GPT-3's few-shot learning abilities reduced the need for task-specific fine-tuning, influencing research in [rlaif](https://www.wikiprompt.org/wiki/rlaif) and instruction following. Its release also raised discussions about AI safety, bias, and the environmental cost of training large models.

---
Source: https://www.wikiprompt.org/wiki/gpt-3-launch-2020
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-09T02:02:13.504716+00:00