GPT-2 (Generative Pre-trained Transformer 2) is a Large language model developed by OpenAI, announced on February 14, 2019. It is the second model in OpenAI's foundational GPT series, succeeding GPT-1 and preceding GPT-3. GPT-2 was pre-trained on a dataset of 8 million web pages, and its ability to generate coherent, contextually relevant text was seen as a significant advancement in Generative AI. The model was initially withheld during its partial release, with the full 1.5-billion-parameter version becoming available only on November 5, 2019. Because of its ability to translate languages, answer questions, and summarize passages, researchers and commentators highlighted its potential for both beneficial uses and misuse, such as spam generation or the creation of fake news.
Architecture
Like its predecessor GPT-1 and its successors, GPT-2 uses a Deep learning Transformer (architecture) architecture, which is a type of Neural network that implements a mechanism known as attention. This attention mechanism has been shown to enhance performance for tasks involving Sequence-to-Sequence (Seq2Seq) prediction. Unlike older models based on recurrence or convolution, the transformer's attention mechanism allows the model to process input text in parallel, which can greatly increase training and inference speed. The model’s architecture was described as a top-scale of GPT-1, with a factor of 10 increase in its parameter count (1.5 billion) and training data size. This parallelization, achieved on specialized hardware, permitted the training on a corpus larger than was previously feasible, which contributed to its capacity to handle general Machine learning tasks.
Training
GPT-2 was trained on WebText, a corpus of over 100 gigabytes of text scraped from outbound links on Reddit that received at least 3 karma prior to December 2017. Reddit was used as a proxy for human-curated content, filtering out low-quality pages. HTML was parsed into plain, duplicate links were eliminated, and Wikipedia pages were removed from the training set to prevent overfitting, as those articles often appear in many other datasets. CommonCrawl, despite its size, was rejected because of the great amount of unintelligible content it contained. The infrastructure for training GPT-2 used 32 TPU v3 chips for 168 hours, costing roughly $43,000 in computation, as stated by Andrej Karpathy, a researcher who was associated with the project at the time. This was relatively low for a model of its size, due to the efficiency of the transformer. The result was a model that could produce text on a level sometimes indistinguishable from human outputs, though it could become repetitive or nonsensical in longer passages, a limitation that future large language models aimed to improve.
Partial Release
OpenAI's initial decision to not immediately release the full model was based on the concern that the model could be used for malicious purposes, such as to generate spam or disinformation. The company released a partial version with 774 million parameters on August 20, 2019, about half the size of the original (also known as the 774M release). This partial release was made to allow researchers to understand the model's behavior while mitigating misuse, that OpenAI’s approach drew backlash from some researchers, including Anima Anandkumar, who claimed the threat was exaggerated. Others, however, argued that the model could be used to fill social media with indistinguishable prose, and the Allen Institute for Artificial Intelligence built a detector for neural fake news in response.
In a February 2019 article in The Verge, James Vincent wrote that GPT-2’s outputs were often clearly non-human, but still “one of the most exciting examples” of language generation AI. The article demonstrated that, with a fake headline, the model could write the rest of the article with fake quotes and statistics, and even write fiction given appropriate prompts. Kelsey Piper of Vox wrote that “one of the coolest AI systems I’ve ever seen may also be the one that will kick me out;” however, she described its output through a neutral lens. The Guardian described its text as “plausible newspaper prose.”
Despite the partial release, OpenAI retained the full model’s code and the corpus. This led to various third-party outputs. In August 2019, the free-software replication OpenGPT-2 was produced using OpenWebText, which cost about $50,000 for the compute. This illustrates the open research and development that followed the model's initial release.
Ethical and Societal Impact
The decision to delay the full release of GPT-2 initiated a public debate about the risks and benefits of Artificial intelligence technology. Some researchers dismissed the perceived dangers, noting that the threats could be mitigated, while others warned of a coming flood of synthetic text that could drown out authentic human speech. There was even a partial release of a fine-tuned model to generate positive or negative product reviews, a demonstration of possible spam potential. In response to the hype and the open-to-open policy, a group of researchers published an open letter requesting the complete release, declining that language models are as threatening as they were dressed to be, citing technologies like Photoshop and the printing press, which have been misused but also integrated into everyday life.
The long-term effect of the withheld release was that it increased the public awareness of Large language model and their capabilities. GPT-2 was eventually used as a focus on Artificial intelligence safety, influencing subsequent open-source models. It was subsequently superseded by GPT-3 and GPT-4; as of 2024, GPT-3 and GPT-4, which are no longer open source, have become more prominent. The episode served as a important point for considerations about how to balance transparency and precaution when developing powerful AI systems.
Legacy
GPT-2 represented a significant development in the field of deep, becoming a benchmark for subsequent language models. Its ability to serve as a general-purpose learner, relying on the prediction of the next item in a sequence, and not on task-specific data, was a fundamental step for later transformers. The model's parameter count and training corpus size established a path for scaling Neural network models. Consequently, the computational demands of Generative AI have increased, and GPT-2 is considered a milestone in the history of artificial intelligence. Its training principles have continued to influence the development and architecture of large language models by a number of fields, and the model remains a well-studied reference point for understanding emergence and machine learning.