GPT-2 (Generative Pre-trained Transformer 2) is a Large language model developed by OpenAI, representing the second iteration in their foundational GPT series. It was pre-trained on a dataset of 8 million web pages and demonstrated significant advancements in text generation, translation, summarization, and question answering. The model was initially withheld from public release due to concerns about potential misuse, but was eventually fully released in November 2019.
GPT-2 is a direct scale-up of its predecessor, GPT-1, with a ten-fold increase in both parameter count and training dataset size. It employs a Transformer (architecture) architecture, which uses attention mechanisms to selectively focus on relevant parts of the input text, enabling greater parallelization and outperforming previous RNN/CNN/LSTM-based models. Its general-purpose learning ability allowed it to perform various tasks by accurately predicting the next item in a sequence, though it could become repetitive or nonsensical on longer passages. GPT-2 was superseded by later models like GPT-3 and GPT-4, which are no longer open source.
Training and Architecture
GPT-2's training leveraged the transformer architecture's capacity for massive parallelization, allowing it to be trained on larger corpora than previous natural language processing models. OpenAI initially considered CommonCrawl, a large web-crawled corpus, but rejected it due to unintelligible content. Instead, they developed a new corpus called WebText, which was created by scraping pages linked from Reddit posts with at least 3 karma prior to December 2017. The corpus was cleaned by parsing HTML to plain text, removing duplicates, and excluding Wikipedia pages to avoid overfitting.
The training cost of GPT-2 is estimated at approximately $43,000, based on a statement by Andrej Karpathy, using 32 TPU v3 chips for 168 hours at about $8 per chip per hour. Other sources cite a cost of about $256 per hour. In comparison, training BERT and XLNet cost $6,912 and $245,000 respectively. GPT-2's architecture is a deep Neural network with a generative pre-trained transformer, implementing attention instead of recurrence or convolution.
Release and Initial Restrictions
GPT-2 was announced on 14 February 2019. Unlike previous OpenAI models, the source code was not immediately released, citing risks of malicious use. Limited access was granted to selected press outlets, and OpenAI demonstrated a version fine-tuned to generate product reviews, which could be used to evade spam filters. Researchers like Jeremy Howard warned of the potential to fill the web with convincing prose, while the Allen Institute for Artificial Intelligence announced a detection tool for "neural fake news."
Opinions were divided on the threat. Some, like Anima Anandkumar, called the restrictions "malicious BS," while The Gradient published an open letter comparing the technology to the printing press and Photoshop. Despite the controversy, OpenAI released a partial version with 774 million parameters on August 20, 2019, and the full 1.5-billion-parameter model on November 5, 2019.
Model Variants and Replications
The GPT-2 series included four models of varying sizes, released in stages. The smallest model was made available in February 2019, with larger models following. The full 1.5-billion-parameter model was the final release. The model's methods were described in publications, allowing others to replicate it as free software. One such replication, OpenGPT-2, was released in August 2019 with a freely licensed version of WebText called OpenWebText, at a compute cost of approximately $50,000.
Impact and Legacy
GPT-2's release marked a significant milestone in Generative AI, demonstrating the potential of large-scale transformers. Its ability to generate coherent text was praised by outlets like The Verge and The Guardian, though it was noted that long passages could become repetitive. The model's initial withholding sparked debates about open access and safety in AI research, influencing future discussions on responsible AI development. GPT-2 was succeeded by GPT-3 and GPT-4, which continued to advance the field but moved away from open-source availability.