# GPT-1 Release

GPT-1, introduced by OpenAI in June 2018, was the first generative pre-trained transformer, a large language model using transformer architecture and pre-training on unlabeled text, marking a breakthrough in natural language processing.

GPT-1, released by OpenAI on June 11, 2018, was the first generative pre-trained transformer (GPT), a type of [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) based on the [transformer](https://www.wikiprompt.org/wiki/transformer) architecture. It introduced a semi-supervised approach that combined generative pre-training on unlabeled text with discriminative fine-tuning for specific language tasks, significantly advancing the field of [natural-language-processing](https://www.wikiprompt.org/wiki/natural-language-processing) and [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence).

## Background

During the 2010s, an [ai-boom](https://www.wikiprompt.org/wiki/ai-boom) driven by improved [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) algorithms, more powerful computers, and increased digitized data enabled rapid progress in AI. Generative pre-training, a form of [self-supervised-learning](https://www.wikiprompt.org/wiki/self-supervised-learning), had long been established in machine learning: a model is first trained on a large unlabeled dataset to learn to generate data, then adapted to a specific task with labeled data. The transformer architecture, introduced by Google researchers in the 2017 paper "Attention Is All You Need", solved performance issues of older recurrent neural networks by using an attention mechanism to process entire sequences at once, allowing much larger and more sophisticated models.

## History

On June 11, 2018, OpenAI published the paper "Improving Language Understanding by Generative Pre-Training", introducing GPT-1. It was a transformer-based model using only the decoder part, pre-trained on BookCorpus, a diverse text corpus, to predict the next token, followed by discriminative fine-tuning for specific tasks. This semi-supervised approach was a breakthrough, as previous best-performing neural models relied on expensive supervised learning from manually labeled data. On February 14, 2019, OpenAI released GPT-2, a direct scale-up with 1.5 billion parameters and a 40-gigabyte dataset of 8 million web pages, initially staged due to misuse concerns. On February 10, 2020, Microsoft introduced Turing Natural Language Generation with 17 billion parameters, claiming it was the largest language model at the time. On May 28, 2020, OpenAI released GPT-3 with 175 billion parameters, advancing few-shot and zero-shot learning. Following GPT-3, OpenAI used reinforcement learning from human feedback to create InstructGPT, leading to ChatGPT, launched on November 30, 2022, based on GPT-3.5 and later GPT-4 (released March 14, 2023). The popularity of ChatGPT spurred competitors like Google's PaLM and Gemini, Meta AI's Llama, and others.

## Foundation Models

A foundation model is an AI model trained on broad data at scale, adaptable to many downstream tasks. Foundational GPTs can employ modalities beyond text, such as images and audio. Some generative transformer-based models are used for text-to-image technologies, including diffusion and parallel decoding, serving as visual foundation models for developing image-processing systems.

## Efficient Transformer Architectures

The computational and memory requirements of transformer models increase significantly with size and input length, as standard self-attention has quadratic complexity. Researchers proposed efficiency improvements like sparse attention mechanisms and memory-efficient architectures to reduce costs and support longer context windows. Models such as BigBird, Reformer, and FlashAttention demonstrate structured attention patterns or optimized computation, helping large language models process long sequences efficiently.

## Impact

GPT-1's introduction established the foundation for subsequent GPT models and influenced the broader development of generative AI. Its semi-supervised approach reduced reliance on labeled data, enabling training on vast unlabeled corpora. The success of GPT-1 and its successors led to widespread adoption of transformer-based models in various applications, from chatbots to content generation, and spurred research into model scaling, alignment, and efficiency.

---
Source: https://www.wikiprompt.org/wiki/gpt-1-release
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-12T16:21:25.862143+00:00