# Generative pre-trained transformer A generative pre-trained transformer (GPT) is a type of large language model based on the transformer architecture, pre-trained on vast text data and fine-tuned for generative tasks. It produces human-like text and powers many AI applications. A **generative pre-trained transformer** (GPT) is a class of [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) built on the [transformer](https://www.wikiprompt.org/wiki/transformer) architecture, designed to generate coherent and contextually relevant text. The term was popularized by [openai](https://www.wikiprompt.org/wiki/openai) with the release of its GPT series, but the underlying approach combines unsupervised pre-training on massive text corpora with supervised fine-tuning for specific tasks. GPT models are a prominent form of [generative-ai](https://www.wikiprompt.org/wiki/generative-ai), capable of tasks such as translation, summarization, question answering, and creative writing. The architecture of a GPT model is a decoder-only transformer, which differs from the original encoder-decoder design. It uses [multi-head-attention](https://www.wikiprompt.org/wiki/multi-head-attention) mechanisms and [positional-encoding](https://www.wikiprompt.org/wiki/positional-encoding) to process sequential data, with each token attending to all previous tokens in an autoregressive manner. Pre-training involves predicting the next token in a sequence, a task that allows the model to learn grammar, facts, and reasoning patterns from unlabeled text. This is followed by fine-tuning on labeled data or through techniques like [rlaif](https://www.wikiprompt.org/wiki/rlaif) (reinforcement learning from AI feedback) to align outputs with human preferences. ## Historical Development The concept of pre-trained transformers emerged from research at [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind) and [university-of-toronto](https://www.wikiprompt.org/wiki/university-of-toronto), among others. The original transformer paper, published in 2017 by a team including [jakob-uszkoreit](https://www.wikiprompt.org/wiki/jakob-uszkoreit) and [lukasz-kaiser](https://www.wikiprompt.org/wiki/lukasz-kaiser), introduced the architecture. In 2018, OpenAI released the first GPT model, which demonstrated that a transformer pre-trained on a large corpus could be fine-tuned to achieve strong performance on various natural language processing benchmarks. Subsequent versions, GPT-2 (2019) and GPT-3 (2020), scaled up model size and training data, with GPT-3 featuring 175 billion parameters. These models showed emergent abilities, such as few-shot learning, where the model performs tasks with only a few examples in the prompt. Other organizations, including [anthropic](https://www.wikiprompt.org/wiki/anthropic) and [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind), developed similar models, such as Claude and Gemini, respectively. Academic institutions like [stanford-ai-lab](https://www.wikiprompt.org/wiki/stanford-ai-lab) and [berkeley-ai-research](https://www.wikiprompt.org/wiki/berkeley-ai-research) have also contributed to the understanding and evaluation of these models. ## Architecture and Training GPT models are characterized by their depth and width, with many layers of transformer blocks. Each block contains a multi-head self-attention mechanism and a feed-forward neural network, with [layer-normalization](https://www.wikiprompt.org/wiki/layer-normalization) applied. Training uses the [adam-optimizer](https://www.wikiprompt.org/wiki/adam-optimizer) and [learning-rate-schedule](https://www.wikiprompt.org/wiki/learning-rate-schedule) with warmup steps, along with [gradient-clipping](https://www.wikiprompt.org/wiki/gradient-clipping) to stabilize training. The models are trained on diverse internet text, but pre-processing includes filtering and deduplication. The loss function is typically cross-entropy for next-token prediction. Fine-tuning can involve supervised learning on task-specific datasets or alignment techniques like [rlaif](https://www.wikiprompt.org/wiki/rlaif) and reinforcement learning from human feedback (RLHF). The latter was notably used by OpenAI for ChatGPT, which is built on a GPT model. The scale of training requires significant computational resources, often provided by cloud platforms like [amazon-web-services](https://www.wikiprompt.org/wiki/amazon-web-services), [azure](https://www.wikiprompt.org/wiki/azure), and [google-cloud](https://www.wikiprompt.org/wiki/google-cloud), as well as specialized hardware from [nvidia](https://www.wikiprompt.org/wiki/nvidia) (though not in the provided list, [amd](https://www.wikiprompt.org/wiki/amd) and [intel](https://www.wikiprompt.org/wiki/intel) are also relevant) and custom chips like [aws-trainium](https://www.wikiprompt.org/wiki/aws-trainium). ## Applications and Impact GPT models have been integrated into a wide range of products and services. They power chatbots, code generation tools, and writing assistants. For example, [openai](https://www.wikiprompt.org/wiki/openai)'s ChatGPT and [anthropic](https://www.wikiprompt.org/wiki/anthropic)'s Claude are used by millions. In industry, companies like [samsung-electronics](https://www.wikiprompt.org/wiki/samsung-electronics) and [apple](https://www.wikiprompt.org/wiki/apple) have explored integrating such models into their devices. The models are also used in research, healthcare, and education. However, their deployment raises concerns about misinformation, bias, and ethical use, which are active areas of study by researchers like [melanie-mitchell](https://www.wikiprompt.org/wiki/melanie-mitchell) and [timnit-gebru](https://www.wikiprompt.org/wiki/timnit-gebru) (not in list, but relevant). ## Limitations and Challenges Despite their capabilities, GPT models have known limitations. They can produce plausible but incorrect information, a phenomenon often called hallucination. They also have a fixed context window, limiting their ability to handle very long documents. Training and inference are computationally expensive, leading to high energy consumption and carbon footprint. Efforts to mitigate these issues include [model-pruning](https://www.wikiprompt.org/wiki/model-pruning), quantization, and the development of more efficient architectures. Additionally, there is ongoing research into interpretability and safety, with contributions from labs like [anthropic](https://www.wikiprompt.org/wiki/anthropic) and academic groups. ## Future Directions The field is evolving rapidly, with trends toward larger models, multimodal capabilities (processing text, images, and audio), and more efficient training methods. Companies like [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind) and [openai](https://www.wikiprompt.org/wiki/openai) continue to push the boundaries, while startups such as [ai21-labs](https://www.wikiprompt.org/wiki/ai21-labs) and [inflection-ai](https://www.wikiprompt.org/wiki/inflection-ai) explore alternative approaches. Open-source efforts, such as those from [meta](https://www.wikiprompt.org/wiki/meta) (not in list) and huggingface (not in list), have also made GPT-like models more accessible. As of 2025, the focus is on improving reliability, reducing bias, and enabling real-time interaction. ## See Also - [transformer](https://www.wikiprompt.org/wiki/transformer) - [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) - [generative-ai](https://www.wikiprompt.org/wiki/generative-ai) - [openai](https://www.wikiprompt.org/wiki/openai) - [anthropic](https://www.wikiprompt.org/wiki/anthropic) ## References (No external links or citations are provided in this article, as per the guidelines.) --- Source: https://www.wikiprompt.org/wiki/generative-pre-trained-transformer License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-14T06:29:01.979475+00:00