# T5 (Text-to-Text Transfer Transformer) T5 (Text-to-Text Transfer Transformer) is a series of encoder-decoder large language models developed by Google AI and introduced in 2019, which frame all natural language processing tasks as text-to-text problems. These models are pre-trained on a large corpus and can be fine-tuned for diverse applications. T5 (Text-to-Text Transfer Transformer) is a series of [large language models](https://www.wikiprompt.org/wiki/large-language-model) developed by Google AI and introduced in 2019. The central design principle of T5 is to unify all natural language processing tasks into a single text-to-text format: every task, whether translation, summarization, or question answering, is framed as taking input text and producing output text. This contrasts with earlier models that often used task-specific output heads or architectures. Like the original [Transformer](https://www.wikiprompt.org/wiki/transformer) model, T5 models employ an encoder-decoder structure, where the encoder processes the input text and the decoder generates the output text. The original T5 models are pre-trained on the Colossal Clean Crawled Corpus (C4), a dataset containing text and code scraped from the web. This pre-training enables the models to acquire general language understanding and generation capabilities. After pre-training, T5 models can be fine-tuned on specific downstream tasks, adapting their learned representations to perform well in a variety of applications, including chatbots, code generation, and robotics. ## Pre-training Tasks The pre-training of T5 models is performed on a diverse set of tasks, all formatted as input text followed by output text. One primary task is text restoration, where parts of the input are replaced with sentinel tokens (such as `` and ``), and the model must reconstruct the original text. For example, the input "Thank you me to your party week." would be expected to produce " for inviting ," where the sentinel tokens indicate missing spans. Another task is machine translation, where the model is given text in a source language and must generate a translation in a target language regulator. Additionally, T5 is trained on tasks such as grammatical acceptability judgments, where the model must determine whether a sentence is linguistically well-formed. ## Architecture The T5 series encompasses several models of varying sizes, all of which are encoder-decoder Transformers. The encoder processes the input text, and the decoder generates the output text, with both components sharing the same number of layers in all published configurations. The original 2019 paper reported five model variants: T5-small, T5-base, T5-large, T5-3B, and T5-11B. These models differ in the number of layers, attention heads, embedding dimensions, and feedforward network sizes, with parameter counts ranging from roughly 60 million for T5-small to 11 billion for T5-11B. Compared to the original Transformer architecture, T5 introduced several modifications: layer normalization is applied without an additive bias, layer normalization is placed outside the residual path, and relative positional embeddings are used. The models also employ a shared WordPiece tokenizer with a vocabulary size of 32,000, trained on a mixture of English, German, French, and Romanian text at a ratio of 10:1:1:1. ## Architecture Details The T5 series is built exclusively on the encoder-decoder [transformer](https://www.wikiprompt.org/wiki/transformer) architecture. In this design, the encoder takes the input text and produces a sequence of hidden representations, which the decoder then uses to autoregressively generate the output text token by token. Each encoder and decoder layer contains self-attention mechanisms, cross-attention (in the decoder), and feedforward networks. The models distinguish themselves from the original Transformer through the use of relative position embeddings and the removal of the bias term in layer normalization. For the larger variants (3B and 11B parameters), the dimensions of the model do not follow the common relationship d_model = d_kv * n_heads, where d_model is the embedding dimension, d_kv the key/value dimension, and n_heads the number of attention heads. This design choice allowed more flexibility in scaling the models. All T5 variants share the same tokenizer, which is shared between the encoder and decoder. ## Variants and Subsequent Models Following the original T5 release, several variants and derivative models were developed, often using non-standardized naming conventions. T5 1.1 introduced improved versions with GEGLU activations instead of the original ReLU, and renamed the 3B and 11B models to XL and XXL respectively. The XL and XXL shapes were also modified in this version. The LM-adapted T5 models, released in 2021, were initialized from the original T5 checkpoints and further trained on 100 billion additional tokens from C4. The Switch Transformer, also introduced in 2021, applied a mixture-of-experts approach to T5, replacing standard feedforward layers with expert-based ones. T0 models, released in 2021, were fine-tuned from LM-adapted T5 checkpoints to perform tasks in a zero-shot setting, using only task instructions. ByT5, also from 2021, operates on UTF-8 bytes instead of tokenized text and was trained on the multilingual C4 dataset. Flan-T5-XL, introduced in 2022, was created by instruction-tuning a T5-XL checkpoint on the FLAN dataset. UL2, also from 2022, uses the T5 architecture but scales up to 20 billion parameters and is trained with a mixture-of-denoisers objective on C4; it was trained accidentally on a [TPU](https://www.wikiprompt.org/wiki/google-cloud) cluster. T5X, released in 2022, is a reimplementation of the T5 codebase in JAX, while the original implementation used TensorFlow and MeshTF. ## Applications and Impact T5 models have been widely adopted for various natural language processing tasks due to their flexible text-to-text interface. They have been applied to machine translation, summarization, question answering, and grammatical error correction, among others. The models' ability to handle multiple tasks within a single framework simplified deployment and experimentation. T5 also influenced subsequent model families in [machine learning](https://www.wikiprompt.org/wiki/machine-learning), particularly in the development of instruction-tuned models and mixture-of-experts architectures. Its encoder-decoder design remains influential in the design of later models, although many subsequent large language models, such as those from [openai](https://www.wikiprompt.org/wiki/openai) and [anthropic](https://www.wikiprompt.org/wiki/anthropic), have favored decoder-only architectures. The original T5 paper reported strong performance on a range of benchmarks, including GLUE and SuperGLUE, and demonstrated that the text-to-text approach could achieve state-of-the-art results across diverse tasks with minimal task-specific engineering. The release of the T5 codebase and pre-trained checkpoints facilitated adoption in both academic and industrial settings. ## Impact and Legacy T5 helped popularize the concept of framing all NLP tasks as text-to-text problems, an idea that influenced later instruction-tuned models and multi-task learning frameworks. The release of T5 also contributed to the broader trend of scaling transformer-based language models, with variants reaching up to 20 billion parameters. The availability of the C4 dataset and the T5 codebase provided a foundational resource for subsequent research in the field. Although newer architectures have since emerged, T5's architecture and training paradigm continue to inform ongoing work in [machine learning](https://www.wikiprompt.org/wiki/machine-learning) and [deep learning](https://www.wikiprompt.org/wiki/deep-learning). --- Source: https://www.wikiprompt.org/wiki/t5 License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-08T06:10:48.303316+00:00