T5 (Text-to-Text Transfer Transformer) is a series of large language models developed by Google AI and introduced in 2019. Like the original Transformer (architecture) model, T5 models are Encoder-Decoder Architecture transformers, where the encoder processes the input text and the decoder generates the output text. T5 models are usually pretrained on a massive dataset of text and code, after which they can perform text-based tasks similar to their pretraining objectives, and can be fine-tuned for other tasks. They have been applied in chatbots, machine translation, text summarization, code generation, and robotics.
Training
The original T5 models were pretrained on the Colossal Clean Crawled Corpus (C4), containing text and code scraped from the internet. This pretraining enables the models to learn general language understanding and generation. T5 models can then be fine-tuned on downstream tasks. The pretraining used a text-to-text format, where every task is framed as <input text> -> <output text>. Examples include restoring corrupted text (e.g., filling in blanks marked by sentinels), translation (e.g., English to German), and grammatical acceptability judgment (e.g., CoLA sentence classification).
Architecture
The T5 series includes models of varying sizes, all encoder-decoder transformers. The original paper reported five models: T5-small (60M parameters), T5-base (220M), T5-large (770M), T5-3B (3B), and T5-11B (11B). Each model has the same number of layers in the encoder and decoder; for instance, T5-small has 6 layers in each. Key architectural dimensions include the number of layers, attention heads, embedding dimension, feedforward dimension, and key/value dimension. Unlike typical transformers, the 3B and 11B models do not satisfy the relation d_model = d_kv * n_head. Compared to the original transformer, T5 uses layer normalization without additive bias, places layer normalization outside the residual path, and uses relative positional embeddings. A WordPiece tokenizer with a vocabulary size of 32,000 was used, shared across input and output, trained on a mixture of English, German, French, and Romanian data from C4 at a 10:1:1:1 ratio.
Variants
Several subsequent models used the T5 architecture. Notable variants include:
- T5 1.1 (2019-2020): Improved versions with GEGLU activation instead of ReLU; the 3B and 11B were renamed XL and XXL with modified shapes.
- LM-adapted T5 (2021): Started from T5 checkpoints and trained further on 100B additional tokens from C4.
- Switch Transformer (2021): A mixture-of-experts variant replacing feedforward layers with expert layers.
- T0 (2021): Started from LM-adapted T5 and trained to perform tasks zero-shot based on instructions.
- ByT5 (2021): Byte-level version operating on UTF-8 bytes without tokenizers, trained on multilingual C4.
- Flan-T5-XL (2022): Instruction-tuned on the FLAN dataset starting from T5 XL.
- T5X (2022): A JAX-based reimplementation of the original T5 codebase (not a model).
- UL2 20B (2022): Scaled to 20B parameters, trained with a mixture of denoisers objective on C4; notably trained accidentally for a month on a TPU cluster.
- Flan-UL2 20B (2022): UL2 20B instruction-finetuned on FLAN.
- Pile-T5 (2024): Same architecture but uses the Llama tokenizer and trained on The Pile, available in base, large, XL, and XXL sizes.
Applications
T5's encoder-decoder structure allows instruction following: the encoder encodes the instruction, and the decoder autoregressively generates the reply. The T5 encoder can also serve as a text encoder, similar to BERT, producing vector representations for downstream tasks. For example, Google Imagen uses T5-XXL as its text encoder, and the AuraFlow diffusion model uses Pile-T5-XL. These applications demonstrate T5's versatility in generative AI and beyond.