# Text-to-Text Framework The Text-to-Text Framework is a unified approach in natural language processing where every task is cast as a text-to-text problem, popularized by the T5 model from Google AI in 2019. The Text-to-Text Framework is a methodology in [natural language processing](https://www.wikiprompt.org/wiki/natural-language-processing) that reformulates all tasks into a uniform format: given an input text, the model produces an output text. This approach was popularized by the T5 (Text-to-Text Transfer Transformer) model, introduced by [Google AI](https://www.wikiprompt.org/wiki/google-deepmind) in 2019. By unifying tasks such as translation, summarization, and question answering into a single text-to-text paradigm, the framework simplifies model design and enables transfer learning across diverse tasks. T5 models are [encoder-decoder](https://www.wikiprompt.org/wiki/encoder-decoder) [transformer](https://www.wikiprompt.org/wiki/transformer) architectures, where the encoder processes the input text and the decoder generates the output text. This design follows the original [transformer](https://www.wikiprompt.org/wiki/transformer) model but incorporates modifications such as relative positional embeddings and layer normalization placed outside the residual path. The framework's flexibility allows it to handle a wide range of tasks, from grammatical error correction to code generation. ## Training The original T5 models were pre-trained on the Colossal Clean Crawled Corpus (C4), a massive dataset of text and code scraped from the internet. Pre-training involved a denoising objective where the model learned to reconstruct corrupted text, using sentinel tokens to mark missing spans. For example, given "Thank you me to your party week.", the model learns to output " for inviting last ", where indicates end of output. After pre-training, T5 models can be fine-tuned on specific downstream tasks. Fine-tuning adapts the model's general language understanding to perform well in applications such as chatbots, machine translation, and text summarization. The framework's unified format means that fine-tuning data is also structured as input-output text pairs, making it straightforward to apply to new tasks. ## Architecture The T5 series includes models of varying sizes, all sharing the encoder-decoder structure. The original paper reported five configurations: small (60M parameters), base (220M), large (770M), 3B, and 11B. Each model has the same number of layers in the encoder and decoder; for example, T5-small has 6 layers in each. Key architectural dimensions include the number of attention heads, embedding dimension, feedforward network size, and key/value dimension. Notably, the 3B and 11B models deviate from the typical relationship where the embedding dimension equals the product of key/value dimension and number of heads. Compared to the original transformer, T5 uses layer normalization without additive bias, places normalization outside the residual path, and employs relative positional embeddings. A WordPiece tokenizer with a vocabulary size of 32,000 is shared across input and output, trained on a mixture of English, German, French, and Romanian data from C4. ## Variants Several subsequent models built upon the T5 architecture, often with non-standardized naming. The original models (2019) were followed by T5 1.1 (improved versions with GEGLU activation and renamed XL/XXL for the larger sizes). LM-adapted T5 (2021) continued training on additional C4 tokens. Switch Transformer (2021) introduced a mixture-of-experts variant. T0 (2021) focused on zero-shot instruction following. ByT5 (2021) operated on byte-level text without tokenizers. Flan-T5 (2022) added instruction tuning. UL2 20B (2022) scaled up to 20B parameters and used a mixture of denoisers objective. Pile-T5 (2024) used the Llama tokenizer and was trained on The Pile. ## Applications The T5 framework's encoder-decoder design enables instruction following: the encoder processes the instruction, and the decoder autoregressively generates the response. The T5 encoder can also serve as a text encoder for other models, producing vector representations for conditioning. For instance, Google Imagen uses T5-XXL as its text encoder, and AuraFlow uses Pile-T5-XL. These applications demonstrate the framework's versatility beyond traditional NLP tasks. The text-to-text paradigm has influenced subsequent [large language model](https://www.wikiprompt.org/wiki/large-language-model) development, emphasizing the benefits of a unified input-output format for transfer learning and multi-task training. --- Source: https://www.wikiprompt.org/wiki/text-to-text License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-12T16:29:03.566894+00:00