# BART BART is a denoising autoencoder for pretraining sequence-to-sequence models in natural language processing, introduced by Facebook AI in 2019. It combines bidirectional and autoregressive transformers to improve text generation and understanding. BART (Bidirectional and Auto-Regressive Transformer) is a denoising autoencoder designed for pretraining sequence-to-sequence models in natural language processing. Introduced by researchers at Facebook AI in 2019, BART corrupts text with an arbitrary noising function and learns a model to reconstruct the original text. It is notable for unifying several pretraining objectives, including those used in models like BERT and GPT, into a single framework that excels at both understanding and generation tasks. The architecture of BART is a standard sequence-to-sequence transformer, with a bidirectional encoder similar to BERT and an autoregressive decoder similar to GPT. This design allows the encoder to capture context from both left and right, while the decoder generates output token by token. The pretraining objective involves corrupting the input text and training the model to minimize the cross-entropy loss between the decoder's output and the original uncorrupted text. BART's flexibility in handling various noising strategies, such as token masking, token deletion, text infilling, sentence permutation, and document rotation, makes it a powerful general-purpose model. ## Development and Release BART was developed by a team at Facebook AI, now part of Meta AI, and was presented in the paper "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension" at the 2020 Annual Meeting of the Association for Computational Linguistics (ACL). The model was released as open-source software, with implementations available in the Fairseq library. The initial release included pretrained models for English, with base and large variants, and was later extended to multilingual versions such as mBART. The development of BART was motivated by the observation that existing pretraining methods were often specialized for either understanding tasks (like BERT) or generation tasks (like GPT). BART aimed to combine the strengths of both by using a sequence-to-sequence architecture that could be fine-tuned for a wide range of applications. The noising functions used during pretraining were designed to mimic various types of corruption that occur in real-world text, making the model robust to noisy inputs. ## Applications and Performance BART achieved state-of-the-art results on several benchmarks at the time of its release. It performed particularly well on abstractive dialogue, question answering, and summarization tasks. For example, on the CNN/DailyMail summarization dataset, BART-large achieved a ROUGE-1 score of 47.6, outperforming previous models. On the SQuAD question answering dataset, it achieved an F1 score of 94.6, matching or exceeding the performance of specialized models. BART also showed strong performance on natural language generation tasks such as response generation in dialogue systems. One of the key advantages of BART is its ability to be fine-tuned for both understanding and generation tasks with minimal task-specific modifications. For classification tasks, the same representation from the decoder is used, while for generation tasks, the decoder is used to produce output sequences. This versatility made BART a popular choice for practitioners in the field of [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) and [machine-learning](https://www.wikiprompt.org/wiki/machine-learning). ## Influence and Legacy BART has had a significant influence on subsequent research in pretraining for natural language processing. Its denoising autoencoder approach inspired later models such as T5, which also uses a text-to-text framework, and various other denoising-based pretraining methods. The concept of corrupting input text and reconstructing it has become a standard technique in the development of [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)s. BART's architecture and training methodology are also used in many domain-specific applications, such as biomedical text summarization and legal document processing. The model's success contributed to the broader trend of using transformer-based models in [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) and [neural-network](https://www.wikiprompt.org/wiki/neural-network) research. It demonstrated that sequence-to-sequence architectures could be effectively pretrained on large corpora and then adapted to a wide variety of tasks, paving the way for more unified models in the field. BART's open-source release also facilitated its adoption in both academic and industrial settings, with many companies integrating it into their natural language processing pipelines. ## Technical Details BART uses a standard transformer architecture with a bidirectional encoder and an autoregressive decoder. The base model has 6 layers in both the encoder and decoder, with a hidden size of 768 and 12 attention heads, totaling about 140 million parameters. The large model has 12 layers, a hidden size of 1024, and 16 attention heads, totaling about 400 million parameters. The pretraining data consisted of text from English Wikipedia and books, similar to the data used for BERT. The noising functions used in BART include token masking, where random tokens are replaced with a mask token; token deletion, where random tokens are removed; text infilling, where spans of text are replaced with a single mask token; sentence permutation, where sentences are shuffled; and document rotation, where the document is rotated to start at a random token. The model is trained to reconstruct the original text from the corrupted version, which forces it to learn both local and global dependencies in the text. ## See Also - [transformer](https://www.wikiprompt.org/wiki/transformer) - [generative-ai](https://www.wikiprompt.org/wiki/generative-ai) - [openai](https://www.wikiprompt.org/wiki/openai) - [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind) --- Source: https://www.wikiprompt.org/wiki/bart License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-07T02:35:25.515666+00:00