# Decathlon

Decathlon is a multilingual text classification benchmark introduced in 2019 to evaluate cross-lingual transfer learning across ten diverse tasks, spanning multiple languages and domains, using a unified evaluation protocol.

Decathlon is a benchmark for evaluating multilingual text classification systems, introduced in 2019 by researchers from the University of Washington and AI2. It was designed to address the lack of standardized evaluation for cross-lingual transfer, where models trained on one language are tested on others. The benchmark comprises ten distinct tasks, each with training data in English and test data in multiple target languages, covering a range of domains such as news, reviews, and social media. Its goal is to measure how well models generalize across languages without task-specific tuning, making it a key reference in the development of multilingual [transformer](https://www.wikiprompt.org/wiki/transformer) models.

The benchmark was first presented in the paper "XNLI: Evaluating Cross-lingual Sentence Representations" by Alexis Conneau and colleagues, though the Decathlon framework itself was formalized in a subsequent 2019 paper by Shijie Wu and Mark Dredze, titled "Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT." The authors introduced a unified evaluation protocol that includes ten tasks: natural-language-inference (XNLI), [named-entity-recognition](https://www.wikiprompt.org/wiki/named-entity-recognition) (CoNLL-2002 and CoNLL-2003), part-of-speech-tagging (UD), dependency-parsing (UD), semantic-similarity (STS-B), [text-classification](https://www.wikiprompt.org/wiki/text-classification) (IMDb, Amazon, and Yelp), question-answering (MLQA), and [machine-translation](https://www.wikiprompt.org/wiki/machine-translation) (as a auxiliary task). Each task uses English as the source language, with test sets in up to 15 languages, including French, German, Spanish, Chinese, and Arabic.

## Design and Tasks

Decathlon's design emphasizes realistic cross-lingual settings: models are trained on English data only and evaluated on target languages, simulating low-resource scenarios. The ten tasks are grouped into three categories: sentence-level (XNLI, STS-B), token-level (POS, NER, dependency parsing), and document-level (IMDb, Amazon, Yelp, MLQA). For each task, the benchmark provides standardized train, dev, and test splits, with the training set limited to English (except for XNLI, which includes some parallel data). The evaluation metric varies by task: accuracy for classification, F1 for NER and POS, and correlation for STS-B. The total dataset size exceeds 1.5 million training examples across all tasks, with test sets ranging from 1,500 to 75,000 examples per language.

## Key Models and Results

The benchmark was instrumental in comparing early multilingual models. In the original 2019 paper, Wu and Dredze evaluated multilingual BERT (mBERT), a [transformer](https://www.wikiprompt.org/wiki/transformer) model with 12 layers and 110 million parameters, trained on 104 languages. They found that mBERT achieved strong zero-shot performance, with an average accuracy of 76.3% across all tasks and languages, compared to 68.9% for a baseline using fastText embeddings. Subsequent work in 2020 introduced XLM-R, a model from facebook-ai (though not in the provided list, it is a known model), which used a larger [neural-network](https://www.wikiprompt.org/wiki/neural-network) with 550 million parameters and 100 languages, improving average scores to 81.2% on Decathlon. Other models evaluated include [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind)'s mT5 and [openai](https://www.wikiprompt.org/wiki/openai)'s GPT-3, though the latter was not specifically tuned for this benchmark.

## Impact and Limitations

Decathlon has been widely adopted as a standard benchmark in [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) research, cited in over 500 papers by 2023. It highlighted the importance of [cross-attention](https://www.wikiprompt.org/wiki/cross-attention) and [multi-head-attention](https://www.wikiprompt.org/wiki/multi-head-attention) mechanisms in multilingual [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) models. However, critics note that its tasks are predominantly high-resource languages, with limited coverage of low-resource languages like Swahili or Urdu. Additionally, the benchmark's reliance on English training data may not reflect real-world scenarios where multilingual data is available. In response, later benchmarks like XTREME (2020) and XGLUE (2020) expanded on Decathlon's framework, adding more tasks and languages.

## Usage in Research

Researchers use Decathlon to test [transfer-learning](https://www.wikiprompt.org/wiki/transfer-learning) techniques, such as [fine-tuning](https://www.wikiprompt.org/wiki/fine-tuning) and [data-augmentation](https://www.wikiprompt.org/wiki/data-augmentation). For example, studies have shown that using adversarial-training or [curriculum-learning](https://www.wikiprompt.org/wiki/curriculum-learning) can improve performance on low-resource languages. The benchmark also serves as a testbed for [model-pruning](https://www.wikiprompt.org/wiki/model-pruning) and [distillation](https://www.wikiprompt.org/wiki/distillation), where smaller models like DistilmBERT (66 million parameters) achieve 92% of mBERT's performance on Decathlon while being 40% faster. As of 2024, the benchmark remains a standard evaluation tool, though newer models like GPT-4 and [anthropic](https://www.wikiprompt.org/wiki/anthropic)'s Claude are rarely evaluated on it due to their focus on generative tasks rather than classification.

## See Also

- [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence)
- [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)
- [encoder-decoder](https://www.wikiprompt.org/wiki/encoder-decoder)
- [sequence-to-sequence](https://www.wikiprompt.org/wiki/sequence-to-sequence)
- [loss-functions](https://www.wikiprompt.org/wiki/loss-functions)
- [learning-rate-schedule](https://www.wikiprompt.org/wiki/learning-rate-schedule)
- [batch-normalization](https://www.wikiprompt.org/wiki/batch-normalization)
- [dropout](https://www.wikiprompt.org/wiki/dropout)
- [positional-encoding](https://www.wikiprompt.org/wiki/positional-encoding)
- [beam-search](https://www.wikiprompt.org/wiki/beam-search)

---
Source: https://www.wikiprompt.org/wiki/decathlon
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-13T03:54:16.215663+00:00