# XTREME

XTREME is a benchmark for evaluating cross-lingual transfer in multilingual models, covering 40 languages and 9 tasks across classification, sequence labeling, and question answering.

XTREME (Cross-lingual TRansfer Evaluation of Multilingual Encoders) is a benchmark suite designed to evaluate the cross-lingual generalization capabilities of multilingual models. Introduced in 2020 by researchers at Google Research, it provides a standardized framework for assessing how well models trained on high-resource languages can transfer knowledge to low-resource languages. The benchmark covers 40 languages and includes 9 tasks spanning three categories: sentence classification, sequence labeling, and question answering. XTREME has become a widely adopted standard for comparing multilingual models, influencing subsequent benchmarks such as XTREME-R and GLUE-X.

The benchmark was created to address the growing need for systematic evaluation of multilingual representations, particularly as [transformer](https://www.wikiprompt.org/wiki/transformer)-based models like multilingual-BERT and XLM-R began to emerge. Its design emphasizes realistic cross-lingual scenarios, where models are fine-tuned on English data and evaluated on other languages without additional training. This zero-shot transfer setting tests the model's ability to learn language-agnostic features, which is critical for practical applications in multilingual [natural-language-processing](https://www.wikiprompt.org/wiki/natural-language-processing).

## Task Composition

XTREME comprises nine tasks that cover a range of linguistic abilities. For sentence classification, it includes the Multilingual Amazon Reviews Corpus (MARC), which involves sentiment analysis across six languages, and the Cross-lingual Natural Language Inference (XNLI) task, which tests entailment and contradiction reasoning. For sequence labeling, it includes the Wikiann dataset for named entity recognition (NER) and the Universal Dependencies (UD) dataset for part-of-speech tagging, both covering numerous languages. For question answering, it includes the Machine Reading Comprehension (MLQA) and Cross-lingual Question Answering (XQuAD) datasets, which require extracting answers from passages in multiple languages. Additionally, it includes the TyDiQA-GoldP task for typologically diverse languages and the PAWS-X task for paraphrase identification.

Each task is designed to be challenging yet feasible for state-of-the-art models, with evaluation metrics tailored to the task type, such as accuracy for classification, F1 score for sequence labeling, and exact match for question answering. The benchmark provides a unified scoring mechanism that averages performance across all tasks, allowing for direct comparison of different models.

## Language Coverage

The 40 languages in XTREME are selected to represent a wide range of linguistic families and scripts, including Indo-European, Sino-Tibetan, Afro-Asiatic, and others. This diversity ensures that the benchmark tests not only lexical transfer but also syntactic and morphological generalization. Languages range from high-resource ones like English, Spanish, and Chinese to low-resource ones like Yoruba, Kinyarwanda, and Uyghur. The inclusion of low-resource languages is particularly important, as it highlights the limitations of models that rely heavily on large amounts of monolingual data.

The benchmark also includes languages with different scripts, such as Arabic, Hindi, and Thai, which pose additional challenges for tokenization and representation. This coverage has made XTREME a standard for evaluating the effectiveness of techniques like [data-augmentation](https://www.wikiprompt.org/wiki/data-augmentation) and [model-pruning](https://www.wikiprompt.org/wiki/model-pruning) in multilingual settings.

## Evaluation Protocol

The standard evaluation protocol for XTREME involves two main stages. First, a model is pre-trained on a large multilingual corpus, often using objectives like masked language modeling. Second, the model is fine-tuned on English training data for each task, and then evaluated on the corresponding test sets in all other languages. This zero-shot transfer setting is intended to simulate real-world scenarios where labeled data is scarce for many languages.

To ensure fair comparison, the benchmark provides a public leaderboard with results from various models, including those from [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind), [openai](https://www.wikiprompt.org/wiki/openai), and academic institutions. The leaderboard tracks performance on each task and the overall average, enabling researchers to identify strengths and weaknesses in different architectures. As of 2023, models like XLM-RoBERTa-large and mT5 have achieved strong results, but no model has yet reached human-level performance on all tasks.

## Impact and Extensions

XTREME has had a significant impact on the field of cross-lingual learning. It has spurred research into techniques such as adversarial training, language-specific adapters, and better tokenization strategies. The benchmark has also been extended to XTREME-R, which adds more tasks and languages, and to XTREME-UP, which focuses on user-centric tasks like speech and image understanding.

Critics have noted that XTREME's zero-shot setting may not fully capture real-world performance, as models often benefit from a small amount of target-language data. Nevertheless, it remains a foundational tool for evaluating multilingual models, and its tasks are frequently used in [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) research. The benchmark's code and datasets are publicly available, facilitating reproducibility and further innovation.

## See Also

- cross-lingual-transfer
- multilingual-model
- [natural-language-processing](https://www.wikiprompt.org/wiki/natural-language-processing)
- benchmark-dataset

---
Source: https://www.wikiprompt.org/wiki/xtreme
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-12T16:27:48.840919+00:00