# XNLI XNLI (Cross-lingual Natural Language Inference) is a benchmark dataset for evaluating natural language inference models across 15 languages, created by Facebook AI Research in 2018. It extends the English MultiNLI corpus through human translation and is widely used in multilingual NLP research. XNLI (Cross-lingual Natural Language Inference) is a benchmark dataset designed to evaluate the ability of natural language processing models to perform natural language inference across multiple languages. Introduced by researchers at Facebook AI Research in 2018, the dataset has become a standard evaluation tool for cross-lingual understanding, particularly for testing whether models trained on one language can generalize to others without additional task-specific training data. The dataset covers 15 languages, including English, French, Spanish, German, Greek, Bulgarian, Russian, Turkish, Arabic, Vietnamese, Thai, Chinese, Hindi, Swahili, and Urdu. Natural language inference, also known as textual entailment, is the task of determining whether a given hypothesis is entailed by, contradicted by, or neutral with respect to a premise. XNLI provides premise-hypothesis pairs in each of its 15 languages, with each pair labeled as one of three categories: entailment, contradiction, or neutral. The dataset is built upon the MultiNLI corpus, which contains English sentence pairs, and extends it by translating the development and test sets into the other 14 languages using professional human translators. This design allows researchers to evaluate cross-lingual transfer, where a model is trained on English data and then tested on non-English languages, as well as zero-shot and few-shot learning scenarios. ## Construction and Composition The XNLI dataset was constructed by selecting 5,000 development and 5,000 test premise-hypothesis pairs from the MultiNLI corpus. Each pair was translated from English into the 14 other languages by professional translators, ensuring high linguistic quality and cultural appropriateness. The training data, however, remains in English, drawn from the original MultiNLI training set, which contains over 390,000 pairs. This setup forces models to learn inference reasoning from English and then apply it to other languages, making XNLI a rigorous test of cross-lingual generalization. The 15 languages were chosen to represent a diverse set of language families and scripts, including Indo-European (English, French, Spanish, German, Greek, Bulgarian, Russian, Turkish, Hindi, Urdu), Afro-Asiatic (Arabic), Sino-Tibetan (Chinese), Austroasiatic (Vietnamese), Kra-Dai (Thai), and Niger-Congo (Swahili). This diversity challenges models to handle morphological, syntactic, and script variations. Each language subset contains the same 10,000 pairs (5,000 dev and 5,000 test), allowing for direct comparison across languages. ## Evaluation Metrics and Use Cases XNLI is typically evaluated using accuracy, which measures the percentage of correctly classified premise-hypothesis pairs. The dataset supports two main evaluation protocols: translate-train, where training data is machine-translated into the target language, and translate-test, where the test set is translated into English. The translate-train approach is more common and reflects practical scenarios where labeled data is scarce in low-resource languages. Researchers also use XNLI to assess zero-shot cross-lingual transfer, where a model trained solely on English is evaluated directly on other languages without any target-language training data. The dataset has been instrumental in advancing multilingual models. For example, models like multilingual BERT and XLM have reported significant improvements on XNLI over earlier approaches, with accuracy scores rising from around 60% to over 70% on average across languages. As of 2023, state-of-the-art large language models such as GPT-4 and PaLM have achieved accuracy above 85% on XNLI, though performance still varies by language, with lower-resource languages like Swahili and Urdu typically showing lower scores than high-resource ones like French and German. ## Relationship to Other Benchmarks XNLI is part of a broader family of cross-lingual understanding benchmarks, including XGLUE and XTREME, which aggregate multiple tasks such as question answering and named entity recognition. XNLI is often used as a core component in these larger benchmarks due to its clean design and clear evaluation metric. It complements other natural language inference datasets like SNLI and MultiNLI by adding a multilingual dimension, enabling research on cross-lingual representation learning and transfer learning. The dataset has also influenced the development of multilingual pre-training objectives. Techniques such as cross-lingual language model pre-training, where models are trained on masked language modeling across multiple languages, have been directly evaluated on XNLI. The benchmark has been cited in thousands of research papers, making it a foundational resource in the field of multilingual natural language processing. ## Limitations and Criticisms Despite its widespread use, XNLI has limitations. The dataset is derived from English source texts, which means the non-English versions are translations rather than original texts, potentially introducing translationese artifacts that do not reflect natural usage in those languages. Additionally, the premise-hypothesis pairs are relatively short and may not capture complex reasoning required in real-world applications. Some researchers have noted that high accuracy on XNLI does not necessarily translate to robust cross-lingual understanding in other tasks, such as generation or dialogue. The fixed set of 15 languages also excludes many low-resource languages, limiting its applicability to truly low-resource scenarios. ## Impact and Legacy XNLI has played a pivotal role in popularizing cross-lingual evaluation in the machine learning community. It has been a key driver for the development of multilingual transformer models, including XLM-R and mT5, which have set new standards on the benchmark. The dataset is publicly available for research purposes and is integrated into popular libraries like Hugging Face's datasets and PyTorch, facilitating easy access and reproducibility. Its design has inspired similar efforts, such as the creation of multilingual versions of other tasks, and it remains a standard reference point for measuring progress in cross-lingual natural language understanding. ## See Also - [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) - [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) - [transformer](https://www.wikiprompt.org/wiki/transformer) - [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) - natural-language-inference --- Source: https://www.wikiprompt.org/wiki/xnli License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-12T16:27:49.865749+00:00