# GLUE

GLUE (General Language Understanding Evaluation) is a benchmark suite introduced in 2018 to evaluate machine learning models on natural language understanding tasks, covering single-sentence, similarity, and inference tasks.

GLUE (General Language Understanding Evaluation) is a benchmark suite introduced in 2018 by researchers at New York University, the University of Washington, and DeepMind. It was designed to evaluate and compare machine learning models on a diverse set of natural language understanding (NLU) tasks, with the goal of encouraging the development of general-purpose models that can handle multiple tasks without task-specific engineering. The benchmark quickly became a standard reference point in the field of [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) and [machine-learning](https://www.wikiprompt.org/wiki/machine-learning)(machine learning), particularly for [deep learning](https://www.wikiprompt.org/wiki/deep-learning) and [neural network](https://www.wikiprompt.org/wiki/neural-network) research.

The benchmark consists of nine tasks that cover three main categories: single-sentence tasks, similarity and paraphrase tasks, and inference tasks. Single-sentence tasks include CoLA (Corpus of Linguistic Acceptability), which tests grammatical acceptability, and SST-2 (Stanford Sentiment Treebank), which tests sentiment analysis. Similarity and paraphrase tasks include MRPC (Microsoft Research Paraphrase Corpus), QQP (Quora Question Pairs), and STS-B (Semantic Textual Similarity Benchmark). Inference tasks include MNLI (Multi-Genre Natural Language Inference), QNLI (Question Natural Language Inference), RTE (Recognizing Textual Entailment), and WNLI (Winograd Natural Language Inference). Each task has its own evaluation metric, such as accuracy, F1 score, or Pearson correlation, and the overall GLUE score is the average of these metrics across all tasks.

## History and Motivation

GLUE was introduced in a 2018 paper titled "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding" by Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. The authors argued that existing benchmarks were often too narrow and allowed models to overfit to specific datasets. GLUE aimed to provide a more comprehensive evaluation by including tasks that require different types of linguistic knowledge, such as syntax, semantics, and common sense. The benchmark also included a public leaderboard, which allowed researchers to compare their models transparently.

The introduction of GLUE coincided with a period of rapid progress in [large language models](https://www.wikiprompt.org/wiki/large-language-model). Shortly after its release, models based on the [transformer](https://www.wikiprompt.org/wiki/transformer) architecture, such as BERT (Bidirectional Encoder Representations from Transformers) from Google, began to dominate the leaderboard. BERT's success on GLUE helped popularize the pretraining and fine-tuning paradigm, where a model is first trained on a large corpus of text and then adapted to specific tasks. This approach became foundational for subsequent models developed by organizations like [openai](https://www.wikiprompt.org/wiki/openai), [anthropic](https://www.wikiprompt.org/wiki/anthropic), and [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind).

## Tasks and Evaluation

The nine tasks in GLUE were selected to cover a range of difficulty and linguistic phenomena. CoLA, for instance, requires models to judge whether a sentence is grammatically acceptable, which tests syntactic knowledge. SST-2 involves binary sentiment classification from movie reviews. MRPC and QQP are paraphrase detection tasks, where models must determine if two sentences have the same meaning. STS-B is a regression task that scores semantic similarity on a scale from 0 to 5.

For inference tasks, MNLI is the largest, with over 400,000 sentence pairs across multiple genres, requiring models to classify entailment, contradiction, or neutrality. QNLI is a question-answering task reformulated as entailment, where the model must determine if a sentence contains the answer to a question. RTE is a smaller entailment task derived from news and Wikipedia. WNLI is a pronoun resolution task based on the Winograd Schema Challenge, but it was later found to be flawed due to a data leakage issue, and many models achieved only random performance on it.

The evaluation metrics vary: CoLA uses Matthews correlation, SST-2 and QNLI use accuracy, MRPC and QQP use F1 score, STS-B uses Pearson and Spearman correlations, and MNLI, RTE, and WNLI use accuracy. The final GLUE score is the average of these metrics, with each task weighted equally.

## Impact and Limitations

GLUE had a significant impact on the field by providing a common benchmark for comparing models. It helped drive improvements in [natural-language-processing](https://www.wikiprompt.org/wiki/natural-language-processing) research, and its leaderboard became a competitive arena for both academic and industrial labs. The benchmark also facilitated the development of multi-task learning approaches and contributed to the rise of transfer learning in NLP.

However, GLUE also faced criticism. Some researchers noted that the benchmark could be gamed by models that memorize patterns in the training data rather than truly understanding language. The WNLI task was particularly problematic, as it was later discovered that the test set contained examples that were not properly aligned with the training data, making it nearly impossible for models to perform well. In response, the GLUE team released a revised version called SuperGLUE in 2019, which included harder tasks and addressed some of these issues.

## Legacy and Successors

Despite its limitations, GLUE remains a widely cited benchmark and is often used as a baseline for evaluating new models. Its successor, SuperGLUE, introduced tasks that require more complex reasoning, such as multi-sentence inference and reading comprehension. Both benchmarks have been influential in shaping the evaluation of [generative AI](https://www.wikiprompt.org/wiki/generative-ai) systems, though newer benchmarks like MMLU and HELM have emerged to address broader capabilities.

GLUE's design also inspired similar benchmarks in other domains, such as XGLUE for cross-lingual understanding and CLUE for Chinese. The benchmark's emphasis on multi-task evaluation has become a standard practice in the field, and its public leaderboard model has been adopted by many subsequent efforts.

## See Also

- [machine-learning](https://www.wikiprompt.org/wiki/machine-learning)
- [deep-learning](https://www.wikiprompt.org/wiki/deep-learning)
- [neural-network](https://www.wikiprompt.org/wiki/neural-network)
- [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)
- [transformer](https://www.wikiprompt.org/wiki/transformer)
- [openai](https://www.wikiprompt.org/wiki/openai)
- [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind)

## References

- Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. arXiv preprint arXiv:1804.07461.
- Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., ... & Bowman, S. R. (2019). SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. arXiv preprint arXiv:1905.00537.

---
Source: https://www.wikiprompt.org/wiki/glue
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-08T06:10:32.555925+00:00