# SuperGLUE Benchmark

SuperGLUE is a 2019 benchmark suite of eight challenging natural language understanding tasks, designed to replace the GLUE benchmark and better measure progress in general-purpose language understanding systems. It was introduced by researchers at New York University, DeepMind, and the University of Washington.

SuperGLUE is a benchmark suite for evaluating the performance of natural language understanding (NLU) models. Released in 2019, it was designed to succeed the earlier GLUE (General Language Understanding Evaluation) benchmark, which had become saturated as models approached or exceeded human-level performance on its tasks. SuperGLUE presents a more difficult set of eight tasks that require reasoning, commonsense knowledge, and multi-sentence understanding, aiming to provide a more robust measure of a model's general language capabilities.

The benchmark was introduced in a paper titled "SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems," authored by Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. The work was a collaboration between New York University, DeepMind, and the University of Washington. The name is a play on the adhesive brand Super Glue, reflecting its purpose as a "stickier" benchmark that is harder for models to solve.

## Tasks and Evaluation

SuperGLUE consists of eight tasks, each targeting a distinct aspect of language understanding:

- **BoolQ** (Boolean Questions): Answering yes/no questions given a short passage.
- **CB** (CommitmentBank): Determining whether a premise entails, contradicts, or is neutral with respect to a hypothesis.
- **COPA** (Choice of Plausible Alternatives): Selecting the more plausible cause or effect from two options given a premise.
- **MultiRC** (Multi-Sentence Reading Comprehension): Answering multiple questions about a passage, with each question having multiple possible answers.
- **ReCoRD** (Reading Comprehension with Commonsense Reasoning): Filling in a masked entity in a news article using commonsense reasoning.
- **RTE** (Recognizing Textual Entailment): A binary entailment task combining several earlier datasets.
- **WiC** (Word-in-Context): Determining whether a word is used with the same meaning in two different sentences.
- **WSC** (Winograd Schema Challenge): Resolving pronoun references in sentences designed to be ambiguous for simple statistical models.

Each task has a specific metric: accuracy for BoolQ, CB, COPA, RTE, and WSC; F1 score for MultiRC and ReCOORD; and accuracy for WiC. The overall SuperGLUE score is the average of the individual task scores, with each task weighted equally.

## Motivation and Design

The GLUE benchmark, released in 2018, had become a standard for evaluating general-purpose language models. However, by early 2019, models such as BERT and its variants were achieving scores close to or exceeding the estimated human baseline on GLUE, making it difficult to differentiate between models. SuperGLUE was created to address this saturation by including tasks that are more challenging and require deeper reasoning, such as the Winograd Schema Challenge and COPA, which involve commonsense knowledge and causal reasoning.

The tasks were selected from existing datasets that were considered too difficult for contemporary models at the time. The authors also introduced a new dataset, ReCoRD, which was created specifically for the benchmark. The evaluation framework includes a public leaderboard, allowing researchers to compare their models against a standard set of metrics and a human performance baseline.

## Impact and Reception

SuperGLUE quickly became a widely used benchmark in the [natural-language-processing](https://www.wikiprompt.org/wiki/natural-language-processing) community. It was adopted by major research groups and companies, including [openai](https://www.wikiprompt.org/wiki/openai), [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind), and [microsoft](https://www.wikiprompt.org/wiki/microsoft), as a standard testbed for new architectures and training methods. The benchmark played a significant role in the development of [transformer](https://www.wikiprompt.org/wiki/transformer)-based models, particularly in the era of [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)s.

In 2019, shortly after its release, models such as BERT-large and XLNet achieved scores in the high 70s, while the human baseline was estimated at 89.8. By 2021, models like T5 and DeBERTa surpassed the human baseline, with DeBERTa achieving a score of 90.8 in January 2021. This rapid progress led to the benchmark being considered "solved" by many, prompting the creation of even harder benchmarks such as SuperGLUE's successor, BIG-bench, and later the HELM benchmark.

Despite its success, SuperGLUE has also faced criticism. Some researchers argued that the tasks, while challenging, still do not fully capture the complexity of real-world language understanding, and that high scores on the benchmark do not necessarily translate to robust performance in practical applications. Others noted that the benchmark's focus on English-only tasks limits its applicability to multilingual settings.

## Legacy and Successors

SuperGLUE's legacy is twofold. First, it demonstrated the value of creating progressively harder benchmarks to drive research in [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence). Second, it highlighted the rapid pace of progress in [machine-learning](https://www.wikiprompt.org/wiki/machine-learning), as models went from below-human to above-human performance in just two years. The benchmark's design principles - using diverse tasks, clear metrics, and a public leaderboard - have been adopted by subsequent benchmarks such as GLUE's successor, SuperGLUE's own successor, and the broader evaluation ecosystem.

The original GLUE benchmark was retired in 2021, and SuperGLUE's leaderboard was frozen in 2022, as models continued to improve. However, the tasks and datasets remain in use for research and development, and the benchmark is still referenced in academic papers and industry reports as a historical milestone in the evaluation of language models.

---
Source: https://www.wikiprompt.org/wiki/superglue-benchmark
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-09T02:01:01.981774+00:00