# SuperGLUE

SuperGLUE is a benchmark suite introduced in 2019 to evaluate natural language understanding systems, designed as a harder successor to the GLUE benchmark, with tasks requiring reasoning, commonsense, and multi-sentence inference.

SuperGLUE is a benchmark suite introduced in 2019 by researchers at Google, DeepMind, and New York University to evaluate the capabilities of natural language understanding (NLU) systems. It was designed as a more challenging successor to the earlier GLUE (General Language Understanding Evaluation) benchmark, which had become saturated by high-performing models. SuperGLUE consists of eight diverse tasks that test a system's ability to perform coreference resolution, question answering, textual entailment, and reasoning over multiple sentences, with a focus on tasks that require deeper linguistic understanding and commonsense knowledge.

The benchmark was created in response to the rapid progress in [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) and [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) models, particularly those based on the [transformer](https://www.wikiprompt.org/wiki/transformer) architecture, which had achieved near-human performance on GLUE. SuperGLUE raises the bar by including tasks that are more difficult for machines, such as recognizing textual entailment with multi-sentence premises and answering questions that require multi-step reasoning. The suite provides a single score that aggregates performance across all tasks, allowing for direct comparison of different models and approaches.

## Task Composition

SuperGLUE includes eight tasks, each targeting a distinct aspect of language understanding. These tasks are: BoolQ (boolean questions with yes/no answers), CB (commitment bank, a textual entailment task), COPA (choice of plausible alternatives, a causal reasoning task), MultiRC (multi-sentence reading comprehension), ReCoRD (reading comprehension with commonsense reasoning), RTE (recognizing textual entailment), WiC (word-in-context, a lexical disambiguation task), and WSC (winograd schema challenge, a pronoun resolution task). Each task is designed to be challenging for models that rely on surface-level patterns, requiring instead an understanding of syntax, semantics, and world knowledge.

Unlike GLUE, which included single-sentence tasks like sentiment analysis, SuperGLUE emphasizes tasks that involve multiple sentences and complex reasoning. For example, the WSC task requires a model to resolve pronouns based on subtle contextual cues, while COPA tests causal reasoning by asking which of two alternatives is more plausible given a premise. These tasks are drawn from existing datasets, but SuperGLUE provides a unified evaluation framework with standardized training, validation, and test splits.

## Scoring and Evaluation

SuperGLUE uses a single aggregate score, computed as the average of each task's metric. For most tasks, the metric is accuracy, but for MultiRC and ReCoRD, it uses F1 score to account for partial credit in multi-answer settings. The benchmark also includes a human performance baseline, which was established by having human annotators complete the tasks under similar conditions to the models. This baseline serves as a reference point for measuring progress.

To prevent overfitting, the test set is held out, and submissions are evaluated through a leaderboard hosted on the SuperGLUE website. Models are allowed to use additional training data, but the benchmark encourages fair comparison by reporting results with and without external data. The evaluation protocol also includes a diagnostic set, which provides fine-grained analysis of model capabilities across linguistic phenomena such as negation, quantification, and coreference.

## Impact on AI Research

SuperGLUE has had a significant impact on the field of [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) by driving the development of more robust language models. Shortly after its release, models like BERT and its successors were benchmarked against SuperGLUE, revealing performance gaps that motivated architectural improvements. For instance, the introduction of [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)s such as GPT-2 and later GPT-3 demonstrated that scaling up model size and training data could lead to substantial gains on SuperGLUE tasks, though even the largest models initially fell short of human performance on several tasks.

The benchmark also influenced the design of new training objectives and model architectures. Researchers at institutions like [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind) and [openai](https://www.wikiprompt.org/wiki/openai) used SuperGLUE to evaluate innovations such as multi-task learning, adversarial training, and the incorporation of external knowledge. By 2021, several models, including T5 and DeBERTa, achieved scores exceeding the human baseline, indicating that the benchmark had become saturated. This led to the development of even harder benchmarks, such as SuperGLUE's successor, which focuses on more complex reasoning and generation tasks.

## Limitations and Criticisms

Despite its success, SuperGLUE has faced criticisms. Some researchers argue that the benchmark's tasks, while challenging, do not fully capture real-world language understanding, as they are primarily multiple-choice or classification problems. The reliance on accuracy as a primary metric can also obscure differences in model behavior, such as calibration or robustness to adversarial inputs. Additionally, the benchmark's English-only focus limits its applicability to multilingual settings, a gap that later benchmarks like XGLUE aimed to address.

Another concern is the potential for models to exploit dataset artifacts or biases, leading to inflated scores that do not reflect genuine understanding. For example, some tasks in SuperGLUE were found to contain spurious correlations that models could leverage without performing the intended reasoning. This has prompted calls for more rigorous evaluation protocols and the development of diagnostic datasets that probe specific capabilities.

## Legacy and Successors

SuperGLUE remains a widely cited benchmark in the [natural language processing](https://www.wikiprompt.org/wiki/natural-language-processing) community, serving as a standard reference for evaluating NLU systems. Its design principles, such as using diverse tasks and a held-out test set, have influenced subsequent benchmarks like GLUE's successor, SuperGLUE's own successor, and the broader trend toward multi-task evaluation suites. The benchmark also contributed to the broader understanding of model scaling, as evidenced by the correlation between model size and performance on SuperGLUE tasks.

As of 2023, SuperGLUE is considered a mature benchmark, with many state-of-the-art models exceeding human performance. Its legacy lies in demonstrating the importance of challenging evaluation suites for advancing [neural-network](https://www.wikiprompt.org/wiki/neural-network) research and in highlighting the need for benchmarks that evolve alongside model capabilities. Future benchmarks are likely to incorporate more generative tasks, multilingual data, and interactive settings, building on the foundation laid by SuperGLUE.

---
Source: https://www.wikiprompt.org/wiki/superglue
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-08T06:10:45.016443+00:00