# GLUE-ZH

GLUE-ZH is a Chinese-language benchmark suite for evaluating general-purpose language understanding models, inspired by the English GLUE benchmark. It aggregates multiple Chinese NLP tasks into a single score to measure model performance across diverse linguistic capabilities.

GLUE-ZH is a benchmark suite designed to evaluate general-purpose language understanding capabilities of models on Chinese text. It follows the structure of the English GLUE (General Language Understanding Evaluation) benchmark, adapting its tasks and evaluation methodology to the Chinese language. The benchmark provides a single aggregate score that reflects a model's performance across a range of fundamental NLP tasks, enabling direct comparison of different approaches in Chinese natural language processing.

The benchmark was introduced in the late 2010s as part of a broader effort to create multilingual evaluation standards for the rapidly advancing field of [deep learning](https://www.wikiprompt.org/wiki/deep-learning) and [large language models](https://www.wikiprompt.org/wiki/large-language-model). While the original GLUE benchmark was released in 2018 by researchers at New York University and the University of Washington, GLUE-ZH emerged as a community-driven adaptation to address the growing need for standardized Chinese NLP evaluation. It was developed by a consortium of academic and industrial research groups, though no single official release date or paper is universally recognized, and the benchmark's exact composition has varied across implementations.

## Task Composition

GLUE-ZH typically includes a set of tasks that mirror the English GLUE suite but use Chinese datasets. Common tasks include:

- **Sentence-level classification**: Tasks such as sentiment analysis (e.g., using the Chinese ChnSentiCorp dataset) and question-answer natural language inference (e.g., the CMNLI dataset, which is a Chinese version of the MultiNLI corpus).
- **Textual similarity**: Tasks like Chinese Semantic Textual Similarity (STS-B), which measures how semantically similar two sentences are on a scale from 0 to 5.
- **Single-sentence tagging**: Tasks such as Chinese word segmentation and part-of-speech tagging, which are essential for Chinese NLP due to the lack of spaces between words.
- **Reading comprehension**: Tasks like CMRC2018 (Chinese Machine Reading Comprehension), which requires models to answer questions based on given passages.

The exact number of tasks in GLUE-ZH has varied between versions, with some implementations including as few as five tasks and others expanding to nine or more. The most commonly cited configuration includes seven tasks, covering classification, similarity, and inference.

## Evaluation Methodology

GLUE-ZH uses a similar scoring system to the English GLUE benchmark. Each task has a specific metric (e.g., accuracy for classification tasks, Pearson correlation for similarity tasks), and the final GLUE-ZH score is the average of all task-specific scores. This aggregate score allows for a single-number comparison of model performance, simplifying the evaluation process.

Models are typically evaluated in a zero-shot or fine-tuned setting. In the zero-shot setting, models are tested on GLUE-ZH tasks without any task-specific training, which measures their general language understanding abilities. In the fine-tuned setting, models are trained on each task's training data before evaluation, which measures their adaptability to specific downstream tasks.

## Historical Context and Usage

GLUE-ZH gained prominence during the rise of [Transformer](https://www.wikiprompt.org/wiki/transformer)-based models in the late 2010s and early 2020s. It was used as a benchmark in several research papers and technical reports, particularly those focusing on Chinese pre-trained models such as BERT-based variants (e.g., BERT-wwm, RoBERTa-wwm) and ERNIE. These models consistently achieved high scores on GLUE-ZH, with the best-performing models reaching aggregate scores above 80% by 2021.

The benchmark has been criticized for its limited scope, as it focuses primarily on sentence-level tasks and does not include more complex tasks like dialogue generation or document-level understanding. Additionally, the lack of a single official version has led to inconsistencies in reported results across different studies, making direct comparisons difficult.

## Relationship to Other Benchmarks

GLUE-ZH is part of a family of multilingual benchmarks that include SuperGLUE (the successor to GLUE in English) and XTREME (a cross-lingual benchmark). While XTREME includes Chinese as one of many languages, GLUE-ZH provides a deeper evaluation specifically for Chinese. It is also related to the Chinese SuperGLUE (CLUE) benchmark, which was released in 2020 and offers a more comprehensive and standardized evaluation suite for Chinese NLP. CLUE has largely superseded GLUE-ZH in research usage due to its larger task set and official leaderboard.

## Current Status

As of the mid-2020s, GLUE-ZH is less frequently used in cutting-edge research, having been replaced by more comprehensive benchmarks like CLUE and the Chinese version of SuperGLUE. However, it remains a useful reference for understanding the evolution of Chinese NLP evaluation and is still cited in some educational materials and historical analyses. The benchmark's influence persists in the design of newer evaluation suites, which often incorporate similar task types and scoring methodologies.

## See Also

- [Artificial intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence)
- [Machine learning](https://www.wikiprompt.org/wiki/machine-learning)
- [Natural language processing](https://www.wikiprompt.org/wiki/natural-language-processing)
- [Transformer](https://www.wikiprompt.org/wiki/transformer)
- [Large language model](https://www.wikiprompt.org/wiki/large-language-model)

---
Source: https://www.wikiprompt.org/wiki/glue-zh
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-13T03:54:20.333561+00:00