CLUE (Chinese Language Understanding Evaluation) is a benchmark suite designed to evaluate the performance of natural language understanding models on Chinese text. Introduced in 2020, it provides a standardized set of tasks that assess a model's ability to handle various aspects of Chinese language processing, including text classification, natural language inference, and reading comprehension. CLUE has become a widely adopted reference point for researchers and developers working on Chinese Natural language processing models, comparable to the role of GLUE and SuperGLUE for English.
The benchmark was created to address the lack of comprehensive evaluation tools for Chinese, which has unique linguistic characteristics such as the absence of word boundaries and a rich system of characters and idioms. CLUE aggregates multiple datasets, each targeting a specific language understanding skill, and provides a leaderboard to track the progress of models over time. It has been instrumental in driving improvements in Chinese Large language models and other Deep learning approaches.
Task Composition
CLUE consists of nine distinct tasks, each derived from existing Chinese NLP datasets. These tasks cover a range of difficulties and linguistic phenomena. The core tasks include:
- TNEWS: A short-text classification task based on news headlines, requiring models to categorize text into 15 topic classes.
- IFLYTEK: A long-text classification task with 119 categories, sourced from user-generated app descriptions, testing the ability to handle noisy, informal text.
- CMNLI: A natural language inference task where models must determine if a hypothesis entails, contradicts, or is neutral to a premise, based on the Chinese Multi-Genre NLI corpus.
- OCNLI: Another inference task, derived from the Original Chinese NLI dataset, focusing on more challenging reasoning scenarios.
- CLUEWSC2020: A coreference resolution task based on Winograd Schema-like sentences, requiring commonsense reasoning to resolve pronoun references.
- CSL: A keyword recognition task where models identify whether a given keyword is present in a scientific paper abstract, testing domain-specific understanding.
- DRCD: A reading comprehension task with extractive question answering, based on the Chinese version of the SQuAD-style dataset.
- CMRC2018: Another reading comprehension task, focusing on span extraction from Chinese passages.
- CHID: A cloze-style task where models select the correct idiom from a set of candidates to fill a blank, testing knowledge of Chinese idioms.
Each task is designed to probe different capabilities, from basic classification to complex reasoning, providing a holistic assessment of a model's Chinese understanding.
Scoring and Leaderboard
CLUE provides a unified scoring mechanism, with each task having a specific metric. For classification tasks, accuracy is used; for inference tasks, accuracy is also the primary metric; for reading comprehension, the F1 score and exact match are reported. The overall CLUE score is computed as the average of the individual task scores, allowing for direct comparison between models. The official leaderboard, hosted on the CLUE website, ranks submissions based on this average score, encouraging competitive development.
Since its introduction, the leaderboard has seen steady improvement. Early models based on Transformer (architecture) architectures like BERT achieved scores in the 60-70 range, while more recent Large language models have pushed scores above 90, reflecting significant advances in Chinese NLP. The benchmark has also been updated with additional tasks, such as CLUE-ABSA for aspect-based sentiment analysis, to keep pace with evolving research needs.
Impact and Usage
CLUE has had a substantial impact on the Chinese NLP community. It serves as a standard evaluation tool for academic research, with many papers reporting results on CLUE tasks to demonstrate model effectiveness. It is also used in industry to benchmark commercial models and guide product development. For instance, Chinese tech companies and research institutions frequently use CLUE to validate their proprietary models before deployment.
The benchmark has also spurred the development of Chinese-specific model architectures and training techniques. For example, models like RoBERTa-wwm-ext and ERNIE, which incorporate whole-word masking and knowledge-enhanced pretraining, were optimized with CLUE in mind. The tasks have also highlighted the importance of handling Chinese-specific challenges, such as the granularity of tokenization and the use of idioms, leading to innovations in Tokenization and Data Augmentation.
Limitations and Criticisms
Despite its popularity, CLUE has faced criticism. Some researchers argue that the tasks are relatively narrow and do not fully capture the complexity of real-world Chinese language understanding. The benchmark primarily focuses on classification and extractive tasks, with limited coverage of generative tasks like summarization or dialogue. Additionally, the datasets may contain biases or artifacts that models can exploit, leading to inflated scores that do not reflect true generalization.
Another limitation is the static nature of the benchmark. As models improve, the tasks may become saturated, making it difficult to differentiate between top-performing systems. To address this, the CLUE team has periodically introduced new tasks and harder versions, but the pace of model development often outstrips these updates. Some have also noted that the leaderboard encourages overfitting to the specific datasets, rather than fostering robust, generalizable models.
Future Directions
The future of CLUE likely involves expansion into more diverse and challenging tasks, including generative and multi-turn dialogue understanding. There is also a push towards more dynamic benchmarks that can adapt to model capabilities, similar to efforts like superglue for English. As Chinese Large language models continue to evolve, CLUE will need to evolve alongside them, ensuring that it remains a relevant and rigorous evaluation standard for the community.