The MMLU benchmark (Measuring Massive Multitask Language Understanding) is a widely used evaluation tool for assessing the capabilities of large language models. Released on 7 September 2020 by Dan Hendrycks and a team of researchers, it was designed to be more challenging than earlier benchmarks such as GLUE, as models began outperforming humans on simpler tests. MMLU consists of 15,908 multiple-choice questions spanning 57 subjects, from STEM fields and international law to nutrition and religion. Of these, 1,540 questions are reserved for selecting optimal model settings like temperature, batch size, and learning rate. By July 2024, the benchmark had been downloaded over 100 million times, making it one of the most common tools for comparing model performance in the Artificial intelligence community.
When MMLU was introduced, most existing models scored near random chance (25%), with the best model, GPT-3 175B, achieving 43.9% accuracy. The creators estimated that human domain experts would achieve around 89.8% accuracy. By mid-2024, leading models such as Claude 3.5 Sonnet, GPT-4o, and Llama 3.1 405B consistently reached about 88% accuracy. However, as of 2025, MMLU has been partially phased out in favor of more difficult alternatives, reflecting the rapid progress in model capabilities.
Structure and Content
The benchmark covers a broad range of subjects, including abstract algebra, international law, professional medicine, and many others. Each question is a multiple-choice item with four options, and models are evaluated on their accuracy across all subjects. The diversity of topics is intended to test not only factual knowledge but also reasoning and cross-domain understanding. The 1,540-question development set is used to tune hyperparameters, ensuring fair comparisons across models.
MMLU has inspired several spin-offs and variants, such as MMLU-Pro, MMMLU, and MMLU-Redux, which aim to address some of the original benchmark's limitations or provide more challenging evaluations. These derivatives have contributed to the ongoing evolution of model evaluation in Machine learning research.
Limitations and Criticism
Despite its popularity, MMLU has faced significant criticism. On 5 June 2024, experts released a paper detailing a manual analysis of 5,700 questions, revealing a substantial number of ground-truth errors. For instance, 57% of questions in the "Virology" subset were found to contain errors, including multiple correct answers (4%), unclear questions (14%), or completely incorrect answers (33%). Overall, the analysis estimated that 6.5% of MMLU questions contained an error, suggesting that the maximum attainable score was significantly below 100%.
Data contamination also posed a serious threat to the benchmark's validity. Companies could inadvertently or deliberately include MMLU questions and answers in their models' training data, effectively rendering the benchmark useless as a measure of generalization. This issue is common in the field of Generative AI evaluation, where models are often trained on vast internet corpora that may include benchmark questions.
Example Questions
The following examples illustrate the types of questions in MMLU, sourced from the "Abstract Algebra", "International Law", and "Professional Medicine" tasks. Correct answers are marked in boldface.
Question 1 (Abstract Algebra):
Find all \( c \) in \( \mathbb{Z}_3 \) such that \( \mathbb{Z}_3[x]/(x^2 + c) \) is a field.
(A) 0 (B) 1 (C) 2 (D) 3
Question 2 (International Law):
Would a reservation to the definition of torture in the International Covenant on Civil and Political Rights (ICCPR) be acceptable in contemporary practice?
(A) This is an acceptable reservation if the reserving country’s legislation employs a different definition.
(B) This is an unacceptable reservation because it contravenes the object and purpose of the ICCPR.
(C) This is an unacceptable reservation because the definition of torture in the ICCPR is consistent with customary international law.
(D) This is an acceptable reservation because under general international law States have the right to enter reservations to treaties.
Question 3 (Professional Medicine):
A 33-year-old man undergoes a radical thyroidectomy for thyroid cancer. During the operation, moderate hemorrhaging requires ligation of several vessels in the left side of the neck. Postoperatively, serum studies show a calcium concentration of 7.5 mg/dL, albumin concentration of 4 g/dL, and parathyroid hormone concentration of 200 pg/mL. Damage to which of the following vessels caused the findings in this patient?
(A) Branch of the costocervical trunk.
(B) Branch of the external carotid artery.
(C) Branch of the thyrocervical trunk.
(D) Tributary of the internal jugular vein.
Impact and Future
MMLU has played a crucial role in tracking the progress of large language models and has been widely adopted by research labs and companies, including OpenAI, Anthropic, and Google DeepMind. Its influence extends to other benchmarks and evaluation methodologies, pushing the field toward more robust and challenging tests. However, the identified errors and contamination issues have led to a gradual shift toward newer benchmarks that are more resistant to these problems. As of 2025, MMLU remains a historical reference point, but its role in cutting-edge model comparison is diminishing.