MMLU, short for Massive Multitask Language Understanding, is a AI benchmark introduced in 2020 by Dan Hendrycks and collaborators to measure the breadth of knowledge and reasoning ability of language models. It consists of roughly 15,900 multiple-choice questions spanning 57 subjects, including elementary mathematics, US history, computer science, law, and professional medicine, drawn largely from real exams and textbooks, ranging in difficulty from elementary school to professional and expert level.
Design and purpose
MMLU was designed to test the kind of broad world knowledge a model would need to acquire primarily through pretraining on large text corpora, rather than task-specific fine-tuning, making it a proxy for how much a model knows across disciplines rather than how well it performs any single narrow task. Its creator, Dan Hendrycks, later became a prominent AI safety researcher and founded the Center for AI Safety, and MMLU's construction reflected an early effort to move LLM evaluation beyond narrow benchmarks like reading comprehension or single-domain question answering toward something closer to general knowledge.
Adoption
Following its release alongside the era of GPT-3 and subsequent large language models, MMLU became one of the most widely reported benchmarks in model release papers and technical reports, cited by essentially every major lab, including OpenAI, Anthropic, Google DeepMind, and Meta AI, as a headline capability metric. Its ubiquity made it a common point of comparison across models trained by different organizations using different methods, filling a role similar to what ImageNet had played for Computer vision a decade earlier.
Saturation
Early language models scored only slightly above the 25% expected from random guessing on four-option questions, but performance rose quickly as models scaled: models exceeded 80% within a few years, and by 2024 leading frontier models surpassed 90%, approaching estimates of the score an expert human panel might achieve. This saturation reduced MMLU's ability to distinguish between top models and prompted the creation of harder successor benchmarks, including MMLU-Pro, which added more difficult questions and expanded answer choices to reduce the impact of guessing.
Criticism
MMLU has drawn criticism on several grounds: researchers identified a nontrivial rate of erroneous or ambiguous questions and answer keys in the original dataset; because it draws on publicly available exam materials, data contamination is a persistent concern, since some questions or near-duplicates likely appear in the web-scale data used to pretrain many models; and the multiple-choice format rewards recognition over generation, which may not reflect how models are actually used in open-ended tasks. Some of these issues were partially addressed in later variants, but MMLU's core design remains a snapshot of 2020-era evaluation priorities.
Legacy
Despite saturation and criticism, MMLU remains widely cited as a baseline reference point in model comparisons and continues to appear in release announcements even after ceasing to meaningfully differentiate top-tier models, illustrating a broader pattern in which influential benchmarks persist as a common vocabulary long after their discriminative power has faded.