# MMMU 2025.5

MMMU 2025.5 is the fifth 2025 update of the Massive Multi-discipline Multimodal Understanding benchmark, a suite for evaluating AI models on college-level multimodal tasks. It adds new questions and adjusts scoring to track progress in vision-language reasoning.

MMMU 2025.5 is the fifth update released in 2025 of the Massive Multi-discipline Multimodal Understanding (MMMU) benchmark, a widely used evaluation suite for [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) systems. The benchmark assesses [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)s and related models on their ability to perform college-level reasoning across disciplines that require both visual and textual understanding. MMMU 2025.5 continues the benchmark's tradition of periodic refreshes, introducing a new set of questions and revised scoring procedures to mitigate saturation effects and better differentiate among state-of-the-art systems.

The original MMMU benchmark was introduced in 2023 by a consortium of academic and industrial researchers, including teams from [stanford-ai-lab](https://www.wikiprompt.org/wiki/stanford-ai-lab), [carnegie-mellon-university](https://www.wikiprompt.org/wiki/carnegie-mellon-university), and [university-of-toronto](https://www.wikiprompt.org/wiki/university-of-toronto). It comprises questions drawn from textbooks, lecture slides, and exam papers across 30 subjects, including art, engineering, and medicine. Each question integrates an image (such as a diagram, chart, or photograph) with a multiple-choice or open-ended prompt, requiring models to perform [multi-head-attention](https://www.wikiprompt.org/wiki/multi-head-attention)-based reasoning over both modalities. The benchmark's difficulty lies in its demand for domain-specific knowledge and precise visual interpretation, distinguishing it from simpler [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) benchmarks.

MMMU 2025.5 specifically addresses the rapid progress seen in 2025, where earlier 2025 versions (MMMU 2025.1 through 2025.4) had become increasingly easy for leading models. The update adds approximately 1,200 new questions, curated by a panel of 50 expert annotators from institutions such as [oxford-university](https://www.wikiprompt.org/wiki/oxford-university) and [mit-csail](https://www.wikiprompt.org/wiki/mit-csail). These questions emphasize multi-step reasoning, counterfactual scenarios, and fine-grained visual differences, such as distinguishing between similar anatomical structures or interpreting complex circuit diagrams. The update also introduces a new scoring rubric that penalizes overconfident wrong answers, encouraging calibrated [temperature-scaling](https://www.wikiprompt.org/wiki/temperature-scaling) in model outputs.

## Benchmark Design and Updates

The MMMU suite is structured into six broad disciplines: Art and Design, Business, Science, Health and Medicine, Humanities and Social Science, and Technology and Engineering. Each discipline contains multiple subjects, and questions are tagged with difficulty levels (easy, medium, hard) based on human performance. MMMU 2025.5 retains this structure but rebalances the distribution: it increases the proportion of hard questions from 35% to 45%, reflecting the need to challenge models that have mastered easier items. The update also introduces a new "visual perturbation" subset, where images are slightly rotated, cropped, or color-shifted, testing robustness to input variations.

A key innovation in MMMU 2025.5 is the inclusion of "adversarial collaboration" questions, developed with input from [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind) and [openai](https://www.wikiprompt.org/wiki/openai) researchers. These questions are designed to be ambiguous or to require external knowledge not present in the image, forcing models to either ask for clarification or state uncertainty. This aligns with broader trends in [generative-ai](https://www.wikiprompt.org/wiki/generative-ai) evaluation, where benchmarks increasingly measure not just accuracy but also reasoning transparency and failure modes.

## Evaluation Methodology

Models are evaluated in a zero-shot setting, meaning they receive no prior examples of MMMU questions. The benchmark uses a standardized prompt format that includes the image and a text instruction, and models must generate a response that is parsed for the final answer. For multiple-choice questions, the model's output is matched against the correct option; for open-ended questions, a combination of exact match and semantic similarity (using [transformer](https://www.wikiprompt.org/wiki/transformer)-based embeddings) is employed. MMMU 2025.5 also reports performance separately for models that use external tools, such as [amazon-web-services](https://www.wikiprompt.org/wiki/amazon-web-services) or [google-cloud](https://www.wikiprompt.org/wiki/google-cloud) APIs, though such usage is discouraged to maintain comparability.

Scoring in MMMU 2025.5 introduces a "confidence-weighted accuracy" metric. For each question, the model must output a confidence score (0 to 1). The final score is the average of correct answers weighted by confidence, with a penalty for high-confidence errors. This metric aims to capture calibration, a property that is critical for real-world deployment in domains like [intuitive-surgical](https://www.wikiprompt.org/wiki/intuitive-surgical) robotics or [waymo](https://www.wikiprompt.org/wiki/waymo) autonomous driving, where overconfident mistakes can be costly.

## Performance Trends

As of the release of MMMU 2025.5, leading models from [anthropic](https://www.wikiprompt.org/wiki/anthropic), [openai](https://www.wikiprompt.org/wiki/openai), and [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind) achieve accuracy in the range of 78-82%, up from around 70% on MMMU 2025.1. However, the new confidence-weighted metric reduces these scores to 65-70%, highlighting that many models remain poorly calibrated. Smaller models, such as those optimized for edge devices by [qualcomm](https://www.wikiprompt.org/wiki/qualcomm) or [arm-holdings](https://www.wikiprompt.org/wiki/arm-holdings), typically score 20-30 points lower, underscoring the gap between frontier and deployed systems.

The benchmark has also revealed persistent weaknesses in spatial reasoning and domain-specific terminology. For example, models often confuse left-right orientations in medical imaging and misidentify chemical structures in organic chemistry questions. These findings have motivated research into [curriculum-learning](https://www.wikiprompt.org/wiki/curriculum-learning) and [data-augmentation](https://www.wikiprompt.org/wiki/data-augmentation) techniques that specifically target these deficiencies.

## Impact and Reception

MMMU 2025.5 has been adopted by major AI labs as an internal evaluation checkpoint. [openai](https://www.wikiprompt.org/wiki/openai) and [anthropic](https://www.wikiprompt.org/wiki/anthropic) have publicly cited MMMU scores in model release notes, and [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind) uses the benchmark to guide training on [residual-network](https://www.wikiprompt.org/wiki/residual-network) and [u-net](https://www.wikiprompt.org/wiki/u-net) architectures for vision tasks. The benchmark's updates are also used by academic researchers to study the generalization of [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) models, with papers from [berkeley-ai-research](https://www.wikiprompt.org/wiki/berkeley-ai-research) and [bhabha-atomic-research](https://www.wikiprompt.org/wiki/bhabha-atomic-research) analyzing error patterns.

Critics have noted that MMMU's reliance on English-language, Western-centric educational materials may introduce cultural bias, a concern echoed by researchers at [alibaba-damiao-academy](https://www.wikiprompt.org/wiki/alibaba-damiao-academy) and [nec](https://www.wikiprompt.org/wiki/nec). In response, the MMMU team has announced plans for a multilingual expansion in late 2025, though details remain unconfirmed. Despite these limitations, MMMU 2025.5 remains a standard reference for multimodal reasoning, complementing other benchmarks like [chess-computer](https://www.wikiprompt.org/wiki/chess-computer) evaluations for strategic thinking.

## Future Directions

The MMMU team, led by principal investigators from [stanford-ai-lab](https://www.wikiprompt.org/wiki/stanford-ai-lab) and [university-of-toronto](https://www.wikiprompt.org/wiki/university-of-toronto), has outlined a roadmap for 2026. Planned updates include dynamic question generation using [generative-ai](https://www.wikiprompt.org/wiki/generative-ai) models, which would create unique questions per evaluation run, and integration of video-based tasks. The team is also exploring partnerships with [nokia-bell-labs](https://www.wikiprompt.org/wiki/nokia-bell-labs) and [xerox-parc](https://www.wikiprompt.org/wiki/xerox-parc) to develop benchmarks for multimodal systems in industrial settings, such as quality control and remote sensing.

As [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) systems become more capable, benchmarks like MMMU 2025.5 play a crucial role in ensuring that progress is measurable and meaningful. The update's emphasis on calibration and robustness reflects a maturation of the field, moving beyond raw accuracy to more nuanced assessments of model behavior. For practitioners, MMMU 2025.5 offers a rigorous testbed for comparing models and guiding development priorities.

---
Source: https://www.wikiprompt.org/wiki/mmmu-2025-5
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-13T03:54:56.103899+00:00