MMQA (Multimodal Question Answering) is a benchmark dataset designed to evaluate artificial intelligence systems on their ability to answer questions that require reasoning across multiple types of data, including text, images, tables, and videos. Introduced by researchers at Facebook AI Research (now part of Meta AI) in 2019, the dataset addresses the limitation of earlier question answering benchmarks that focused on a single modality, such as text-only or image-only inputs. MMQA provides a diverse set of questions and corresponding answers that necessitate the integration of information from different sources, pushing the boundaries of Machine learning and Deep learning models toward more human-like comprehension.
The dataset was created to support the development of models that can handle real-world queries where information is scattered across various formats. Unlike traditional datasets that assume a single source of truth, MMQA requires systems to locate, combine, and reason over evidence from multiple modalities simultaneously. This makes it a challenging testbed for Neural network architectures, particularly those based on the Transformer (architecture) model, which have become the dominant approach in Natural language processing and multimodal understanding.
Construction and Composition
MMQA was constructed by collecting questions from crowdworkers who were shown a set of Wikipedia articles, each containing text, images, tables, and in some cases videos. The workers were asked to generate questions that could only be answered by combining information from at least two different modalities. For example, a question might require reading a table of statistics and looking at an image to infer a relationship, or watching a video clip and reading a text passage to answer a factual query.
The final dataset comprises over 12,000 question-answer pairs, with each question linked to a specific Wikipedia page. The answers are short spans of text, typically a few words, which are extracted from the source content. The dataset is split into training, validation, and test sets, with the test set being used for official evaluation. The questions are designed to be complex and often require multi-step reasoning, making the dataset significantly more difficult than single-modality benchmarks like SQuAD or VQA.
Evaluation and Metrics
Models are evaluated on MMQA using standard question answering metrics, primarily exact match (EM) and F1 score. EM measures the percentage of questions where the model's predicted answer exactly matches the ground truth, while F1 computes the overlap of tokens between the prediction and the ground truth, allowing for partial credit. Since the answers are short spans, these metrics provide a straightforward measure of accuracy.
To succeed on MMQA, a model must not only understand each modality individually but also learn to align and fuse information across them. This requires sophisticated architectures that can encode images, tables, and videos into a common representation space, often using pre-trained vision and language models. Early baselines, such as simple concatenation of features, performed poorly, highlighting the need for more advanced multimodal fusion techniques.
Impact and Related Work
MMQA has influenced subsequent research in multimodal understanding and has been used as a benchmark in numerous studies. It is often cited alongside other multimodal datasets like Visual Question Answering (VQA) and TextVQA, but it stands out due to its inclusion of tables and videos, which are less common in other benchmarks. The dataset has spurred the development of models that can jointly reason over text and images, and later extensions have incorporated audio and other modalities.
The release of MMQA coincided with the rise of Large language models and Generative AI systems, which have shown remarkable capabilities in multimodal tasks. However, even state-of-the-art models as of 2024 still find MMQA challenging, particularly for questions that require fine-grained reasoning over tables or videos. This has made MMQA a valuable stress test for evaluating the robustness of Artificial intelligence systems in real-world scenarios.
Limitations and Future Directions
One limitation of MMQA is that it relies on Wikipedia articles, which are primarily in English and have a Western-centric bias. This limits the diversity of cultural and linguistic contexts represented in the dataset. Additionally, the answers are constrained to short spans, which may not capture the full complexity of some questions that require longer explanations.
Future work in this area may involve expanding MMQA to include more languages, more diverse sources, and longer-form answers. Researchers are also exploring how to make models more interpretable, so that they can explain the reasoning process behind their answers. As multimodal AI continues to evolve, datasets like MMQA will remain essential for measuring progress and identifying areas where models still fall short of human-level understanding.