# HotpotQA HotpotQA is a question answering dataset for training and evaluating AI systems on multi-hop reasoning, requiring models to synthesize information from multiple Wikipedia documents to answer complex questions. HotpotQA is a large-scale dataset designed for training and evaluating artificial intelligence systems on multi-hop question answering. It was introduced in 2018 by a team of researchers from Carnegie Mellon University and the University of Washington. The dataset consists of over 113,000 question-answer pairs, each requiring the model to reason across multiple supporting documents from Wikipedia to arrive at the correct answer. Unlike simpler question answering benchmarks that rely on a single passage, HotpotQA explicitly tests a system's ability to perform multi-hop reasoning, which involves connecting information from several sources in a logical chain. The primary goal of HotpotQA is to advance research in machine reading comprehension and reasoning. It addresses a significant limitation of earlier datasets, which often contained questions answerable from a single paragraph. By requiring the synthesis of information from two or more documents, HotpotQA pushes models toward more sophisticated understanding and inference. The dataset is widely used as a benchmark in the fields of [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) and [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence), particularly for evaluating the reasoning capabilities of [large language models](https://www.wikiprompt.org/wiki/large-language-model) and other neural architectures. ## Dataset Construction The HotpotQA dataset was created using a two-stage crowdsourcing pipeline. First, workers were asked to write questions that required information from multiple Wikipedia articles, with the constraint that the answer could not be found in any single article. Second, these questions were validated and the supporting facts were identified by annotators. Each question is accompanied by a list of supporting facts, which are specific sentences from the source documents that provide the evidence needed to answer the question. This design allows for both answer prediction and evidence extraction, enabling more transparent evaluation of model reasoning. The dataset includes two main types of questions: bridge and comparison. Bridge questions require linking an entity from one document to another, such as identifying a person's birthplace by connecting a film to its director. Comparison questions require contrasting attributes of two entities, such as determining which of two rivers is longer. This variety ensures that models must handle different reasoning patterns, not just simple fact retrieval. ## Evaluation Metrics HotpotQA is typically evaluated using several metrics. For answer prediction, the primary metric is exact match (EM), which measures the percentage of predictions that exactly match the ground truth answer. Additionally, F1 score is used to account for partial matches, considering the overlap of tokens between the predicted and true answers. For the supporting fact prediction task, joint F1 and EM scores are computed, requiring models to identify the correct evidence sentences. These metrics provide a comprehensive assessment of both the final answer accuracy and the reasoning process. In the original 2018 release, the dataset was split into training, development, and test sets, with the test set used for a public leaderboard. The leaderboard allowed researchers to compare their models against state-of-the-art approaches, fostering rapid progress in the field. Over time, HotpotQA has become a standard benchmark in the [natural-language-processing](https://www.wikiprompt.org/wiki/natural-language-processing) community, although it is not explicitly listed in the provided link slugs. ## Impact on AI Research HotpotQA has had a significant impact on the development of reasoning models. It has driven innovations in architectures that incorporate attention mechanisms, memory networks, and graph-based reasoning. For example, early models used [transformer](https://www.wikiprompt.org/wiki/transformer)-based encoders to jointly encode the question and multiple documents, while later approaches employed graph neural networks to model relationships between entities across documents. The dataset also highlighted the importance of explainability, as the supporting fact annotations enable researchers to analyze why a model made a particular prediction. The dataset has been used to evaluate the reasoning abilities of modern [large language models](https://www.wikiprompt.org/wiki/large-language-model) such as those developed by [openai](https://www.wikiprompt.org/wiki/openai), [anthropic](https://www.wikiprompt.org/wiki/anthropic), and [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind). These models, often fine-tuned or prompted with few-shot examples, have achieved high scores on HotpotQA, demonstrating their capacity for multi-step inference. However, the dataset remains challenging, as models still struggle with questions requiring complex temporal or causal reasoning. As of recent years, state-of-the-art systems achieve over 90% exact match on the development set, but the test set leaderboard shows continued room for improvement. ## Limitations and Criticisms Despite its popularity, HotpotQA has faced criticism. One limitation is that the questions are generated by crowdworkers, which can introduce biases or unnatural phrasing. Additionally, the dataset focuses on Wikipedia as the sole knowledge source, which limits the diversity of topics and may not reflect real-world question answering scenarios where information is scattered across various domains. Some researchers have noted that models can exploit shortcuts, such as relying on entity co-occurrence patterns, rather than performing genuine reasoning. This has led to the development of more challenging benchmarks that aim to mitigate such issues. Another concern is the static nature of the dataset. Since Wikipedia is constantly updated, the documents used in HotpotQA may become outdated, potentially affecting the validity of evaluations over time. However, the dataset remains a valuable resource for controlled experiments, as it provides a fixed set of documents and annotations. ## Related Benchmarks HotpotQA is part of a broader family of multi-hop question answering datasets. It is often compared with other benchmarks such as QAngaroo, ComplexWebQuestions, and MuSiQue. These datasets vary in scale, question complexity, and the number of hops required. HotpotQA's emphasis on supporting fact annotations distinguishes it from many others, making it particularly useful for studying evidence-based reasoning. In the context of [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) and [neural networks](https://www.wikiprompt.org/wiki/neural-network), HotpotQA serves as a rigorous testbed for developing novel architectures and training paradigms. ## See Also - [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) - [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) - [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) - [transformer](https://www.wikiprompt.org/wiki/transformer) --- Source: https://www.wikiprompt.org/wiki/hotpotqa License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-07T21:34:31.273571+00:00