QAngaroo is a benchmark dataset designed to evaluate the ability of artificial intelligence systems to perform multi-hop reasoning. Unlike standard question answering tasks that require retrieving a single fact, QAngaroo presents questions that necessitate combining information from multiple distinct documents or passages. The name is a portmionteau of "question answering" and "kangaroo," reflecting the task's focus on jumping between pieces of evidence. It was introduced in 2018 by researchers at University College London and Facebook AI Research, including Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel.
The benchmark comprises two main subsets: WikiHop and MedHop. WikiHop is constructed from Wikipedia articles, where each question is derived from a set of supporting documents that collectively contain the answer. MedHop is built from medical abstracts, focusing on drug-drug interaction predictions, requiring reasoning over biomedical literature. Both subsets are designed so that the answer cannot be found in any single document, forcing models to aggregate evidence across multiple sources.
Task Design and Evaluation
Each instance in QAngaroo consists of a query, a set of candidate answer options, and a collection of supporting documents. The model must select the correct answer by reasoning over the provided context. The evaluation metric is accuracy, measuring the percentage of correctly answered questions. The benchmark is intentionally challenging, as the supporting documents are often noisy and contain irrelevant information, requiring models to identify and combine only the relevant pieces.
The dataset was created using a semi-automatic pipeline. For WikiHop, the creators used Wikipedia's internal hyperlinks to generate multi-hop questions. They identified pairs of entities connected through an intermediate entity, then constructed a question that requires inferring the relationship between the two endpoints. For MedHop, they used a similar approach on medical abstracts, focusing on interactions between drugs.
Impact on Research
QAngaroo has become a standard benchmark for evaluating multi-hop reasoning capabilities in Machine learning and Deep learning models. It has been used extensively in research on Neural network architectures, particularly those employing attention-mechanisms and memory-augmented-networks. The benchmark highlighted the limitations of early Large language models, which often struggled with tasks requiring explicit multi-step inference. It spurred the development of specialized models, such as graph neural networks that operate on the document structure, and iterative reading approaches that refine evidence selection over multiple passes.
The benchmark also contributed to the broader understanding of reasoning in AI. It demonstrated that simple Sequence-to-Sequence (Seq2Seq) models often fail on multi-hop tasks, while models with explicit reasoning components, such as Graph Neural Networks or those using Reinforcement learning for evidence selection, perform significantly better. As of 2023, state-of-the-art results on WikiHop have reached over 70% accuracy, but the benchmark remains unsolved, with human performance estimated at around 90%.
Limitations and Criticisms
Despite its influence, QAngaroo has faced criticism. Some researchers have noted that the benchmark's construction may allow models to exploit shortcuts, such as answer leakage through candidate options or superficial patterns in the supporting documents. For instance, the answer is often the entity that appears most frequently across documents, which can be exploited by frequency-based heuristics. This has led to concerns that high scores on QAngaroo may not necessarily reflect genuine reasoning ability.
Additionally, the benchmark's focus on Wikipedia and medical abstracts limits its domain coverage. The questions are relatively short and the reasoning steps are typically limited to two or three hops, which may not capture the complexity of real-world multi-hop reasoning tasks. As a result, some researchers have called for more diverse and challenging benchmarks, leading to the development of subsequent datasets like HotpotQA and 2WikiMultiHopQA.
Legacy and Continued Use
Despite these limitations, QAngaroo remains a widely used tool for evaluating and comparing AI models. It is included in several public leaderboards and is often used as a baseline in research papers on question answering and reasoning. The benchmark's design principles, particularly the use of multiple supporting documents and the requirement for evidence aggregation, have influenced the creation of many subsequent datasets. It also serves as a valuable resource for studying the interpretability of models, as the multi-hop structure allows researchers to trace the reasoning path taken by a model.
In the context of the broader field, QAngaroo is part of a lineage of reasoning benchmarks that have pushed the boundaries of Artificial intelligence. It has been used to evaluate models from various institutions, including Google DeepMind, OpenAI, and academic labs like Stanford AI Lab and BAIR (Berkeley AI Research). While newer benchmarks have emerged, QAngaroo's role in highlighting the importance of multi-hop reasoning and its impact on model design ensures its continued relevance in AI research.
See Also
- HotpotQA
- multi-hop-reasoning
- question-answering
- Natural language processing