Natural Questions: A Large-Scale Benchmark for Open-Domain Question Answering
Natural Questions (NQ) is a large-scale dataset for open-domain question answering, introduced by researchers at Google in 2019. The dataset consists of real anonymized queries issued to the Google search engine, each paired with a Wikipedia page that contains the answer. It was designed to move beyond reading comprehension benchmarks by focusing on authentic user questions rather than artificially constructed ones.
Annotation and Structure
The dataset was built from anonymized search queries sampled between January and December 2018. Each query was paired with a Wikipedia page that Google's ranking system deemed likely to contain the answer. Annotators then performed a two-step labeling process: they first selected the long answer, which could be a paragraph, a table, or a list, and then extracted the short answer, which could be a single entity, a span of text, or a "yes"/"no" response. If no answer was present on the page, annotators marked it as such. This process resulted in a dataset with a natural distribution of answer types, including questions with no answer, which is a common but often overlooked scenario in real-world QA.
Impact on Question Answering Research
Natural Questions quickly became a standard benchmark for open-domain question answering, complementing earlier datasets like SQuAD. Unlike SQuAD, which provides a single passage containing the answer, Natural Questions requires systems to retrieve relevant information from a large corpus (typically Wikipedia) and then extract the answer. This setup aligns with the retrieval-augmented generation paradigm that later became central to large language model applications. The dataset has been used to evaluate both extractive and generative models, and it spurred the development of dense retrieval techniques, such as the Dense Passage Retriever (DPR) model introduced by Facebook AI in 2020, which achieved strong results on the benchmark.
Limitations and Criticisms
Despite its influence, Natural Questions has limitations. The dataset is limited to Wikipedia as the answer source, which may not reflect the full diversity of web content. The annotation process, while thorough, is time-consuming and expensive, limiting the dataset's size compared to synthetic corpora. Additionally, some researchers have noted that the short answer annotations can be inconsistent, particularly for questions with multiple valid answers or where the answer is implicit. These issues have motivated the creation of complementary datasets, such as TyDi QA, which covers multiple languages, and ELI5, which focuses on long-form explanatory answers.
Legacy and Continued Use
Natural Questions remains a key resource for evaluating open-domain QA systems, especially in the context of retrieval-augmented generation pipelines used in modern artificial intelligence applications. It is also used in academic research to study model robustness, interpretability, and the gap between human and machine performance on real-world queries. The dataset's design principles, particularly its emphasis on authentic user questions, have influenced how researchers approach the problem of answering questions in open-domain settings, and it continues to be a reference point for new benchmarks and models.