Natural Questions is a large-scale dataset for open-domain question answering research, introduced by Google researchers in 2019. The dataset consists of real anonymized queries issued to the Google search engine, each paired with a Wikipedia page that contains the answer, along with a long answer (a paragraph or table) and a short answer (a span or entity) where applicable. It was designed to move question answering beyond reading comprehension benchmarks by focusing on genuine information-seeking questions that users actually type into a search engine, rather than artificially constructed queries.
The dataset was created to address the limitations of earlier QA datasets, which often relied on synthetic questions or assumed the answer was always present in a given passage. Natural Questions instead reflects the complexity of real-world search behavior, including ambiguous queries, multi-part questions, and cases where no answer exists on the page. The corpus includes 307,373 training examples, 7,830 development examples, and 7,842 test examples, with answers annotated by human raters using a two-stage process: first identifying the long answer (a paragraph or table) and then the short answer (a span or entity) within it.
Construction and Annotation
The dataset was built from anonymized Google search queries sampled between January and December 2018. Each query was matched to a Wikipedia page that Google's ranking system deemed likely to contain the answer. Annotators then performed a two-step labeling process: they first selected the long answer, which could be a paragraph, a table, or a list, and then extracted the short answer, which could be a single entity, a span of text, or a 'yes'/'no' response. If no answer was present on the page, annotators marked it as such. This process resulted in a dataset with a natural distribution of answer types, including questions with no answer, which is a common but often overlooked scenario in real-world QA.
Benchmark Significance
Natural Questions quickly became a standard benchmark for open-domain question answering, complementing earlier datasets like SQuAD. Unlike SQuAD, which provides a passage containing the answer, Natural Questions requires systems to retrieve relevant information from a large corpus (typically Wikipedia) and then extract the answer. This setup aligns with the retrieval-augmented generation paradigm that later became central to large language model applications. The dataset has been used to evaluate both extractive and generative models, and it spurred the development of dense retrieval techniques, such as the DPR (Dense Passage Retriever) model introduced by Facebook AI in 2020, which achieved strong results on the benchmark.
Impact on Question Answering Research
Natural Questions influenced the design of subsequent QA datasets and systems. Its focus on real user queries highlighted the importance of handling noise, ambiguity, and unanswerable questions. The dataset also contributed to the shift from purely extractive QA to generative QA, where models synthesize answers rather than merely copying spans. As of the early 2020s, many state-of-the-art systems on Natural Questions used transformer-based architectures, including BERT-style encoders for retrieval and T5 or GPT-style decoders for generation. The dataset remains widely cited and is often used as a pretraining or evaluation resource for machine learning models in the natural language processing community.
Limitations and Criticisms
Despite its influence, Natural Questions has limitations. The dataset is limited to Wikipedia as the answer source, which may not reflect the full diversity of web content. The annotation process, while thorough, is time-consuming and expensive, limiting the dataset's size compared to automatically generated corpora. Additionally, some researchers have noted that the short answer annotations can be inconsistent, particularly for questions with multiple valid answers or where the answer is implicit. These issues have motivated the creation of complementary datasets, such as TyDi QA, which covers multiple languages, and ELI5, which focuses on long-form explanatory answers.
Legacy and Continued Use
Natural Questions remains a key resource for evaluating open-domain QA systems, especially in the context of retrieval-augmented generation pipelines used in modern artificial intelligence applications. It is also used in academic research to study model robustness, interpretability, and the gap between human and machine performance on real-world queries. The dataset's design principles, particularly its emphasis on authentic user questions, have influenced how researchers approach the problem of answering questions in open-domain settings, and it continues to be a reference point for new benchmarks and models.