Natural Questions Open is a benchmark dataset for open-domain question answering (QA). It was introduced by Google in 2019 as part of the Natural Questions (NQ) dataset, which consists of real anonymized queries issued to the Google search engine. The 'Open' variant removes the context of a single Wikipedia page and requires answering questions directly from a large corpus, such as Wikipedia, making it a standard evaluation set for retrieval-augmented QA systems.
The dataset includes over 300,000 natural questions, each paired with a short answer (a span of text) and a long answer (a Wikipedia paragraph). The Open version, formally known as Natural Questions - Open, simplifies the original task by discarding the page-level supervision and evaluating a system's ability to find the answer from an open document collection. This setup has become a canonical testbed for comparing open-domain QA models, including those based on Transformer (architecture) architectures and large language models.
The development of Natural Questions Open aligns with the broader progress in artificial intelligence, particularly in machine learning and deep learning. The benchmark has been used to measure the performance of systems built on neural networks such as BERT and later generative AI models. Researchers often combine retrieval-augmented generation with models that leverage multi-head attention and positional encoding to reason over retrieved texts.
Task and Evaluation
In the open-domain setting, a system must return an answer, typically a short span, for each question. Evaluation uses the exact match (EM) and F1 score metrics, comparing the predicted answer to the annotated answer, with normalization for punctuation and articles. The total dataset includes dev and test splits, allowing for robust model development and comparison.
Systems typically employ a two-stage approach: first, a retriever selects relevant passages from Wikipedia; second, a reader (often an Encoder-Decoder Architecture or Sequence-to-Sequence (Seq2Seq) model) extracts the answer. The beam search decoding and top-k sampling are common strategies to generate answers.
Benchmark Impact
Natural Questions Open became one of the widely adopted benchmarks for open-domain QA, alongside SQuAD and TriviaQA. It contributed to the rise of retrieval-augmented architectures, such as REALM (2020) and RAG (2020), which combined Transformer (architecture)-based retrieval and generation. In 2021, the Andorra system combine a retriever-reader architecture with a fine-tuned Generative AI model. The dataset has also been used by Google DeepMind and other Artificial intelligence labs to advance question answering capabilities.
The task of open-domain QA is closely related to the construct of the Natural Questions original, which includes a richer annotation with long answers. The openness makes it a challenge for large language model systems, which may have to yield an answer without explicit retrieval.
Recent Developments
As of the early 2020s, systems based on large pre-trained models have achieved strong results on the test set, with EM scores exceeding 50%. For instance, a 2021 model from Google Research achieved an EM of 54.6% on the test set, while an approach combining retrieval with Generative AI reached higher, at 56.4% later. In 2022, notices from AI labs noted the application of reinforcement learning with RLAIF to align text generation. However, exact leaderboard claims may vary over time.
Real-World Applications
The insights from Natural Questions Open search have influenced commercial systems, for example: the retrieval degradation of question dilution and the need for dense retrieval are used by Google Cloud and Amazon Web Services for search. A similar set is often used to evaluate voice assistants and computer-based systems. Even so, the dataset remains as a stay sub-line of QA research.
See Also
- Machine learning
- Deep learning
- Large language model
- Transformer (network architecture)