SQuAD v2.0 (Stanford Question Answering Dataset version 2.0) is a benchmark dataset for machine reading comprehension and question answering. It extends the original SQuAD 1.1 dataset by adding a substantial set of unanswerable questions, requiring models to not only extract correct answer spans but also recognize when a question cannot be answered from the given context. This design addresses a key limitation of earlier datasets, where models could achieve high scores by guessing answer spans without truly understanding the text.
The dataset was introduced in June 2018 by a team at Stanford University, led by Pranav Rajpurkar, Robin Jia, and Percy Liang. It comprises 100,000 answerable questions and 50,000 unanswerable questions, all derived from the same set of Wikipedia articles used in SQuAD 1.1. The unanswerable questions are crafted by human annotators who paraphrase plausible questions that are not actually supported by the article text, making them challenging to distinguish from answerable ones.
Motivation and Design
The primary motivation for SQuAD v2.0 was to address the problem of "overconfidence" in reading comprehension systems. In SQuAD 1.1, every question had a guaranteed answer span in the context, so models could be trained to always extract something, even if incorrect. This led to systems that performed well on the benchmark but failed in real-world applications where questions often have no answer. By introducing unanswerable questions, SQuAD v2.0 forces models to learn a more robust skill: determining answerability before extracting a span.
The construction process involved crowdworkers who first wrote answerable questions based on a paragraph, then wrote additional questions that are plausible but unanswerable, often by altering entities, numbers, or relationships. This ensures that unanswerable questions are semantically similar to answerable ones, making the task genuinely difficult.
Evaluation Metrics
SQuAD v2.0 uses two primary metrics: Exact Match (EM) and F1 score. However, unlike SQuAD 1.1, the scoring treats unanswerable questions specially. For unanswerable questions, a model receives full credit (EM=1, F1=1) if it predicts "no answer" (an empty span). If it predicts any non-empty span, it gets zero for both metrics. For answerable questions, the standard span-level EM and F1 are computed. The overall score is the average across all questions, giving equal weight to answerable and unanswerable instances.
This scoring scheme incentivizes models to be conservative: they must balance precision in answering answerable questions against the risk of incorrectly answering unanswerable ones. A model that always guesses an answer will score near zero on the unanswerable half, while a model that always abstains will score zero on the answerable half.
Impact on Research
SQuAD v2.0 quickly became a standard benchmark in the NLP community, alongside its predecessor. It pushed the development of more sophisticated models that incorporate answerability prediction as an explicit component. Many early deep learning approaches, such as those based on transformers, were evaluated on this dataset, leading to rapid improvements.
The dataset also influenced the design of later benchmarks like Natural Questions and TriviaQA, which naturally include unanswerable questions. It highlighted the importance of calibration and abstention in question answering, a topic that remains relevant in modern large language models.
Leaderboard and State of the Art
Upon release, the best models achieved EM scores around 60-65%, significantly lower than the 80%+ scores on SQuAD 1.1, reflecting the added difficulty. Over time, models improved substantially. By 2019, systems using BERT and its variants reached EM scores above 80%. As of 2023, the top entries on the official leaderboard achieve EM scores above 90%, often using ensemble methods and external knowledge.
Despite these high scores, researchers note that SQuAD v2.0 still has limitations, such as its reliance on Wikipedia and the fact that unanswerable questions are artificially constructed. Nonetheless, it remains a valuable tool for evaluating reading comprehension robustness.
Related Work and Extensions
SQuAD v2.0 has inspired several follow-up datasets and tasks. For example, squad-shift introduced distribution shifts to test generalization, while other works have explored adversarial perturbations. The concept of answerability has been incorporated into many question answering systems, including those used in production by companies like Google Cloud and Amazon Web Services.
The dataset is freely available for research and is hosted on the official SQuAD website, along with evaluation scripts and a public leaderboard. It continues to be a reference point for comparing new architectures and training techniques in the field of machine comprehension.