BoolQ

BoolQ is a natural language processing benchmark dataset for boolean question answering, where models answer yes/no questions based on Wikipedia passages. It was introduced in 2019 by researchers at Carnegie Mellon University and Google AI Language.

BoolQ (Boolean Questions) is a dataset for natural language understanding that evaluates a machine's ability to answer yes/no questions using a given passage of text. It was introduced in 2019 by Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova, from Carnegie Mellon University and Google AI Language. The dataset contains 15,942 questions, each paired with a short passage from Wikipedia and a binary answer (yes or no). The questions are naturally occurring, generated by human annotators who were asked to formulate questions based on the passage content, making them more realistic and challenging than synthetic or template-based queries.

BoolQ is part of the SuperGLUE benchmark, a suite of tasks designed to evaluate general-purpose language understanding models. In SuperGLUE, BoolQ serves as a test of reading comprehension and logical reasoning, requiring models to identify relevant information in a passage and make a simple but often nuanced decision. The dataset's difficulty arises from the fact that the answer is not always explicitly stated; models must infer the answer from the context, handle negation, and deal with questions that may have multiple plausible interpretations.

Construction and Annotation

The BoolQ dataset was constructed by first selecting passages from Wikipedia articles on a diverse range of topics, including history, science, and culture. Annotators were then shown a passage and asked to write a yes/no question that could be answered using the information in the text. They were instructed to avoid questions that were too easy or too dependent on external knowledge, ensuring that the questions required genuine comprehension of the passage. Each question was then answered by a separate group of annotators, with the majority vote determining the final label. The dataset was split into 9,427 training examples, 3,270 development examples, and 3,245 test examples, with the test set held out for official evaluation.

Task and Evaluation

In the BoolQ task, a model is given a passage and a question, and it must output either "yes" or "no". The primary evaluation metric is accuracy, which is the percentage of correctly answered questions. Random guessing would achieve 50% accuracy, but human performance on the dataset is estimated at around 90%, indicating the need for sophisticated reasoning. Early models, such as those based on Transformer (architecture) architectures, struggled with the task, with the first published results achieving around 60-70% accuracy. The introduction of Large language models and Neural network approaches, particularly those fine-tuned on BoolQ, significantly improved performance, with state-of-the-art models exceeding 90% accuracy by 2021.

Challenges and Significance

BoolQ presents several challenges for Machine learning systems. The questions often contain presuppositions or require the model to handle complex linguistic phenomena such as coreference resolution, temporal reasoning, and world knowledge. For example, a question like "Was the Battle of Waterloo fought in the 19th century?" requires the model to locate the date in the passage and map it to the correct century. Additionally, the passages are not always directly relevant to the question, requiring the model to filter out irrelevant information. BoolQ has been influential in driving research on reading comprehension and has been used as a benchmark for evaluating Artificial intelligence systems, including those developed by OpenAI, Anthropic, and Google DeepMind.

Relation to Other Benchmarks

BoolQ is one of several datasets in the SuperGLUE benchmark, which also includes tasks like MultiRC (multi-sentence reading comprehension), ReCoRD (reading comprehension with commonsense reasoning), and COPA (causal reasoning). Unlike some other benchmarks that focus on extractive question answering, where the answer is a span of text, BoolQ requires a binary decision, making it a more direct test of understanding. It is also related to the earlier SQuAD (Stanford Question Answering Dataset) but differs in that SQuAD questions are typically answerable with a span, whereas BoolQ questions are designed to be more open-ended and require inference.

Impact on Model Development

BoolQ has been used extensively in the development and evaluation of Deep learning models. It was a key benchmark in the development of Transformer (architecture)-based models like BERT and RoBERTa, which showed significant gains over previous approaches. The dataset has also been used to study the limitations of models, such as their tendency to rely on surface-level cues or to be misled by adversarial examples. As of 2024, BoolQ remains a standard evaluation tool in natural language processing research, and it is often included in the training and testing of new Large language models, contributing to the broader field of Generative AI.