# OpenBookQA

OpenBookQA is a question-answering benchmark designed to test reasoning with open-book knowledge, requiring models to combine provided science facts with common knowledge to answer multiple-choice questions.

OpenBookQA is a question-answering benchmark introduced in 2018 to evaluate a machine's ability to perform multi-step reasoning using both explicit scientific facts and implicit common knowledge. The benchmark was created by a team of researchers including Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal at the Allen Institute for Artificial Intelligence (AI2). It consists of 5,957 multiple-choice questions at a grade-school science level, each paired with a small set of core science facts from an "open book" of 1,326 elementary science facts. The task requires models to select the correct answer from four options, but crucially, the provided facts alone are insufficient; models must combine them with broader world knowledge and logical reasoning to arrive at the correct answer.

The benchmark was designed to address limitations in earlier question-answering datasets, which often allowed models to succeed through shallow pattern matching or memorization. In OpenBookQA, the questions are crafted so that the correct answer is not directly stated in the provided facts, forcing systems to reason about how the facts apply to the question. This makes the benchmark a more rigorous test of reasoning ability, and it has become a standard evaluation tool in the field of [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) and [machine-learning](https://www.wikiprompt.org/wiki/machine-learning).

## Design and Structure

OpenBookQA contains 5,957 questions, split into training (4,957), development (500), and test (500) sets. Each question is a multiple-choice item with four answer options. The benchmark includes an "open book" of 1,326 science facts, which are short, declarative statements such as "Plants need sunlight to make food" or "Water boils at 100 degrees Celsius." These facts are drawn from elementary science textbooks and cover topics like biology, physics, and chemistry.

The questions are designed so that a human with basic science knowledge can answer them easily, but they require combining the given facts with common sense. For example, a question might ask: "Which of the following is most likely to happen if a plant is placed in a dark room?" The provided facts might state that "Plants need light to grow," but the correct answer requires inferring that the plant will not grow well, which is not explicitly stated. This design forces models to go beyond simple retrieval and engage in reasoning.

The benchmark also includes a set of "crowdsourced" questions that were validated by human annotators to ensure they are unambiguous and answerable by non-experts. The creators used a multi-stage process to generate and filter questions, ensuring high quality and difficulty.

## Evaluation and Performance

OpenBookQA has become a widely used benchmark for evaluating [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)s and other AI systems. Early models, including those based on [transformer](https://www.wikiprompt.org/wiki/transformer) architectures, performed poorly, with accuracy often below 60%, compared to human performance of around 92%. The gap highlighted the challenge of combining explicit knowledge with implicit reasoning.

Over time, performance improved significantly with advances in [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) and the development of larger models. By 2021, models like GPT-3 achieved accuracy around 80% on the benchmark, and by 2023, state-of-the-art systems, including those from [openai](https://www.wikiprompt.org/wiki/openai) and [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind), were approaching or exceeding 90% accuracy. These improvements were driven by scaling up model size, better training data, and techniques like chain-of-thought prompting, which encourages models to reason step-by-step.

Despite these gains, OpenBookQA remains a challenging benchmark because it tests reasoning that is not easily captured by simple pattern matching. Even as models achieve high scores, researchers note that they may still rely on statistical regularities rather than true understanding, making the benchmark a useful tool for probing the limits of current AI systems.

## Influence and Legacy

OpenBookQA has influenced the design of subsequent benchmarks, such as ARC (AI2 Reasoning Challenge) and CommonsenseQA, which similarly emphasize reasoning over retrieval. It has also spurred research into knowledge-augmented models, which incorporate external knowledge bases or retrieval mechanisms to improve reasoning. The benchmark is often used in conjunction with other evaluations to assess a model's general intelligence, and it appears in leaderboards from organizations like [stanford-ai-lab](https://www.wikiprompt.org/wiki/stanford-ai-lab) and [berkeley-ai-research](https://www.wikiprompt.org/wiki/berkeley-ai-research).

The benchmark's focus on open-book reasoning has also contributed to discussions about the difference between memorization and understanding in AI. It has been cited in hundreds of research papers and is a standard component of many model evaluation suites, including those used by [amazon-web-services](https://www.wikiprompt.org/wiki/amazon-web-services) and [google-cloud](https://www.wikiprompt.org/wiki/google-cloud) in their AI services.

## Limitations and Criticisms

Some researchers have criticized OpenBookQA for being too narrow, as it focuses on grade-school science and may not generalize to more complex reasoning tasks. Others have noted that the benchmark's multiple-choice format can be gamed by models that exploit statistical biases in the answer options, such as selecting the longest answer or the one with the most overlap with the question. To mitigate this, the creators introduced a "partial credit" scoring scheme, but it has not fully addressed these concerns.

Additionally, the benchmark's reliance on a fixed set of facts means that models trained on large corpora may have already seen similar questions or facts, leading to potential data contamination. This has led some researchers to call for more dynamic benchmarks that are updated over time to avoid saturation.

Despite these limitations, OpenBookQA remains a valuable tool for evaluating reasoning abilities and has played a key role in advancing research in [neural-network](https://www.wikiprompt.org/wiki/neural-network) and [generative-ai](https://www.wikiprompt.org/wiki/generative-ai) systems. Its design principles have informed the development of more comprehensive benchmarks that aim to capture a broader range of cognitive skills.

## See Also

- [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence)
- [machine-learning](https://www.wikiprompt.org/wiki/machine-learning)
- [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)
- [transformer](https://www.wikiprompt.org/wiki/transformer)
- [openai](https://www.wikiprompt.org/wiki/openai)
- [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind)

---
Source: https://www.wikiprompt.org/wiki/openbookqa
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-07T21:34:05.887303+00:00