NarrativeQA is a benchmark dataset designed for reading comprehension research, focusing on the ability of artificial intelligence systems to understand and answer questions about long-form narrative texts. Introduced in 2018 by researchers including Tomáš Kočiský and colleagues, the dataset addresses a gap in earlier comprehension tasks that typically relied on short passages or synthetic documents. NarrativeQA requires models to process entire books and movie scripts, making it a demanding test for Machine learning and Deep learning systems.
The dataset comprises 1,567 documents, including 360 books and 1,207 movie scripts, sourced from Project Gutenberg and the Internet Movie Script Database (IMSDb). For each document, human annotators generated 10 to 20 questions and answers, yielding a total of 46,765 question-answer pairs. The questions are open-ended and often require reasoning across multiple chapters or scenes, rather than simple fact retrieval. Answers are provided in two forms: free-form text and a multiple-choice version, allowing for both generative and discriminative evaluation.
Construction and Annotation
The creation of NarrativeQA involved a two-stage process. First, annotators read each full narrative and wrote questions that tested comprehension of plot, character motivations, and causal events. Second, they provided concise answers, typically one to five words, based on the text. The annotation protocol emphasized questions that could not be answered by looking at a single sentence, encouraging models to build a global understanding of the story. The dataset was split into training, validation, and test sets, with the test set containing 10,611 questions. This design contrasts with earlier benchmarks like SQuAD, which focused on Wikipedia articles and local context.
Evaluation and Metrics
NarrativeQA is evaluated using standard natural language processing metrics: BLEU-1, BLEU-4, ROUGE-L, and METEOR. These metrics compare generated answers against reference answers, measuring n-gram overlap and longest common subsequence. Because answers are short, BLEU-1 and ROUGE-L are often the primary indicators of performance. The multiple-choice version is scored by accuracy, where models must select the correct answer from a set of options. Early baselines, including sequence-to-sequence models and memory-augmented networks, achieved low scores, highlighting the difficulty of the task. As of 2025, the best systems, often based on Large language model architectures, still do not reach human-level performance, with human accuracy on the multiple-choice task around 95%.
Challenges and Research Impact
NarrativeQA has driven research in several areas of Artificial intelligence. The primary challenge is the length of input documents, which can exceed 100,000 tokens, far beyond the context window of early Transformer (architecture) models. This has motivated work on hierarchical attention mechanisms, retrieval-augmented generation, and efficient memory architectures. The dataset also tests the ability to handle narrative structure, such as temporal ordering and character coreference, which are less prominent in fact-based QA datasets. Researchers have used NarrativeQA to evaluate Neural network models' capacity for long-range dependency modeling, and it has become a standard benchmark in the Natural language processing community, alongside datasets like RACE and HotpotQA.
Relation to Modern Language Models
With the advent of Large language models like those developed by OpenAI, Anthropic, and Google DeepMind, NarrativeQA has been used to probe comprehension of extended texts. Modern models, when given the full narrative in chunks or with retrieval, can achieve significantly higher scores than early baselines, but still struggle with questions requiring global inference or subtle plot details. The dataset has also been incorporated into evaluation suites for Generative AI systems, where it serves as a stress test for long-context handling. Some models now support context windows of 128,000 tokens or more, enabling direct processing of entire books, yet performance on NarrativeQA remains below human levels, indicating that raw context length is not sufficient for deep narrative understanding.
Limitations and Criticisms
Critics have noted that NarrativeQA's free-form answers are short and often extractive, which may understate the complexity of narrative reasoning. The multiple-choice version can be gamed by models exploiting statistical regularities in answer options. Additionally, the dataset's focus on Western literature and film scripts limits its cultural diversity. Despite these issues, NarrativeQA remains a valuable resource for studying comprehension at scale, and it has inspired follow-up datasets like NarrativeQA-2 and long-document QA benchmarks. As of 2025, it continues to be cited in research on Machine learning and Deep learning for its unique combination of length and narrative complexity.