Sherlock is a large-scale dataset and benchmark for visual commonsense reasoning, introduced in 2020 by researchers at the University of North Carolina at Chapel Hill and the Allen Institute for Artificial Intelligence. The dataset comprises over 103,000 question-answer pairs derived from approximately 21,000 images sourced from the Visual Genome project, with each question designed to require commonsense knowledge beyond simple object recognition. Sherlock was created to address a gap in visual question answering (VQA) benchmarks, which often focus on literal scene descriptions rather than the implicit, everyday knowledge that humans use to interpret images.
The dataset is structured around two complementary tasks: a multiple-choice question answering task and a rationale generation task. In the multiple-choice task, models must select the correct answer from four options, where the correct answer requires inferring unstated properties or relationships, such as predicting that a person holding a surfboard is likely near a beach. The rationale generation task asks models to produce a free-form explanation for why a given answer is correct, encouraging models to articulate the commonsense reasoning behind their predictions. This dual structure makes Sherlock a rigorous test of both recognition and reasoning capabilities.
Construction and Annotation
Sherlock was built using a two-stage crowdsourcing pipeline. First, annotators were shown images from Visual Genome and asked to write questions that could not be answered by simply listing visible objects or attributes, but instead required commonsense inference. For example, a question might ask, "Why is the person wearing a helmet?" when the helmet is visible but the reason (e.g., riding a motorcycle) is not. Second, each question was paired with four answer choices, one correct and three plausible distractors, and annotators also wrote a short rationale explaining the correct answer. The final dataset contains 103,302 questions, with a train/test split of approximately 80/20, and each image is associated with an average of five questions.
To ensure quality, the annotation process included multiple rounds of review and filtering. Questions that were ambiguous, required external factual knowledge (such as specific historical events), or had multiple plausible answers were discarded. The resulting dataset emphasizes everyday, universal commonsense, such as understanding that a person holding an umbrella is likely expecting rain or that a kitchen knife is used for cutting food.
Evaluation and Metrics
Models are evaluated on two primary metrics: accuracy on the multiple-choice task and the quality of generated rationales. For the multiple-choice task, accuracy is simply the percentage of questions answered correctly. For rationale generation, the original paper used BLEU scores, though subsequent work has often employed more robust metrics like ROUGE or human evaluation. The benchmark is designed to be challenging for contemporary models; in the original paper, a strong visual question answering model based on a Residual Network (ResNet) and attention mechanisms achieved an accuracy of around 55%, compared to a human performance of approximately 90% on a sampled subset. This significant gap highlighted the difficulty of visual commonsense reasoning and motivated further research.
Significance and Impact
Sherlock has become a standard benchmark in the fields of Computer vision and Natural language processing, particularly for research at the intersection of vision and language. It has been used to evaluate numerous models, including those based on Transformer (architecture) architectures and Large language models. The dataset's emphasis on reasoning over simple recognition has influenced subsequent benchmarks, such as VCR (Visual Commonsense Reasoning) and OK-VQA, which similarly require external knowledge. Sherlock also contributed to the development of methods for generating explanations in vision-language tasks, an area of growing interest for interpretability and trust in Artificial intelligence systems.
The dataset has been particularly useful for studying the limitations of Deep learning models. For example, analyses have shown that models often rely on statistical regularities in the answer choices rather than true reasoning, and that they struggle with questions requiring spatial or temporal inference. This has spurred research into more robust training techniques, including Curriculum Learning and Data Augmentation, as well as the integration of external knowledge bases.
Limitations and Criticisms
Despite its influence, Sherlock has been criticized on several grounds. The use of BLEU for rationale evaluation is known to correlate poorly with human judgment, leading to misleadingly low scores for valid but differently worded rationales. Additionally, the dataset's reliance on Visual Genome images introduces biases, such as an overrepresentation of common objects and scenes from Western, urban contexts, which can limit the generalizability of models trained on it. Some researchers have also noted that the multiple-choice format can be gamed by models that exploit subtle linguistic cues in the answer options, a problem common to many VQA benchmarks. These limitations have prompted calls for more diverse and adversarially constructed datasets.
Legacy and Future Directions
Sherlock remains a widely cited resource in the vision-language community and is often used as a pretraining or evaluation target for models that aim to combine visual and textual understanding. Its design has inspired follow-up datasets that focus on specific types of commonsense, such as physical reasoning or social interactions. As of the mid-2020s, state-of-the-art models, including those built on Transformer (architecture) architectures and trained on massive multimodal corpora, have achieved significantly higher accuracy on Sherlock, with some exceeding 80%, though human-level performance remains elusive. The dataset continues to serve as a valuable tool for probing the reasoning capabilities of Generative AI systems and for developing methods that make such reasoning more explicit and verifiable.
See Also
- Visual Question Answering
- commonsense-reasoning
- dataset
- Visual Genome
References
The original Sherlock paper was presented at the 2020 Conference on Computer Vision and Pattern Recognition (CVPR) and is available in the CVPR proceedings. The dataset is publicly hosted and can be accessed through the project's official website.