# VCR

Visual Commonsense Reasoning (VCR) is an AI benchmark and task requiring models to answer questions about images using commonsense knowledge, involving visual understanding, reasoning, and explanation generation.

Visual Commonsense Reasoning (VCR) is a research benchmark and task in artificial intelligence that evaluates a system's ability to understand images and answer questions about them using commonsense knowledge. Introduced in 2019, VCR requires models to not only identify objects and actions in a scene but also infer motivations, mental states, and plausible future events. It is designed to test higher-level cognition beyond basic visual recognition, bridging the gap between [computer-vision](https://www.wikiprompt.org/wiki/computer-vision) and [natural-language-processing](https://www.wikiprompt.org/wiki/natural-language-processing).

The VCR dataset consists of over 290,000 multiple-choice questions derived from 110,000 movie scenes, each paired with a question, four answer options, and four rationale options. A model must select the correct answer and then justify its choice by selecting the correct rationale. This two-stage structure forces systems to produce explanations, making the benchmark more challenging than traditional visual question answering (VQA). The task is split into two sub-tasks: Query-Answer (QA) and Answer-Rationale (AR), with the overall performance measured by the VCR score, which is the product of the accuracy on both sub-tasks.

## Dataset and Annotation

The VCR dataset was created by researchers at the University of North Carolina at Chapel Hill and the Allen Institute for Artificial Intelligence. Annotations were crowdsourced using Amazon Mechanical Turk, with workers asked to write questions that require commonsense reasoning, such as "Why is the person running?" or "What will happen next?" Each question is grounded in a specific image from a movie, providing rich contextual cues like social interactions, emotional expressions, and physical causality. The dataset includes 110,000 unique movie clips, each with an average of 2.6 questions, resulting in a diverse set of scenarios that are not biased toward object recognition alone.

## Task Formulation

Formally, VCR is defined as a multiple-choice task. Given an image I, a question Q, and a set of four answer candidates A = {a1, a2, a3, a4}, the model must predict the correct answer a*. Then, given the same image, question, and the correct answer, the model must select the correct rationale R from a set of four rationale candidates. The final VCR score is computed as the accuracy of selecting both the correct answer and the correct rationale, i.e., the joint probability P(a* | I, Q) * P(r* | I, Q, a*). This formulation encourages models to produce coherent explanations rather than just guessing the answer.

## Model Approaches

Early VCR systems relied on [visual-question-answering](https://www.wikiprompt.org/wiki/visual-question-answering) architectures that combined convolutional neural networks (CNNs) for image feature extraction with recurrent neural networks (RNNs) or [transformer](https://www.wikiprompt.org/wiki/transformer)-based encoders for text. A common baseline was the Visual Commonsense Reasoning with Transformer (VCRT) model, which used a [multi-head-attention](https://www.wikiprompt.org/wiki/multi-head-attention) mechanism to fuse visual and textual representations. Later, [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)-based approaches, such as fine-tuning [BERT](https://www.wikiprompt.org/wiki/bert) or GPT variants, were adapted to handle the multimodal input. These models often employ a two-stage pipeline: first, they encode the image and question jointly, then they use a classification head to select the answer and rationale. More recent work has explored end-to-end training with [cross-attention](https://www.wikiprompt.org/wiki/cross-attention) layers, allowing the model to attend to relevant image regions while reasoning about the question.

## Challenges and Limitations

Despite progress, VCR remains a difficult benchmark. Models often struggle with questions that require deep social reasoning, such as inferring a character's intention or emotional state. The dataset also contains biases, as some questions can be answered by statistical regularities in the text rather than visual evidence. For instance, a model might learn to associate certain verbs with typical objects without truly understanding the scene. Additionally, the rationale selection task is particularly challenging because it requires the model to generate a plausible explanation that is not just a paraphrase of the answer. As of 2025, the best-performing models on VCR achieve around 80% accuracy on the QA sub-task and 75% on the AR sub-task, but the joint VCR score remains below 60%, indicating significant room for improvement.

## Impact and Related Work

VCR has influenced subsequent benchmarks such as Visual Commonsense Reasoning in the Wild (VCR-W) and has been used as a testbed for evaluating commonsense-reasoning capabilities in AI systems. It has also spurred research in [explainable-ai](https://www.wikiprompt.org/wiki/explainable-ai), as the rationale selection component forces models to provide interpretable justifications. The dataset is publicly available and has been widely adopted in the research community, with over 1,000 citations as of 2025. VCR is often compared with other visual reasoning tasks like visual-entailment and [visual-question-answering](https://www.wikiprompt.org/wiki/visual-question-answering), but its emphasis on explanations sets it apart. The benchmark is maintained by the Allen Institute for AI and is regularly used in academic competitions and workshops.

## Future Directions

Future work on VCR may involve integrating more sophisticated [neural-network](https://www.wikiprompt.org/wiki/neural-network) architectures, such as vision-language-models that are pre-trained on large-scale image-text pairs. There is also interest in using VCR to evaluate the reasoning abilities of [generative-ai](https://www.wikiprompt.org/wiki/generative-ai) systems, where models must generate free-form rationales instead of selecting from a fixed set. Additionally, researchers are exploring ways to reduce dataset bias by introducing adversarial examples or counterfactual questions. As [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) continues to advance, VCR serves as a critical benchmark for measuring progress toward human-level visual commonsense reasoning.

---
Source: https://www.wikiprompt.org/wiki/vcr
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-12T16:27:14.05527+00:00