VSWinoground is a benchmark dataset designed to evaluate the visual spatial reasoning capabilities of vision-language models. It was introduced in 2023 by a team of researchers including Tristan Thrush, Ryan Jiang, and colleagues at the University of Toronto and other institutions. The dataset builds on the Winoground benchmark, which tests compositional understanding in text-image pairs, but focuses specifically on spatial relationships such as 'above', 'below', 'left of', and 'right of'.
The benchmark consists of 400 image-caption pairs, each with two captions that describe the same image but differ in the spatial arrangement of objects. For example, one caption might read 'the cat is above the dog' while the other reads 'the dog is above the cat'. Models are tasked with selecting the correct caption for a given image, and vice versa. The dataset is publicly available and has been used in several research studies to assess model performance.
Design and Evaluation
VSWinoground is constructed using a semi-automated pipeline that generates images from synthetic 3D scenes, ensuring precise control over spatial layouts. Each image is rendered with two objects placed in a specific spatial relation, and captions are generated using templates. The benchmark includes both a 'text-to-image' and 'image-to-text' retrieval task, and models are scored on accuracy for each task, as well as a combined 'group' score that requires correct performance on both tasks for a given pair.
Initial evaluations on several state-of-the-art models, including CLIP and ViLT, showed that they perform only slightly better than chance (around 50% accuracy) on the image-to-text task, and significantly worse on the text-to-image task. For instance, CLIP achieved approximately 52% accuracy on image-to-text and 48% on text-to-image, while ViLT scored around 50% on both. These results highlight the difficulty of visual spatial reasoning for current models.
Related Benchmarks
VSWinoground is part of a broader family of Winoground-style benchmarks that test compositional and spatial understanding. The original Winoground dataset, introduced in 2022 by researchers at Google DeepMind, contains 400 image-caption pairs with more general compositional variations. Other related benchmarks include VSR (Visual Spatial Reasoning), which uses real images and focuses on spatial relations, and the more recent Winoground-V2, which expands the original dataset. These benchmarks are often used together to evaluate the robustness of vision-language models.
Limitations and Criticisms
One limitation of VSWinoground is its relatively small size (400 pairs), which can lead to high variance in evaluation results. Additionally, because the images are synthetic, they may not fully capture the complexity of real-world spatial reasoning. Some researchers have argued that the benchmark's focus on simple spatial prepositions may not reflect the full range of spatial reasoning required in practical applications. Nonetheless, VSWinoground remains a widely cited resource for probing model capabilities.
Impact and Future Directions
VSWinoground has been used in several studies to analyze the limitations of vision-language models, particularly in the context of large language models and multimodal learning. It has also inspired follow-up work on improving spatial reasoning, such as incorporating explicit spatial encodings or using synthetic data augmentation. As of 2025, no model has achieved human-level performance on the benchmark, and it continues to serve as a challenging testbed for the AI research community.
See Also
- Winoground (original benchmark)
- Visual Spatial Reasoning
- Vision-Language Models