Open Images is a large-scale dataset created by Google for computer vision research. It contains millions of images annotated with a variety of labels and metadata, designed to train and evaluate Machine learning models. The dataset is notable for its scale and diversity, spanning a wide range of everyday objects and scenes, with annotations that include image-level labels, object bounding boxes, and visual relationship triplets.
The first version of Open Images was released in 2016, containing approximately 9 million images with image-level labels across nearly 6,000 categories. Later versions expanded the dataset to include more than 30 million images, with over 15 million bounding boxes demarcating objects in more than 600 classes. The annotations were produced through a combination of automated methods and human verification, reflecting Google's investment in large-scale data infrastructure for advancing Artificial intelligence capabilities.
Dataset Structure
Open Images is structured to support multiple computer vision tasks. The core components are the image collection, image-level labels indicating the presence of objects or concepts, and localized annotations such as bounding boxes and segmentation masks. For each image, the dataset provides a unique identifier, the original URL, and metadata such as the predicted confidence of labels. The bounding box annotations are supplied in a standardized format, listing the coordinates relative to image dimensions, along with a label for the object class.
Additionally, the dataset includes a set of visual relationship annotations that describe interactions between objects, such as "person riding horse" or "dog chasing ball." These relationships are formatted as subject-predicate-object triplets)Skip to main content. This richer annotation layer enables research into relational reasoning and scene understanding, which are active areas within visual recognition and Deep learning research.
Annotation Methodology
The creation of Open Images relied on a semi-automatic pipeline that combined machine-generated candidates with human curation. Google's internal systems first generated noisy label candidates using existing image classifiers and web-scale data mining. These candidates were then presented to human annotators through crowdsourcing platforms, where workers verified or corrected the labels and refined bounding boxes. This hybrid approach allowed the dataset to scale to millions of images while maintaining a level of quality suitable for training modern neural networks.
For bounding boxes, annotators were instructed to draw the tightest possible box around each visible instance of a target class. The final annotations were validated through a series of quality checks, including an agreement analysis between multiple annotators on a subset of images. As a result, the dataset has become a benchmark for both single-label and multi-label classification, as well as object detection.
Impact on Research
Open Images has been widely used by both academic and industrial research groups. It served as the foundation for several public benchmark challenges, including the Open Images Challenge hosted on Kaggle, which ran in 2018 and 2019. These challenges attracted hundreds of participating teams and pushed the state of the art in object detection and visual relationship detection. The dataset has been cited in thousands of research papers, influencing approaches to large-scale Neural network training and transfer learning.
The availability of a large and diverse labeled dataset also supported the development of Generative AI systems that require grounding in real-world visual concepts. Models pretrained on Open Images have been used for downstream tasks such as image captioning, visual question answering, and autonomous driving perception. The dataset complements other resources like ImageNet and COCO, offering a broader set of classes and a more realistic distribution of everyday imagery.
Versions and Availability
Google released several versions of Open Images under a Creative Commons license, making it freely available for research and commercial use. Version 4, introduced in 2018, added the visual relationship annotations and expanded the number of images to over 9 million with 30 million box-level annotations. Version 5, released in 2019, increased the number of bounding box classes to 600 and added segmentation masks for a subset of images. The dataset is accessible through Google's storage infrastructure and can also be explored via an interactive visualization tool on the official website.
All files are provided in CSV format, with separate files for each annotation type. The dataset has a companion paper first presented at the 2017 Conference on Computer Vision and Pattern Recognition, detailing its construction and statistics. The project has been maintained by the Google Research team, with updates driven by community feedback and internal research needs. As of 2025, the dataset remains a standard reference point for large-scale visual recognition benchmarks.
Limitations and Future Directions
Despite its scale, Open Images has inherent limitations. The image-level labels are based on machine-generated predictions and can contain noise, and the class distribution is biased toward common objects found in web imagery. The annotations primarily focus on object presence and coarse relationships, lacking the fine-grained attributes or detailed scene graphs available in some smaller datasets. Researchers often supplement Open Images with other sources or apply domain adaptation techniques when working on specialized tasks.
Future directions for the dataset may include more extensive video annotations, richer relationship types, or incorporation of multi-modal data such as text descriptions. The success of Open Images has inspired similar large-scale data collection efforts, including those from Amazon AI and other cloud providers, reflecting a broader trend in making massive annotated corpora accessible to accelerate progress in Artificial intelligence.