# Flickr30k Entities Flickr30k Entities is a phrase grounding dataset extending Flickr30k with 244k coreference chains and 31,783 bounding boxes linking noun phrases to image regions, used for evaluating visual grounding and multimodal models. Flickr30k Entities is a dataset for phrase grounding, the task of linking natural language phrases in captions to the corresponding regions in images. It was created by extending the Flickr30k image captioning dataset with detailed annotations that connect textual mentions of entities to their visual locations. The dataset is widely used in computer vision and multimodal machine learning research to train and evaluate models that must understand the relationship between language and visual content. The dataset contains 31,783 bounding boxes that localize the visual referents of noun phrases within images. It also provides 244k coreference chains, which group different mentions of the same entity across multiple captions for a single image. This structure allows researchers to study not only direct phrase-region alignment but also cross-sentence coreference resolution in a multimodal context. The annotations cover a broad vocabulary of everyday objects, people, and animals, making the dataset a realistic benchmark for grounded language understanding. ## Annotation Structure Each image in Flickr30k Entities is associated with five independent human-written captions. Annotators identified noun phrases within these captions and linked each phrase to a bounding box in the image when a visual referent existed. When the same entity was mentioned multiple times, either within a caption or across the five captions, the mentions were grouped into a coreference chain. This design enables the evaluation of models on both phrase localization and entity-level reasoning, where a model must aggregate evidence from multiple textual references. The bounding boxes are axis-aligned rectangles that tightly enclose the referenced object. The dataset does not provide segmentation masks, focusing instead on coarse localization sufficient for grounding tasks. The coreference chains are essential for tasks that require a model to track an entity across different descriptions, such as visual question answering or image retrieval based on entity state. ## Construction and Statistics The dataset was built on top of the original Flickr30k corpus, which consists of 31,783 images collected from the photo-sharing website Flickr. The images depict a wide variety of scenes, including people engaged in activities, animals, and everyday objects. The entity annotations were produced through a crowdsourcing pipeline, with quality control measures to ensure consistency. The final dataset includes 244k coreference chains and 31,783 bounding boxes, with an average of roughly one bounding box per image, though many images contain multiple entities. A notable feature is the inclusion of phrases that refer to entities not visually present in the image. These are marked as such, allowing models to learn to distinguish between grounded and ungrounded references. This aspect is important for real-world applications where captions may mention objects outside the frame. ## Usage in Research Flickr30k Entities has become a standard benchmark for phrase grounding and referring expression comprehension. Models are typically evaluated on their ability to predict the correct bounding box for a given noun phrase. The dataset has been used to train and test a variety of architectures, including those based on [transformer](https://www.wikiprompt.org/wiki/transformer) models and [neural-network](https://www.wikiprompt.org/wiki/neural-network) approaches. It is also a common component in the evaluation of [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) based vision-language systems, which often use the dataset to measure fine-grained visual understanding. Researchers have used the dataset to develop methods for weakly supervised grounding, where models learn from image-caption pairs without explicit bounding box supervision, and for fully supervised grounding. The coreference annotations have also enabled work on multimodal coreference resolution, a task that combines linguistic coreference with visual evidence. The dataset complements other resources like [visual-genome](https://www.wikiprompt.org/wiki/visual-genome) and referitgame, each offering different annotation granularities and challenges. ## Limitations and Extensions The dataset inherits biases from the original Flickr30k collection, which tends to feature Western, consumer-oriented photography. The captions are relatively short and describe salient objects, which may not reflect the complexity of more specialized domains. The bounding boxes are coarse and do not capture occlusions or part-level details. Despite these limitations, the dataset remains a valuable resource because of its scale and the richness of its coreference annotations. Several extensions have been proposed, including augmenting the dataset with additional attributes or using it to generate synthetic training data for other tasks. The dataset has also been incorporated into larger benchmarks that combine multiple grounding datasets to test generalization across domains. As of the early 2020s, it continues to be cited in hundreds of papers in computer vision and natural language processing. ## See Also - phrase-grounding - [visual-genome](https://www.wikiprompt.org/wiki/visual-genome) - referitgame - image-captioning --- Source: https://www.wikiprompt.org/wiki/flickr30k-entities License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-13T03:53:36.378019+00:00