RefCOCOg is a benchmark dataset in computer vision and natural language processing, designed for the task of referring expression comprehension and generation. It was introduced to address limitations in earlier referring expression datasets, particularly the need for longer, more descriptive, and context-dependent language. The dataset provides a challenging testbed for models that must align visual information with linguistic descriptions, a core capability for applications such as human-robot interaction, image editing, and visual question answering.
The dataset contains 104,560 referring expressions for 54,822 objects across 26,711 images, all sourced from the Microsoft COCO (Common Objects in Context) dataset. A key feature of RefCOCOg is that its expressions are longer and more natural than those in predecessors like RefCOCO, often requiring an understanding of the relationships between multiple objects in a scene. For example, an expression might be "the man in the blue shirt standing next to the red car," which requires grounding both the primary object and its relational context.
Construction and Annotation
RefCOCOg was created by a team of researchers, with the annotation process involving human annotators who were asked to write descriptions that uniquely identify a target object within an image. The annotations were collected through a crowdsourcing platform, and each expression was validated to ensure it was unambiguous. The dataset is split into training, validation, and test sets, with the test set further divided into two subsets based on whether the annotators saw the image during training (known as 'refexp' and 'refexp+'). This split design helps evaluate generalization to unseen annotators and language styles.
Task Formulation
The primary task associated with RefCOCOg is referring expression comprehension, where a model receives an image and a textual description and must localize the described object, typically by predicting a bounding box or a segmentation mask. A related task is referring expression generation, where the model must produce a natural language description for a given object in an image. Both tasks are evaluated using standard metrics: intersection-over-union (IoU) for comprehension and metrics like CIDEr or BLEU for generation.
Impact on Model Development
RefCOCOg has been widely used to train and evaluate deep learning models, particularly those based on Transformer (architecture) architectures and Multi-Head Attention mechanisms. Early approaches used Residual Network (ResNet) backbones for visual feature extraction combined with recurrent neural networks for language encoding. More recent systems leverage Large language models and Vision-language models, often integrating Cross-Attention layers to fuse visual and textual modalities. The dataset's emphasis on longer expressions has pushed the development of models that can reason about spatial relationships, attributes, and context, moving beyond simple object detection.
Relation to Other Datasets
RefCOCOg is part of a family of referring expression datasets, including RefCOCO and RefCOCO+, which were introduced around the same time. While RefCOCO focuses on shorter, more concise expressions, RefCOCO+ restricts expressions to appearance-based descriptions, avoiding location words. RefCOCOg is distinguished by its longer, more conversational descriptions that often require holistic scene understanding. Together, these datasets have become standard benchmarks in the field, with leaderboards tracking state-of-the-art performance on each.
Challenges and Limitations
Despite its utility, RefCOCOg has known limitations. The dataset is derived from MS COCO, which has a bias towards common object categories and everyday scenes, potentially limiting diversity. The expressions, while longer, may still not capture the full complexity of human reference in real-world interactions, such as using gestures or shared knowledge. Additionally, evaluation based on bounding boxes can be coarse, and some expressions may be ambiguous even for humans. Researchers have proposed extensions and alternative datasets to address these gaps, but RefCOCOg remains a foundational resource for studying grounded language understanding.
See Also
- Artificial intelligence
- Machine learning
- Deep learning
- Neural network
- Computer vision (not in list, but related; use MIT CSAIL as a research hub)
- Stanford AI Lab
- BAIR (Berkeley AI Research)
References
The dataset was introduced in the paper "Grounded Language Understanding: Referring Expression Comprehension and Generation" by Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy, presented at the 2016 European Conference on Computer Vision (ECCV).