NoCaps (Novel Object Captioning at Scale) is a benchmark dataset and evaluation protocol designed to assess the ability of image captioning systems to describe images containing objects that were not present in their training data. Introduced in 2019 by researchers from the University of Texas at Austin and others, it addresses a critical limitation in standard captioning benchmarks: models often overfit to the limited object vocabulary seen during training and fail to generalize to the long tail of visual concepts. NoCaps provides a controlled setting to measure this generalization capability, using images from the Open Images dataset and a large set of object categories that are disjoint from those in the widely used COCO dataset.
The benchmark consists of over 15,000 images, each paired with multiple human-written reference captions. The images are split into three evaluation subsets based on how many novel objects they contain: in-domain (no novel objects), near-domain (a few novel objects), and open-domain (many novel objects). This tiered structure allows researchers to analyze performance degradation as the novelty and complexity of the visual content increases. The primary evaluation metric is CIDEr, a standard metric for captioning that measures consensus between generated and reference captions, though other metrics like BLEU, METEOR, and SPICE are also reported.
Motivation and Challenges
Traditional captioning datasets like COCO contain only 80 object categories, which is a tiny fraction of the visual world. Models trained on such datasets tend to memorize frequent object co-occurrences and fail when encountering unfamiliar combinations or entirely new objects. NoCaps was created to push the field toward more robust and generalizable captioning systems. The key challenge is that a model must not only recognize novel objects (a visual perception problem) but also generate fluent and contextually appropriate language to describe them, often using words it has never seen paired with those visual inputs during training.
Construction and Annotation
The images in NoCaps are sourced from the Open Images dataset, which contains millions of images with diverse object annotations. The creators selected a subset of images and filtered them to ensure a balanced distribution across the three difficulty tiers. They used a vocabulary of 500 object categories from Open Images, of which 200 are considered novel relative to COCO's 80 categories. Human annotators on Amazon Mechanical Turk wrote five reference captions per image, following guidelines that encouraged natural, descriptive sentences. The annotation process included quality control measures to ensure captions were accurate and varied.
Evaluation Protocol
To evaluate a model on NoCaps, a system generates a caption for each image in the test set. The generated captions are compared against the human references using CIDEr, which computes the average cosine similarity between the candidate and reference n-gram vectors, weighted by term frequency-inverse document frequency. The benchmark provides a public validation set for development and a hidden test set for final evaluation, with results reported on the official leaderboard. This setup encourages fair comparison across different approaches, from classical Sequence-to-Sequence (Seq2Seq) models to modern Transformer (architecture)-based architectures.
Impact and Usage
NoCaps has become a standard benchmark in the Machine learning and computer vision communities, particularly for evaluating the generalization capabilities of image captioning models. It has been used to demonstrate the effectiveness of techniques such as Data Augmentation, Curriculum Learning, and the incorporation of external knowledge sources like Large language models. For instance, some approaches use a pretrained object detector to identify novel objects and then condition a language model on the detected labels, while others employ Cross-Attention mechanisms to align visual and textual features more effectively. The benchmark has also highlighted the importance of scaling training data and model capacity, as larger Neural networks trained on massive datasets tend to perform better on the open-domain subset.
Limitations and Criticisms
Despite its influence, NoCaps has some limitations. The benchmark relies on a fixed set of novel objects, which may not fully capture the open-ended nature of real-world novelty. The evaluation metric CIDEr is known to be sensitive to style and length, and it may not fully reflect semantic correctness. Additionally, the human references, while high quality, are limited in number and may not cover all valid ways to describe an image. Some researchers have argued that the benchmark's focus on object-level novelty overshadows other aspects of generalization, such as novel relationships or attributes. Nevertheless, NoCaps remains a valuable tool for tracking progress in robust image captioning.
Future Directions
As the field moves toward more capable Generative AI systems, NoCaps is likely to be extended or complemented by benchmarks that test compositional generalization, zero-shot learning, and interaction with user instructions. The integration of Large language models as decoders, often combined with vision-language models, has already shown promise in handling novel objects by leveraging broad world knowledge. Future work may also explore dynamic benchmarks that adapt to a model's training data, ensuring that novelty is always genuinely novel. NoCaps has set a precedent for rigorous evaluation of generalization, and its principles are likely to influence the design of future benchmarks in multimodal AI.