CLIPScore is a metric for evaluating the quality of image captions generated by artificial intelligence systems. It is a reference-free metric, meaning it does not require ground-truth human-written captions for comparison. Instead, it measures the semantic alignment between an image and a candidate caption directly using a pre-trained neural network called CLIP (Contrastive Language-Image Pre-training).
The metric was introduced in 2021 by researchers at the University of Copenhagen and the Technical University of Denmark, led by Jack Hessel. It was presented at the Conference on Empirical Methods in Natural Language Processing (EMNLP) that year. The core idea is to leverage the joint embedding space learned by CLIP, where images and their corresponding text descriptions are mapped to vectors that are close to each other. CLIPScore computes the cosine similarity between the CLIP embedding of the image and the CLIP embedding of the caption, optionally weighted by a penalty for overly short captions.
Calculation and Variants
The base CLIPScore is defined as the cosine similarity between the normalized embeddings of the image and the caption, multiplied by a factor that accounts for caption length. Specifically, the formula is: CLIPScore(I, c) = w * max(cos(I, c), 0), where w is a weighting term that is 2.5 if the caption has fewer than a certain number of tokens (typically 10) and 1 otherwise. This penalty discourages overly terse captions that might trivially match the image.
A variant called RefCLIPScore incorporates reference captions by averaging the CLIPScore between the image and each reference, then combining this with the base CLIPScore using a harmonic mean. This variant is used when human references are available, but the primary advantage of CLIPScore is its ability to work without them.
Advantages Over Traditional Metrics
Traditional metrics for image captioning, such as BLEU, ROUGE, METEOR, and CIDEr, rely on n-gram overlap between the generated caption and reference captions. These metrics are reference-based and often fail to capture semantic similarity, penalizing paraphrases that are linguistically different but semantically equivalent. CLIPScore addresses this by operating in a high-dimensional semantic space learned from a large corpus of image-text pairs. This allows it to recognize that "a dog playing fetch" and "a canine retrieving a ball" describe the same scene, even with no shared words.
Empirical studies have shown that CLIPScore correlates better with human judgments of caption quality than traditional metrics across several benchmark datasets, including Flickr8k, Flickr30k, and MS COCO. Its reference-free nature also makes it particularly useful for evaluating captions in new domains or for zero-shot captioning models where reference captions are unavailable.
Applications and Limitations
CLIPScore has been widely adopted in the evaluation of generative-ai models for image captioning and text-to-image generation. It is often used alongside other metrics to provide a holistic assessment. For instance, in text-to-image models, a variant called CLIPScore is used to measure the alignment between a generated image and a text prompt, helping researchers tune model parameters.
Despite its strengths, CLIPScore has known limitations. It is sensitive to the choice of CLIP model checkpoint, as different checkpoints may produce different scores. It can also be biased towards captions that mention common objects or attributes, and it may not fully capture fine-grained details or compositional reasoning. Additionally, it does not directly measure grammatical correctness or fluency, so it is often paired with language-model-based fluency metrics.
Relationship to Other AI Metrics
CLIPScore is part of a broader trend in machine-learning evaluation that moves away from n-gram-based metrics towards learned, embedding-based metrics. It shares conceptual similarities with metrics like BERTScore for text summarization and translation, which use embeddings from large-language-model transformers. However, CLIPScore is unique in that it operates cross-modally, bridging vision and language.
The development of CLIPScore was enabled by the release of OpenAI's CLIP model in early 2021, which itself is a transformer-based neural network trained on 400 million image-text pairs. The success of CLIPScore has inspired subsequent work on reference-free evaluation in other multimodal tasks, such as video captioning and visual question answering.