TextCaps is a large-scale dataset and benchmark for image captioning that specifically focuses on generating captions that incorporate text appearing within images. Unlike traditional image captioning tasks that describe objects and scenes, TextCaps requires models to read and reason about the text present in the image, such as street signs, product labels, or book covers, and integrate that information into a coherent natural language description. The dataset was introduced in 2020 by researchers from Google and the University of California, Berkeley, and has become a standard evaluation for vision-language models that aim to bridge the gap between visual perception and optical character recognition (OCR).
The core challenge of TextCaps is that it combines two distinct capabilities: visual understanding and text recognition. A model must not only identify objects and their relationships but also accurately transcribe the text in the image and decide which parts of that text are relevant to the overall scene description. For example, an image of a coffee shop might require the caption to mention the name on the storefront, the type of drink on the menu board, or the text on a poster in the background. This requires a deep integration of Computer vision techniques with Natural language processing and Optical character recognition (OCR) systems.
Dataset Composition and Annotations
The TextCaps dataset contains 28,000 images, each paired with multiple human-written captions, totaling over 145,000 captions. The images are sourced from the text-in-images domain, which includes a wide variety of real-world scenes, such as street views, indoor settings, and product shots. Each image has an average of five captions, and the captions are designed to be descriptive and natural, often including specific text strings that appear in the image. The dataset is split into training, validation, and test sets, with the test set used for the official benchmark evaluation. Annotations were collected via a crowdsourcing platform, with careful instructions to ensure that captions mention text only when it is relevant to the scene.
The dataset also provides OCR annotations for each image, including bounding boxes and transcribed text, which are used as input to many models. This allows researchers to focus on the reasoning and generation aspects rather than the raw OCR detection, although some models also incorporate end-to-end OCR learning.
Evaluation Metrics and Challenges
The primary evaluation metric for TextCaps is CIDEr, which measures the similarity between generated captions and human references, with a focus on consensus and informativeness. Other metrics such as BLEU, METEOR, and ROUGE are also reported, but CIDEr is the main ranking criterion. The task is particularly challenging because captions must be both grammatically correct and factually accurate with respect to the text in the image. A model that correctly identifies the objects but misreads a sign will receive a low score.
Early baselines using standard image captioning models, such as those based on Encoder-Decoder Architecture architectures with Residual Network (ResNet) features, performed poorly on TextCaps, achieving CIDEr scores below 50. This highlighted the need for specialized architectures that can incorporate OCR tokens into the generation process. Subsequent work introduced methods that use a Transformer (architecture)-based decoder with cross-attention to OCR features, leading to significant improvements.
Model Architectures and Approaches
Most successful approaches to TextCaps use a two-stage pipeline: first, an OCR system extracts text from the image, and second, a captioning model generates the description using both visual features and OCR tokens. The captioning model is often a Transformer (architecture) that takes a sequence of image region features and OCR token embeddings as input, and outputs a caption token-by-token. For example, the M4C (Multimodal Multi-Copy Mesh) model, introduced by Hu et al., uses a transformer with a copy mechanism that can directly copy text from the OCR tokens, which is crucial for accurately reproducing strings like brand names or addresses.
Another line of work integrates OCR directly into the visual encoder, using a Neural network that processes both pixels and text embeddings simultaneously. These models often leverage pre-trained Large language models for the text generation part, fine-tuning them on the TextCaps training set. The use of Cross-Attention between visual and textual modalities is a common design pattern, allowing the model to align OCR tokens with image regions.
Impact and Related Benchmarks
TextCaps has spurred the development of more comprehensive benchmarks that combine text and vision, such as OCR-VQA and ST-VQA, which focus on visual question answering with text. It has also influenced the design of multimodal Large language models, such as those developed by OpenAI and Google DeepMind, which are now capable of reading text in images and generating descriptive responses. The dataset remains a standard testbed for evaluating the OCR-aware reasoning abilities of AI systems, and it is often used in conjunction with other datasets like text-vqa and COCO-Text.
As of 2025, state-of-the-art models on TextCaps achieve CIDEr scores above 140, a significant improvement over the initial baselines, but the task is still considered open, with room for improvement in handling rare text styles, complex layouts, and ambiguous text relevance. The benchmark continues to be a valuable resource for advancing research in Generative AI and multimodal understanding.
Future Directions
The future of TextCaps research lies in developing models that can not only read text but also reason about its semantic meaning in context, such as understanding that a "Sale" sign implies a discount or that a "Do Not Enter" sign indicates a restriction. This requires a deeper integration of world knowledge and common sense, which is an active area of study in Artificial intelligence. Additionally, there is interest in extending TextCaps to video, where text appears dynamically, and to multilingual settings, where OCR and captioning must handle multiple scripts. The dataset's design has also inspired similar efforts in other domains, such as medical imaging and autonomous driving, where reading text in the environment is critical.
See Also
- image-captioning
- Optical character recognition
- Vision-language model
- multimodal-learning
References
- Hu, R., Singh, A., Darrell, T., & Rohrbach, M. (2020). TextCaps: A Dataset for Image Captioning with Reading Comprehension. In European Conference on Computer Vision (ECCV).
- Sidorov, O., Hu, R., Rohrbach, M., & Singh, A. (2020). TextCaps: A Dataset for Image Captioning with Reading Comprehension. arXiv preprint arXiv:2003.12462.