COCO test-dev is a designated test split of the Common Objects in Context (COCO) dataset, created for the evaluation of models in the COCO detection and segmentation challenges. Unlike the standard test split, test-dev is a "blind" set: its ground-truth annotations are not publicly released. Researchers submit their model predictions to the evaluation server, which computes metrics such as mean Average Precision (mAP) and returns scores. This protocol is designed to prevent overfitting to the test set and to ensure fair comparison among competing methods.
The COCO dataset was introduced in 2014 by Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. It contains over 200,000 images with more than 1.5 million object instances labeled across 80 categories. The dataset is widely used in Computer vision research for tasks including object detection, instance segmentation, and keypoint detection. The test-dev split includes approximately 20,000 images, a subset of the full test set, which itself contains around 40,000 images. The remaining test images are used for the "test-challenge" split, which is reserved for challenge evaluation and has no public annotations either.
Evaluation Protocol
Participants in the COCO challenges, such as the COCO Detection Challenge held at the Conference on Computer Vision and Pattern Recognition (CVPR) from 2015 onward, submit results for the test-dev split. The evaluation server computes standard metrics: Average Precision (AP) at IoU thresholds from 0.5 to 0.95 in steps of 0.05, AP at IoU 0.5, AP at IoU 0.75, and average recall (AR) for different numbers of detections. The primary metric is AP at IoU 0.50:0.95, often referred to as COCO AP. This metric is more stringent than the traditional PASCAL VOC metric, which uses a single IoU threshold of 0.5.
The blind nature of test-dev means that researchers cannot iterate on the test set directly. Instead, they typically use a held-out validation set (val2014 or val2017) for development and hyperparameter tuning. The test-dev split is used only for final reporting, which helps maintain the integrity of the benchmark. Over the years, the COCO evaluation server has become a standard for comparing Machine learning models in object detection, with many state-of-the-art systems reporting their performance on test-dev.
Historical Context
COCO was created to address limitations of earlier datasets like PASCAL VOC and ImageNet, which had fewer object categories or focused on classification rather than localization. The test-dev split was introduced as part of the dataset's design to support challenge-based evaluation. The first COCO challenge was held in 2015, and subsequent editions occurred annually until 2020, with the last official challenge at the 2020 ECCV workshop. During this period, test-dev became the de facto benchmark for comparing detection and segmentation algorithms.
The dataset has seen multiple versions: COCO 2014, COCO 2015, and COCO 2017. The test-dev split exists for each version, but the 2017 version is the most commonly used in recent research. The 2017 test-dev contains 20,288 images, and its annotations are withheld. The evaluation server for COCO 2017 remains operational, allowing researchers to submit predictions and obtain metrics.
Impact on Research
COCO test-dev has been instrumental in driving progress in object detection and instance segmentation. Many influential models have reported results on this split, including Faster R-CNN, Mask R-CNN, and YOLO variants. For example, Mask R-CNN, introduced by Kaiming He and colleagues in 2017, achieved an AP of 37.1 on test-dev, setting a new state of the art at the time. Subsequent models like EfficientDet and Swin Transformer have pushed AP above 50. The availability of a standardized blind test set has enabled fair comparisons and accelerated the development of techniques such as feature pyramid networks, anchor-free detectors, and transformer-based detectors.
Beyond detection, test-dev is also used for evaluating panoptic segmentation, a task introduced in 2018 that combines semantic and instance segmentation. The COCO panoptic challenge uses a separate test-dev split with 80 thing categories and 91 stuff categories. This extension has broadened the utility of the benchmark.
Limitations and Alternatives
Despite its widespread use, test-dev has limitations. The small number of images (about 20,000) compared to the scale of modern datasets can lead to noisy metrics, especially for rare categories. Additionally, the blind evaluation protocol requires submitting predictions to a server, which can be a barrier for researchers without internet access or for those who want to evaluate on local hardware. To address these issues, some researchers have created alternative benchmarks, such as LVIS (Large Vocabulary Instance Segmentation), which has a larger category set and a different evaluation protocol. However, COCO test-dev remains a standard reference point in the field.
The COCO dataset itself is hosted by the COCO Consortium, and the evaluation server is maintained by the team at the University of Michigan and other contributors. As of 2025, the server is still active, though the official challenges have ceased. Researchers continue to use test-dev for reporting results in papers, and it remains a common benchmark in Deep learning and Artificial intelligence research.
See Also
- Computer vision
- Object Detection
- Instance Segmentation
- mean-average-precision