COCO-Stuff 164K is a large-scale dataset for semantic segmentation and scene understanding, built as an extension of the Microsoft Common Objects in Context (COCO) dataset. It provides dense pixel-level annotations for both 'stuff' classes (amorphous background regions such as sky, grass, and road) and 'thing' classes (countable objects such as person, car, and dog). The dataset contains 164,000 images, each annotated with one of 172 semantic categories, making it one of the most comprehensive resources for training and evaluating models in computer vision tasks like semantic segmentation, instance segmentation, and scene parsing.
The dataset was introduced in 2018 by researchers at the University of Freiburg, including Holger Caesar, Jasper Uijlings, and Vittorio Ferrari, and was published in the paper 'COCO-Stuff: Thing and Stuff Classes in Context'. It builds upon the original COCO-Stuff dataset, which had 10,000 images, by scaling up to the full COCO training and validation splits. The 172 classes are divided into 80 thing classes (identical to COCO's thing categories) and 92 stuff classes, which include natural and man-made background elements.
Construction and Annotation
The annotation process for COCO-Stuff 164K involved both automatic and manual steps. The initial stuff annotations were generated using a combination of deep learning models and human refinement. The dataset leverages the existing COCO annotations for thing classes, which were created by crowd workers, and adds pixel-level masks for stuff classes. The final annotations were validated through a series of quality control checks to ensure consistency and accuracy across the 164,000 images.
Each image in the dataset is annotated with a single label per pixel, and the annotations are provided in a compact run-length encoding format to reduce storage requirements. The dataset is split into a training set of 118,000 images and a validation set of 5,000 images, with the remaining images reserved for testing (annotations for the test set are not publicly released).
Applications in Deep Learning
COCO-Stuff 164K has become a standard benchmark for semantic segmentation, a task in Machine learning and Deep learning where a model assigns a class label to every pixel in an image. It is widely used to evaluate the performance of Neural network architectures, including Residual Network (ResNet) and U-Net based models, as well as transformer-based approaches. The dataset's diversity of both stuff and thing classes challenges models to understand both fine-grained object boundaries and large-scale contextual regions.
The dataset is also used for panoptic segmentation, a task that unifies semantic and instance segmentation by requiring models to predict both stuff and thing labels simultaneously. This has driven research into unified architectures that can handle both types of classes in a single forward pass.
Benchmark and Evaluation Metrics
The primary evaluation metric for COCO-Stuff 164K is mean Intersection over Union (mIoU), which measures the overlap between predicted and ground-truth segmentation masks averaged over all classes. Researchers also report per-class IoU and pixel accuracy. The dataset's official evaluation server allows for fair comparison of models, and leaderboards track the state of the art over time.
Since its release, the dataset has been used in hundreds of research papers, and top-performing models have achieved mIoU scores above 50 on the validation set, with significant progress driven by advances in Transformer (architecture)-based architectures and attention mechanisms.
Relation to Other Datasets
COCO-Stuff 164K is part of a family of datasets derived from COCO, including COCO-Stuff 10K and the original COCO dataset. It complements other scene understanding datasets such as Cityscapes, which focuses on urban driving scenes, and ADE20K, which covers a broader range of indoor and outdoor scenes. The dataset's emphasis on context and stuff classes makes it particularly useful for applications in autonomous driving, robotics, and augmented reality, where understanding background regions is as important as detecting objects.
The dataset is publicly available for research purposes and is hosted on the official COCO website. It is commonly used in conjunction with pre-trained models from Artificial intelligence research labs and cloud platforms, and it remains a key resource for advancing the field of computer vision.