LVIS, short for Large Vocabulary Instance Segmentation, is a dataset used in computer vision and Machine learning to evaluate and train models for instance segmentation and object detection. It was introduced in 2019 by researchers at Facebook AI Research (now part of Meta AI) to address the challenge of long-tail visual recognition, where most object categories in the real world are rare, yet standard datasets focused on a small set of common classes. LVIS provides a large vocabulary of 1,203 object categories, ranging from everyday items like 'banana' to niche objects like 'triceratops skull', with over 2 million high-quality instance segmentation masks annotated on approximately 164,000 images sourced from the COCO dataset. The dataset's design emphasizes exhaustive annotation, meaning annotators were instructed to label every instance of every category present in an image, which contrasts with earlier datasets that often ignored rare objects. This makes LVIS a rigorous benchmark for Deep learning models, particularly for measuring performance on rare categories, and it has driven research into techniques like federated loss and dynamic gradient clipping to mitigate the long-tail effect.
The creation of LVIS was motivated by the observation that real-world visual data follows a power-law distribution, where a few categories (e.g., 'person', 'car') appear frequently, while thousands of others appear only occasionally. Traditional datasets like COCO or ImageNet either had a limited number of categories or did not provide pixel-level masks for all instances, limiting their utility for fine-grained recognition. LVIS was built by re-annotating a subset of COCO images with a much larger category set, using a crowd-sourced annotation pipeline that involved a two-stage process: first, a 'discovery' phase where annotators identified all objects in an image, and second, a 'verification' phase to ensure completeness and accuracy. The final dataset includes 1,203 categories, split into three frequency groups: frequent (over 100 images), common (10-100 images), and rare (fewer than 10 images), with roughly 77% of categories falling into the rare group. This imbalance is intentional, as it mirrors real-world conditions and forces models to generalize beyond head categories.
Dataset Structure and Annotation
LVIS is built on top of the COCO dataset, using 164,000 of its training and validation images, but with a significantly expanded category vocabulary. Each image is annotated with instance-level segmentation masks, meaning every object instance is outlined at the pixel level, not just bounded by a box. The annotation process was designed to be exhaustive: annotators were required to label all instances of all categories present, even if they were small, occluded, or blurry. This is a key difference from COCO, which only required annotating 80 categories and often skipped rare objects. The dataset provides a JSON file with annotations in a format similar to COCO, including fields for image IDs, category IDs, segmentation polygons, and bounding boxes. The category list was curated from a combination of WordNet synsets and manual selection, ensuring a broad coverage of everyday objects, animals, foods, and tools.
Benchmark and Evaluation Metrics
The primary evaluation metric for LVIS is the standard COCO-style mean Average Precision (AP), computed at multiple Intersection over Union (IoU) thresholds (0.5 to 0.95). However, LVIS introduces a critical twist: AP is reported separately for the three frequency groups (rare, common, frequent), and the overall AP is the average of these three group APs, rather than a simple average over all categories. This 'balanced' AP ensures that performance on rare categories is not overshadowed by common ones, making it a more sensitive measure of long-tail generalization. For example, a model that performs well on 'person' but poorly on 'abacus' will score lower on LVIS than on COCO. The benchmark also includes a 'mask AP' for instance segmentation and a 'box AP' for detection, with the primary leaderboard focusing on mask AP. Since its release, LVIS has been used in several challenges, including the LVIS Challenge at ICCV 2019 and subsequent workshops, where top entries have employed techniques like class-balanced sampling, re-weighting losses, and using larger backbones like Transformer (architecture)-based detectors.
Impact on Long-Tail Learning
LVIS has become a standard testbed for long-tail learning, a subfield of Artificial intelligence concerned with training models on imbalanced data. The dataset's design has spurred numerous algorithmic innovations. One notable example is the 'federated loss' proposed by the LVIS authors, which groups categories into 'federated' sets based on their frequency and computes a softmax over each group, preventing rare categories from being overwhelmed by frequent ones in the loss function. Another is 'dynamic gradient clipping', which adjusts the gradient magnitude for rare categories during training. Subsequent work has explored decoupled training (first learning features, then fine-tuning the classifier), data augmentation for rare classes, and using Neural network architectures with better feature extraction. LVIS has also influenced other datasets, such as Open Images and Objects365, which adopted similar exhaustive annotation strategies. As of 2025, LVIS remains a widely cited benchmark, with over 1,000 citations in academic literature, and it is often used in conjunction with Large language model-based vision-language models to test their zero-shot or few-shot recognition capabilities on rare objects.
Related Datasets and Extensions
LVIS has spawned several extensions and related efforts. The LVIS Challenge dataset includes a validation set of 20,000 images and a test set, with annotations for the same 1,203 categories. In 2020, the authors released LVIS v1.0, which fixed annotation errors and expanded the training set to 100,000 images (from the original 57,000), improving consistency. A variant called LVIS-OW (Open World) was introduced for open-world instance segmentation, where models must identify unknown objects. Additionally, LVIS has been integrated into the Detectron2 framework, making it easy for researchers to train and evaluate models. The dataset's category list has also been used in other tasks, such as panoptic segmentation and video instance segmentation, where it serves as a common vocabulary. Despite its success, LVIS has limitations: it is derived from COCO images, which are biased toward web photos, and its category set, while large, is still finite. Future datasets, such as those from the Ego4D project, have built on LVIS's principles to create even more diverse benchmarks.