NYU Depth V2 is a large-scale RGB-D dataset for indoor scene understanding, released in 2012 by researchers at New York University. It consists of video sequences recorded with a Microsoft Kinect sensor, capturing both color (RGB) images and aligned depth maps. The dataset is a standard benchmark for tasks such as monocular depth estimation, semantic segmentation, and scene parsing, and it has been instrumental in advancing Deep learning approaches for indoor perception.
The dataset contains 1,449 densely labeled pairs of RGB and depth images, where each pixel is annotated with one of 894 object categories (later consolidated into 40 classes for common benchmarks). In addition, it provides 407,024 unlabeled frames from 464 diverse indoor scenes, covering rooms like bedrooms, living rooms, offices, and kitchens. The scenes were captured from a first-person perspective, mimicking a person moving through a space, which makes the data particularly relevant for robotics and augmented reality applications.
Acquisition and Annotation
The original NYU Depth dataset (V1) was released in 2011, but V2 expanded the number of scenes and improved the annotation quality. The RGB-D data were recorded using a Kinect sensor at a resolution of 640x480 pixels for both color and depth streams. The depth maps were captured using structured light, which provides metric depth information in millimeters. To create the labeled subset, the researchers selected frames from the video sequences and manually annotated them using a custom tool, labeling each pixel with a semantic class. The annotation process involved multiple passes to ensure consistency, and the final labels were verified by human raters.
Benchmark Tasks
NYU Depth V2 is primarily used for two tasks: depth estimation and semantic segmentation. For depth estimation, models are trained to predict a dense depth map from a single RGB image, with evaluation metrics such as root mean squared error (RMSE) and relative error. For semantic segmentation, models classify each pixel into one of the 40 classes, with metrics like pixel accuracy and mean intersection over union (mIoU). The dataset also supports multi-task learning, where a model simultaneously predicts depth and semantics, which is common in modern Neural network architectures.
Impact on Research
Since its release, NYU Depth V2 has been a cornerstone for indoor scene understanding. It has been used in hundreds of papers, including early works on Residual Network (ResNet)-based encoders and later transformer-based models. The dataset's depth maps have also been used for 3D reconstruction and scene completion tasks. Its popularity stems from the realistic indoor environments and the availability of both dense labels and large unlabeled data, which enables semi-supervised and self-supervised learning techniques. Researchers have also used the unlabeled frames for pre-training models with Data Augmentation and contrastive learning.
Limitations and Extensions
The dataset has known limitations, such as sensor noise in depth maps, particularly on reflective or dark surfaces, and a bias toward residential interiors. The 40-class label set is coarse, merging many fine-grained categories. To address these issues, extensions like NYU Depth V2 with improved labels and synthetic datasets have been proposed. Despite these challenges, NYU Depth V2 remains a reference point for comparing indoor perception algorithms, and its influence extends to commercial applications in robotics and Amazon Web Services cloud-based vision services.
See Also
- Machine learning
- Computer vision (not in list, but related)
- U-Net
- Batch Normalization