nuScenes is a large-scale public dataset for autonomous driving research, released by Aptiv in 2019. It provides a comprehensive collection of multimodal sensor data captured from a fleet of vehicles operating in urban environments. The dataset is widely used by researchers and developers to train and evaluate algorithms for perception, prediction, and planning tasks in Artificial intelligence and Machine learning. Its primary goal is to accelerate progress in self-driving technology by offering a standardized benchmark with rich annotations and diverse driving scenarios.
The dataset comprises 1,000 driving scenes, each lasting approximately 20 seconds, resulting in over 1.4 million annotated objects across 23 object classes. The sensor suite includes six cameras providing 360-degree coverage, five long-range radars, one lidar, and a full vehicle state report with GPS and IMU data. Data was collected in two geographically distinct cities - Boston, Massachusetts, and Singapore - to capture varied traffic patterns, road layouts, and weather conditions. This diversity makes nuScenes particularly valuable for testing the robustness of models under different environmental and cultural driving contexts.
Annotation and Benchmark Tasks
nuScenes provides detailed 3D bounding box annotations for all objects, including position, size, orientation, and velocity. Each object is tracked across frames, enabling temporal analysis. The dataset supports several core benchmark tasks: 3D object detection, tracking, and motion prediction. For detection, the primary evaluation metric is the nuScenes Detection Score (NDS), which combines mean Average Precision (mAP) with penalties for errors in translation, scale, orientation, velocity, and attribute classification. The tracking benchmark uses AMOTA (Average Multi-Object Tracking Accuracy), while the prediction task evaluates the accuracy of future trajectories over 2, 4, and 6-second horizons.
Sensor Fusion and Design
A distinctive feature of nuScenes is its emphasis on sensor fusion. Unlike many earlier datasets that relied primarily on lidar or cameras alone, nuScenes includes synchronized data from all sensors, allowing researchers to explore multi-modal approaches. The radar data, often underutilized in other benchmarks, is fully annotated and integrated, enabling studies on cost-effective perception systems. The dataset also provides a unified coordinate system and calibration files for all sensors, simplifying the development of fusion algorithms. This design has made nuScenes a standard testbed for comparing Deep learning architectures, including Neural network-based detectors and Transformer (architecture)-based models for 3D perception.
Impact and Adoption
The release of nuScenes marked a shift toward more realistic and challenging benchmarks in autonomous driving research. Prior datasets like KITTI were smaller and captured in controlled settings; nuScenes offered greater scale, sensor diversity, and urban complexity. It has been cited in thousands of research papers and is a common choice for academic competitions and industry evaluations. The dataset has also influenced subsequent efforts, such as the Waymo Open Dataset and Argoverse, by setting standards for annotation granularity and evaluation protocols. Its public availability under a research license has democratized access to high-quality driving data, enabling smaller labs and startups to participate in cutting-edge Machine learning research.
Extensions and Related Tools
Aptiv and the research community have released several extensions to nuScenes. The nuScenes-devkit, an open-source Python library, provides utilities for data loading, visualization, and evaluation, lowering the barrier to entry. In 2020, the nuScenes-lidarseg benchmark added per-point semantic segmentation labels for lidar data, expanding the scope to include scene understanding beyond object detection. The dataset also includes a map expansion with detailed lane and drivable area information, supporting path planning and behavior prediction research. These resources have made nuScenes not just a static dataset but a growing ecosystem for autonomous driving Artificial intelligence development.
Limitations and Criticisms
Despite its strengths, nuScenes has known limitations. The data was collected primarily in clear weather conditions, with limited rain, snow, or night-time scenes, which may reduce model generalization to adverse conditions. The sensor suite, while comprehensive, uses a 32-beam lidar, which has lower resolution than the 64-beam or 128-beam sensors used in some other datasets. Additionally, the annotation process, though rigorous, can introduce errors in dense traffic or occluded scenes. Researchers have noted that the dataset's urban focus may not capture highway or rural driving scenarios, limiting its applicability to all autonomous driving contexts. These factors are important to consider when using nuScenes as a sole benchmark for production-ready systems.
See Also
- Waymo - Another major autonomous driving dataset and company
- Cruise - Autonomous vehicle company using similar sensor setups
- Tesla - A vision-based approach to self-driving, contrasting with nuScenes' multi-sensor design
- Deep learning - Core methodology used in nuScenes benchmarks