ImageNet is a large-scale visual database designed for use in visual object recognition software research. It contains more than 14 million hand-annotated images, with over 20,000 categories, each typically consisting of several hundred images. At least one million images include bounding box annotations. The database of annotations and third-party image URLs is freely available, though the actual images are not owned by ImageNet. Since 2010, the project has run an annual software contest, the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), where programs compete to classify and detect objects and scenes using a trimmed list of one thousand non-overlapping classes.
The project originated from the vision of AI researcher Fei-Fei Li, who sought to expand the data available for training AI algorithms at a time when most research focused on models and algorithms. ImageNet has since become a cornerstone of the deep learning revolution, enabling breakthroughs in computer vision and influencing the broader field of Artificial intelligence.
History
Fei-Fei Li began working on the idea for ImageNet in 2006. In 2007, she met with Princeton professor Christiane Fellbaum, one of the creators of WordNet, to discuss the project. This meeting led Li to build ImageNet starting from the roughly 22,000 nouns of WordNet and using many of its features. She was also inspired by a 1987 estimate that the average person recognizes roughly 30,000 different kinds of objects.
As an assistant professor at Princeton, Li assembled a team of researchers to work on the project. They used Amazon Mechanical Turk to help with the classification of images. Labeling started in July 2008 and ended in April 2010, involving 49,000 workers from 167 countries who filtered and labeled over 160 million candidate images. The budget allowed each of the 14 million images to be labeled three times.
The original plan called for 10,000 images per category across 40,000 categories, totaling 400 million images, each verified three times. However, humans can classify at most two images per second, and at that rate it was estimated to take 19 human-years of labor without rest. The full plan was not achieved.
The database was first presented as a poster at the 2009 Conference on Computer Vision and Pattern Recognition (CVPR) in Florida, titled "ImageNet: A Preview of a Large-scale Hierarchical Dataset." The poster was reused at Vision Sciences Society 2009.
In 2009, Alex Berg suggested adding object localization as a task. Li approached the PASCAL Visual Object Classes contest for a collaboration, which resulted in the ImageNet Large Scale Visual Recognition Challenge starting in 2010. The challenge had 1000 classes and object localization, compared to PASCAL VOC which had just 20 classes and 19,737 images in 2010.
Significance for Deep Learning
On 30 September 2012, a convolutional neural network (CNN) called AlexNet achieved a top-5 error of 15.3% in the ImageNet 2012 Challenge, more than 10.8 percentage points lower than the runner-up. The use of graphics processing units (GPUs) during training made this feasible, an essential ingredient of the deep learning revolution. According to The Economist, "Suddenly people started to pay attention, not just within the AI community but across the technology industry as a whole."
In 2015, AlexNet was outperformed by Microsoft's very deep CNN with over 100 layers, which won the ImageNet 2015 contest with a 3.57% error on the test set. Andrej Karpathy estimated in 2014 that with concentrated effort, he could reach a 5.1% error rate, and about 10 people from his lab reached around 12-13% with less effort. It was estimated that with maximal effort, a human could reach 2.4%.
These results demonstrated the power of Deep learning and Neural network architectures, accelerating the adoption of Machine learning techniques across the technology industry.
Dataset
ImageNet crowdsources its annotation process. Image-level annotations indicate the presence or absence of an object class in an image, such as "there are tigers in this image" or "there are no tigers in this image." Object-level annotations provide a bounding box around the visible part of the indicated object. ImageNet uses a variant of the broad WordNet schema to categorize objects, augmented with 120 categories of dog breeds to showcase fine-grained classification.
In 2012, ImageNet was the world's largest academic user of Mechanical Turk. The average worker identified 50 images per minute.
The summary statistics given on April 30, 2010:
- Total number of non-empty synsets: 21,841
- Total number of images: 14,197,122
- Number of images with bounding box annotations: 1,034,908
- Number of synsets with SIFT features: 1,000
- Number of images with SIFT features: 1.2 million
Categories
The categories of ImageNet were filtered from WordNet concepts. Each concept, since it can contain multiple synonyms (for example, "kitty" and "young cat"), is called a "synonym set" or "synset." There were more than 100,000 synsets in WordNet 3.0, the majority being nouns (80,000+). The ImageNet dataset filtered these to 21,841 synsets that are countable nouns that can be visually illustrated.
Each synset in WordNet 3.0 has a "WordNet ID" (wnid), which is a concatenation of part of speech and an "offset" (a unique identifying number). Every wnid starts with "n" because ImageNet only includes nouns. For example, the wnid of synset "dog, domestic dog, Canis familiaris" is "n02084071."
The categories in ImageNet fall into 9 levels, from level 1 (such as "mammal") to level 9 (such as "German shepherd").
Image Format
The images were scraped from online image search engines (Google, Picsearch, MSN, Yahoo, Flickr, etc.) using synonyms in multiple languages. For example, for German shepherd, the search terms included "German police dog," "Alsatian," "ovejero alemán," "pastore tedesco," and "德国牧羊犬."
ImageNet consists of images in RGB format with varying resolutions. For example, in ImageNet 2012, the "fish" category has resolutions ranging from 4288 x 2848 to 75 x 56. In machine learning, these are typically preprocessed into a standard constant resolution and whitened before further processing by neural networks. For instance, in PyTorch, ImageNet images are normalized by dividing pixel values to fall between 0 and 1, then subtracting by [0.485, 0.456, 0.406], and dividing by [0.229, 0.224, 0.225]. These are the mean and standard deviations for ImageNet, so this whitens the input data.
Labels and Annotations
Each image is labelled with exactly one wnid. Dense SIFT features (raw SIFT descriptors, quantized codewords, and coordinates of each descriptor/codeword) for ImageNet-1K were available for download, designed for bag of visual words. The bounding boxes of objects were available for about 3,000 popular synsets with on average 150 images in each synset. Furthermore, some images have attribute annotations, though the full extent of these is not detailed here.
Legacy
ImageNet's creation marked a shift toward data-centric approaches in AI, influencing subsequent datasets and benchmarks. Its annual challenge became a standard for measuring progress in computer vision, and its success catalyzed the growth of Generative AI and other AI applications. The dataset remains a foundational resource for training and evaluating models, and its impact extends to fields such as autonomous-driving and medical-imaging.
Despite its achievements, ImageNet has also raised questions about bias and annotation quality, prompting discussions in the AI community about dataset curation and ethics. Nevertheless, its role in the development of modern AI is undisputed, and it continues to be widely used in research and industry.