The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) was an annual software competition held from 2010 to 2017 as part of the ImageNet project. It tasked algorithms with correctly classifying and detecting objects and scenes within a curated set of 1,000 non-overlapping object classes. The challenge is widely credited with catalyzing the deep learning revolution in computer vision, most notably after 2012 when a convolutional neural network achieved a dramatic improvement in accuracy.
The competition grew out of the ImageNet project, a large visual database designed for visual object recognition research. Initiated by AI researcher Fei-Fei Li, ImageNet contains over 14 million hand-annotated images spanning more than 20,000 categories, with at least one million images also providing bounding box annotations. ILSVRC used a subset of these images and categories to provide a standardized benchmark for evaluating and comparing computer vision algorithms.
History
Fei-Fei Li began conceptualizing ImageNet in 2006, aiming to expand the data available for training AI algorithms at a time when most research focused on models and algorithms. In 2007, she met Princeton professor Christiane Fellbaum, a creator of WordNet, which led to building ImageNet based on WordNet's roughly 22,000 nouns. Li was also inspired by a 1987 estimate that the average person recognizes about 30,000 different kinds of objects.
As an assistant professor at Princeton, Li assembled a team and used Amazon Mechanical Turk for image classification. Labeling ran from July 2008 to April 2010, involving 49,000 workers from 167 countries who filtered and labeled over 160 million candidate images. Each of the 14 million images was labeled three times. The original plan called for 10,000 images per category across 40,000 categories, but this proved impractical; humans can classify at most 2 images per second, requiring an estimated 19 human-years of labor.
The database was first presented as a poster at the 2009 Conference on Computer Vision and Pattern Recognition (CVPR) in Florida, titled "ImageNet: A Preview of a Large-scale Hierarchical Dataset." In 2009, Alex Berg suggested adding object localization as a task, leading to collaboration with the PASCAL Visual Object Classes contest. The first ILSVRC was held in 2010, featuring 1,000 classes and object localization, compared to PASCAL VOC's 20 classes and 19,737 images.
Significance for Deep Learning
On 30 September 2012, a convolutional neural network (CNN) called AlexNet achieved a top-5 error of 15.3% in the ImageNet 2012 Challenge, more than 10.8 percentage points lower than the runner-up. This success was made feasible by using graphics processing units (GPUs) during training, an essential ingredient of the deep learning revolution. According to The Economist, "Suddenly people started to pay attention, not just within the AI community but across the technology industry as a whole."
In 2015, AlexNet was outperformed by Microsoft's very deep CNN with over 100 layers, which won the ImageNet 2015 contest with a 3.57% error on the test set. Andrej Karpathy estimated in 2014 that with concentrated effort, he could reach a 5.1% error rate, and about 10 people from his lab reached roughly 12-13% with less effort. It was estimated that with maximal effort, a human could achieve 2.4% error.
The challenge's impact extended beyond computer vision, spurring innovations in neural network architectures, training techniques, and hardware acceleration. It also influenced the development of machine learning frameworks and the broader field of artificial intelligence.
Dataset
ImageNet crowdsources its annotation process. Image-level annotations indicate the presence or absence of an object class in an image, while object-level annotations provide a bounding box around the visible part of the object. ImageNet uses a variant of the WordNet schema to categorize objects, augmented with 120 categories of dog breeds for fine-grained classification.
In 2012, ImageNet was the world's largest academic user of Mechanical Turk, with the average worker identifying 50 images per minute. The original plan for the full ImageNet would have roughly 50 million clean, diverse, and full-resolution images spread over approximately 50,000 synsets, but this was not achieved.
As of April 30, 2010, the summary statistics were:
- Total number of non-empty synsets: 21,841
- Total number of images: 14,197,122
- Number of images with bounding box annotations: 1,034,908
- Number of synsets with SIFT features: 1,000
- Number of images with SIFT features: 1.2 million
Categories
The categories of ImageNet were filtered from WordNet concepts. Each concept, which can contain multiple synonyms (e.g., "kitty" and "young cat"), is called a "synonym set" or "synset." WordNet 3.0 contains more than 100,000 synsets, mostly nouns (80,000+). ImageNet filtered these to 21,841 synsets that are countable nouns that can be visually illustrated.
Each synset in WordNet 3.0 has a "WordNet ID" (wnid), a concatenation of part of speech and an offset. Every wnid starts with "n" because ImageNet only includes nouns. For example, the wnid of the synset "dog, domestic dog, Canis familiaris" is "n02084071". The categories fall into 9 levels, from level 1 (such as "mammal") to level 9 (such as "German shepherd").
Image Format
The images were scraped from online image search engines (Google, Picsearch, MSN, Yahoo, Flickr, etc.) using synonyms in multiple languages. For example, for German shepherd, they used terms like "German police dog," "Alsatian," "ovejero alemán," "pastore tedesco," and "德国牧羊犬."
ImageNet consists of images in RGB format with varying resolutions. For instance, in the ImageNet 2012 "fish" category, resolutions range from 4288 x 2848 to 75 x 56. In machine learning, these are typically preprocessed into a standard constant resolution and whitened before further processing by neural networks. For example, in PyTorch, ImageNet images are normalized by dividing pixel values to fall between 0 and 1, then subtracting [0.485, 0.456, 0.406], and dividing by [0.229, 0.224, 0.225], which are the mean and standard deviations for ImageNet.
Labels and Annotations
Each image is labelled with exactly one wnid. Dense SIFT features (raw SIFT descriptors, quantized codewords, and coordinates) for ImageNet-1K were available for download, designed for bag of visual words. Bounding boxes of objects were available for about 3,000 popular synsets, with an average of 150 images per synset. Additionally, some images have attribute annotations, though the challenge primarily focused on classification and localization tasks.
The ILSVRC concluded in 2017, but its legacy persists in the form of the ImageNet dataset, which remains a standard benchmark for evaluating computer vision models. The competition's emphasis on large-scale, real-world data and rigorous evaluation helped drive rapid progress in the field, influencing subsequent developments in generative AI and other areas of machine learning.