AlexNet is a Convolutional neural network designed by Alex Krizhevsky, with Ilya Sutskever and Geoffrey Hinton, at the University of Toronto. Entered in the 2012 ImageNet 2012 (AlexNet moment) competition, it classified ImageNet photographs into 1,000 categories with a top-5 error rate of 15.3 percent, more than ten percentage points ahead of the next best entrant, which still relied on traditional hand-engineered computer-vision features. The result is widely treated as the starting point of the modern Deep learning era.
The network's name is a shortening of its lead author's first name and was not intended as a formal product title; the paper itself is usually cited simply as "ImageNet Classification with Deep Convolutional Neural Networks" (2012).
History
Krizhevsky built AlexNet as a graduate student in Hinton's lab, extending earlier convolutional-network work by Yann LeCun in the 1990s. Convolutional networks had existed for two decades, and the ImageNet dataset, assembled by Fei-Fei Li's group, had been public since 2009, but no team had combined a sufficiently deep network with enough compute and data to beat classical vision pipelines. Krizhevsky trained the model on two NVIDIA GPU (in AI) cards (GTX 580s), splitting the network across them because a single card lacked enough memory, using NVIDIA's CUDA platform to write the low-level kernels himself.
Architecture
AlexNet has eight learned layers: five convolutional layers followed by three fully connected layers, roughly 60 million parameters in total. Several choices distinguished it from earlier networks. It used the ReLU (rectified linear unit) activation function instead of the slower saturating sigmoid or tanh functions, which sped up training considerably. It applied dropout, a Regularization technique, in the fully connected layers to reduce overfitting on a dataset of 1.2 million training images. It used data augmentation (random crops and horizontal flips) to further limit overfitting, and local response normalization between layers, a technique later generations of networks mostly dropped. Training used Backpropagation with stochastic gradient descent over roughly six days on the two GPUs, an unusually large computational commitment for a vision model at the time.
Impact and legacy
AlexNet's margin of victory convinced much of the computer vision field, which had been skeptical of Neural network approaches after the setbacks of the second AI winter, that depth plus scale plus GPU compute was a winning combination. Citations of the original paper number in the tens of thousands, and the paper is frequently taught as the canonical example of a field-changing empirical result. Industry adoption followed quickly: within two years, most entrants in the ImageNet challenge used deep convolutional networks, and companies including Google, Facebook, and Baidu built large internal deep learning teams. NVIDIA's stock and strategy shifted in AlexNet's wake as GPUs were increasingly marketed for training rather than only rendering graphics, a relationship that deepened over the following decade as networks grew from AlexNet's eight layers to hundreds of layers and, later, to the very different architecture of the Transformer (architecture).
AlexNet itself was quickly superseded by deeper networks such as VGGNet, GoogLeNet, and ResNet in subsequent ImageNet competitions, and convolutional architectures for many tasks have since been challenged by vision transformers. Its lasting significance is historical rather than technical: it is generally cited as the single result that convinced the wider machine learning and Computer vision community to commit to Deep learning methods at scale, reshaping both academic research priorities and the hardware industry that supports them.