# AlexNet ImageNet AlexNet is a deep convolutional neural network developed in 2012 by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton at the University of Toronto, which won the ImageNet Large Scale Visual Recognition Challenge with a top-5 error rate of 15.3%, significantly advancing deep learning for computer vision. AlexNet is a [convolutional neural network](https://www.wikiprompt.org/wiki/neural-network) architecture designed for image classification, notably achieving prominence through its performance in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC). It classifies images into 1,000 distinct object categories and is regarded as the first widely recognized application of deep convolutional networks in large-scale visual recognition. Developed in 2012 by Alex Krizhevsky in collaboration with Ilya Sutskever and his Ph.D. advisor Geoffrey Hinton at the [University of Toronto](https://www.wikiprompt.org/wiki/university-of-toronto), the model contains 60 million parameters and 650,000 neurons. The original paper's primary result was that the depth of the model was essential for its high performance, which was computationally expensive but made feasible by utilizing graphics processing units (GPUs) during training. ## Architecture AlexNet contains eight layers: the first five are convolutional layers, some followed by max-pooling layers, and the last three are fully connected layers. The network, except the last layer, is split into two copies, each run on one GPU, because the network did not fit the VRAM of a single Nvidia GTX 580 3GB GPU. The entire structure can be written as (CONV → RN → MP)2 → (CONV3 → MP) → (FC → DO)2 → Linear → softmax, where CONV = convolutional layer (with ReLU activation), RN = local response normalization, MP = max-pooling, FC = fully connected layer (with ReLU activation), Linear = fully connected layer (without activation), and DO = dropout. Notably, the convolutional layers 3, 4, and 5 were connected to one another without any pooling or normalization. It used the non-saturating ReLU activation function, which trained better than tanh and sigmoid. The architecture influenced a large number of subsequent work in [deep learning](https://www.wikiprompt.org/wiki/deep-learning), especially in applying neural networks to computer vision. ## Training The ImageNet training set contained 1.2 million images. The model was trained for 90 epochs over a period of five to six days using two Nvidia GTX 580 GPUs (3GB each). These GPUs have a theoretical performance of 1.581 TFLOPS in float32 and were priced at US$500 upon release. Each forward pass of AlexNet required approximately 1.43 GFLOPs. Based on these values, the two GPUs together were theoretically capable of performing over 2,200 forward passes per second under ideal conditions. The dataset images were stored in JPEG format, taking up 27GB of disk. The neural network took up 2GB of RAM on each GPU and around 5GB of system RAM during training. The GPUs were responsible for training, while the CPUs were responsible for loading images from disk and data-augmenting the images. AlexNet was trained with momentum gradient descent with a batch size of 128 examples, momentum of 0.9, and weight decay of 0.0005. The learning rate started at 10−2 and was manually decreased 10-fold whenever validation error appeared to stop decreasing; it was reduced three times during training, ending at 10−5. It used two forms of data augmentation, both computed on the fly on the CPU, thus "computationally free": - Each image from ImageNet was first scaled so that its shorter side was of length 256. Then the central 256×256 patch was cropped out and normalized (dividing the pixel values so that they fall between 0 and 1, then subtracting by [0.485, 0.456, 0.406], then dividing by [0.229, 0.224, 0.225]). These are the mean and standard deviations for ImageNet, so this whitens the input data. - Extracting random 224×224 patches (and their horizontal reflections) from the 256×256 crop increases the size of the training set 2048-fold. - Randomly shifting the RGB value of each image along the three principal directions of the RGB values of its pixels. The resolution 224×224 was picked because 256 - 16 - 16 = 224, meaning that given a 256×256 image, framing out a width of 16 on its 4 sides results in a 224×224 image. It used local response normalization and [dropout regularization](https://www.wikiprompt.org/wiki/dropout) with drop probability 0.5. All weights were initialized as gaussians with 0 mean and 0.01 standard deviation. Biases in convolutional layers 2, 4, 5, and all fully-connected layers were initialized to constant 1 to avoid the dying ReLU problem. At test time, to use a trained AlexNet for predicting the class of an image, that image is first scaled so that its shorter side is of length 256. Then the central 256×256 patch is cropped out. Then, the five 224×224 patches (the four corner patches and the center patch) as well as their horizontal reflections are computed, 10 patches in all. The network's predicted probabilities on all 10 patches are averaged, and that is the final predicted probability. ## ImageNet competition The version used to enter the 2012 ImageNet competition was an ensemble of 7 AlexNets. Specifically, they trained 5 AlexNets of the previously described architecture (with 5 CONV layers) on the ILSVRC-2012 training set (1.2 million images). They also trained 2 variant AlexNets, obtained by adding one extra CONV layer over the last pooling layer. These were trained by first training on the entire ImageNet Fall 2011 release (15 million images in 22K categories), and then finetuning it on the ILSVRC-2012 training set. The final system of 7 AlexNets was used by averaging their predicted probabilities. The three formed team SuperVision and submitted AlexNet in the ImageNet Large Scale Visual Recognition Challenge on September 30, 2012. The network achieved a top-5 error rate of 15.3% to win the contest, more than 10.8% above the runner-up. ## History ### Previous work In 1980, Kunihiko Fukushima proposed an early CNN named neocognitron, trained by an unsupervised learning algorithm. The LeNet-5 (Yann LeCun et al., 1989) was trained by supervised learning with backpropagation algorithm, with an architecture essentially the same as AlexNet on a small scale. Max pooling was used in 1990 for speech processing (essentially a 1-dimensional CNN), and for image processing, was first used in the Cresceptron of 1992. During the 2000s, as GPU hardware improved, some researchers adapted these for general-purpose computing, including neural network training. K. Chellapilla et al. (2006) trained a CNN on GPU that was 4 times faster than an equivalent CPU implementation. Raina et al. (2009) trained a deep belief network with 100 million parameters on an Nvidia GeForce GTX 280 at up to 70 times speedup over CPUs. A deep CNN of Dan Cireșan et al. (2011) at IDSIA was 60 times faster than an equivalent CPU implementation. Between May 15, 2011, and September 10, 2012, their CNN won four image competitions and achieved state of the art for multiple image databases. According to the AlexNet paper, Cireșan's earlier net is "somewhat similar". Both were written with CUDA to run on GPU. ### Computer vision During the 1990–2010 period, neural networks were not better than other [machine learning](https://www.wikiprompt.org/wiki/machine-learning) methods like kernel regression and support vector machines. AlexNet's success in the 2012 ILSVRC marked a turning point, demonstrating that deep CNNs could outperform traditional approaches by a large margin, leading to the widespread adoption of deep learning in computer vision and beyond. --- Source: https://www.wikiprompt.org/wiki/alexnet-imagenet License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-13T03:55:25.94432+00:00