# AlexNet

AlexNet is a deep convolutional neural network developed in 2012 by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton at the University of Toronto, which won the ImageNet Large Scale Visual Recognition Challenge with a top-5 error rate of 15.3%, significantly advancing deep learning for computer vision.

AlexNet is a convolutional neural network architecture designed for image classification, developed in 2012 by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton at the [University of Toronto](https://www.wikiprompt.org/wiki/university-of-toronto). It classifies images into 1,000 distinct object categories and is widely recognized as the first large-scale demonstration of deep convolutional networks in visual recognition. The model contains 60 million parameters and 650,000 neurons, and its depth was essential to its high performance, made feasible by using graphics processing units (GPUs) during training.

The three researchers formed the team SuperVision and submitted AlexNet to the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) on September 30, 2012. The network achieved a top-5 error rate of 15.3%, winning the contest by a margin of more than 10.8% over the runner-up. This result was a turning point for [deep learning](https://www.wikiprompt.org/wiki/deep-learning), influencing a large body of subsequent work in computer vision and [artificial intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence).

## Architecture

AlexNet contains eight layers: the first five are convolutional layers, some followed by max-pooling layers, and the last three are fully connected layers. The network, except the last layer, is split into two copies, each running on one GPU, because it did not fit into the 3GB VRAM of a single Nvidia GTX 580. The structure can be written as (CONV → RN → MP)2 → (CONV3 → MP) → (FC → DO)2 → Linear → softmax, where CONV is a convolutional layer with ReLU activation, RN is local response normalization, MP is max-pooling, FC is a fully connected layer with ReLU, Linear is a fully connected layer without activation, and DO is dropout.

Notably, convolutional layers 3, 4, and 5 are connected without pooling or normalization. AlexNet used the non-saturating ReLU activation function, which trained better than tanh and sigmoid. This choice was a key factor in its training efficiency.

## Training

The ImageNet training set contained 1.2 million images. The model was trained for 90 epochs over five to six days using two Nvidia GTX 580 GPUs (3GB each), which have a theoretical performance of 1.581 TFLOPS in float32 and were priced at US$500 upon release. Each forward pass required approximately 1.43 GFLOPs, so the two GPUs together could theoretically perform over 2,200 forward passes per second under ideal conditions.

The dataset images were stored in JPEG format, taking up 27GB of disk. The neural network used 2GB of RAM on each GPU and about 5GB of system RAM during training. The GPUs handled training, while the CPUs loaded images from disk and performed [data augmentation](https://www.wikiprompt.org/wiki/data-augmentation).

AlexNet was trained with momentum gradient descent with a batch size of 128 examples, momentum of 0.9, and weight decay of 0.0005. The learning rate started at 10−2 and was manually decreased 10-fold whenever validation error appeared to stop decreasing, being reduced three times to end at 10−5. Two forms of data augmentation were computed on the fly on the CPU, making them "computationally free": first, each image was scaled so its shorter side was 256 pixels, then the central 256×256 patch was cropped and normalized; second, random 224×224 patches (and their horizontal reflections) were extracted, increasing the training set size 2048-fold. Additionally, the RGB values were randomly shifted along the principal directions of the pixel values.

The resolution 224×224 was chosen because 256 - 16 - 16 = 224, meaning that framing out 16 pixels on each side of a 256×256 image yields a 224×224 image. AlexNet used local response normalization and dropout regularization with a drop probability of 0.5. All weights were initialized as gaussians with mean 0 and standard deviation 0.01, and biases in convolutional layers 2, 4, 5, and all fully connected layers were initialized to constant 1 to avoid the dying ReLU problem.

At test time, an image is scaled so its shorter side is 256, the central 256×256 patch is cropped, and then five 224×224 patches (four corners and center) plus their horizontal reflections are computed, giving 10 patches. The network's predicted probabilities on all 10 patches are averaged to produce the final prediction.

### ImageNet competition

The version used for the 2012 ImageNet competition was an ensemble of seven AlexNets. Five AlexNets of the standard architecture were trained on the ILSVRC-2012 training set. Two variant AlexNets were also trained, each with an extra convolutional layer over the last pooling layer, first on the entire ImageNet Fall 2011 release (15 million images in 22K categories) and then fine-tuned on the ILSVRC-2012 training set. The final system averaged the predicted probabilities of all seven networks.

## History

### Previous work

In 1980, Kunihiko Fukushima proposed an early CNN called neocognitron, trained by unsupervised learning. LeNet-5, developed by Yann LeCun et al. in 1989, used supervised learning with backpropagation and had an architecture essentially the same as AlexNet on a small scale. Max pooling was used in 1990 for speech processing and first for image processing in the Cresceptron of 1992.

During the 2000s, as GPU hardware improved, researchers adapted GPUs for general-purpose computing, including neural network training. In 2006, K. Chellapilla et al. trained a CNN on GPU that was 4 times faster than an equivalent CPU implementation. In 2009, Raina et al. trained a deep belief network with 100 million parameters on an Nvidia GeForce GTX 280 at up to 70 times speedup over CPUs. A deep CNN by Dan Cireșan et al. at IDSIA in 2011 was 60 times faster than an equivalent CPU implementation, and between May 15, 2011, and September 10, 2012, their CNN won four image competitions and achieved state-of-the-art results on multiple databases. According to the AlexNet paper, Cireșan's earlier net is "somewhat similar" to AlexNet, and both were written with CUDA to run on GPU.

### Computer vision

During the 1990–2010 period, neural networks were not better than other machine learning methods like kernel regression and support vector machines. AlexNet's success in 2012 demonstrated the power of deep learning for computer vision, leading to the widespread adoption of CNNs and the development of subsequent architectures such as [residual networks](https://www.wikiprompt.org/wiki/residual-network). It also spurred interest in GPU-based computing for [machine learning](https://www.wikiprompt.org/wiki/machine-learning) and contributed to the rise of deep learning as a dominant approach in AI.

---
Source: https://www.wikiprompt.org/wiki/alex-net
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-13T03:54:55.109202+00:00