# AlexNet Paper (2012)

AlexNet is a deep convolutional neural network developed in 2012 by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton at the University of Toronto. It won the ImageNet ILSVRC-2012 competition with a top-5 error rate of 15.3%, sparking the deep learning revolution.

AlexNet is a convolutional neural network architecture for image classification, developed in 2012 by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton at the [University of Toronto](https://www.wikiprompt.org/wiki/university-of-toronto). It classifies images into 1,000 object categories and is widely regarded as the first large-scale demonstration of deep convolutional networks in visual recognition. Its decisive victory in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012 marked a turning point in [machine learning](https://www.wikiprompt.org/wiki/machine-learning), catalyzing the rapid adoption of [deep learning](https://www.wikiprompt.org/wiki/deep-learning) across [artificial intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) research and industry.

The model contains 60 million parameters and 650,000 neurons. Its architecture consists of eight layers: five convolutional layers (some followed by max-pooling) and three fully connected layers. The network used the non-saturating ReLU activation function, which improved training speed over traditional tanh and sigmoid activations. Due to memory constraints, the network was split across two GPUs, with each copy processing half of the neurons. The final fully connected layer combined the outputs from both GPUs.

## Architecture

AlexNet's structure can be summarized as: (CONV → RN → MP)2 → (CONV3 → MP) → (FC → DO)2 → Linear → softmax. Here, CONV denotes a convolutional layer with ReLU activation, RN is local response normalization, MP is max-pooling, FC is a fully connected layer with ReLU, Linear is a fully connected layer without activation, and DO is dropout. Notably, convolutional layers 3, 4, and 5 were connected sequentially without pooling or normalization in between.

The network used local response normalization and dropout regularization with a drop probability of 0.5 to reduce overfitting. All weights were initialized as Gaussian distributions with mean 0 and standard deviation 0.01. Biases in convolutional layers 2, 4, 5, and all fully connected layers were initialized to 1 to avoid the dying ReLU problem.

## Training

The model was trained on the ImageNet training set containing 1.2 million images. Training ran for 90 epochs over five to six days using two Nvidia GTX 580 GPUs (3GB each), which had a theoretical performance of 1.581 TFLOPS in float32. Each forward pass required approximately 1.43 GFLOPs. The dataset images, stored in JPEG format, occupied 27GB of disk space; the network used 2GB of RAM on each GPU and about 5GB of system RAM during training. GPUs handled training while CPUs performed image loading and data augmentation.

Training used momentum gradient descent with a batch size of 128, momentum of 0.9, and weight decay of 0.0005. The learning rate started at 10⁻² and was manually reduced by a factor of 10 whenever validation error plateaued, ending at 10⁻⁵. Two forms of data augmentation were applied on the fly on the CPU, effectively free: first, images were scaled so their shorter side was 256 pixels, then a central 256×256 crop was taken and normalized; second, random 224×224 patches (and their horizontal reflections) were extracted from the 256×256 crop, increasing the training set size 2048-fold. Additionally, the RGB values of each image were randomly shifted along the principal components of the pixel color distribution.

## ImageNet Competition

The team, named SuperVision, submitted an ensemble of seven AlexNets to the ILSVRC-2012 competition on September 30, 2012. Five networks used the standard architecture, while two variants had an extra convolutional layer and were pre-trained on the larger ImageNet Fall 2011 release (15 million images, 22,000 categories) before fine-tuning on the ILSVRC-2012 training set. The final prediction was the average of the probabilities from all seven networks. The ensemble achieved a top-5 error rate of 15.3%, more than 10.8 percentage points better than the runner-up, a dramatic improvement that stunned the computer vision community.

## Impact and Legacy

AlexNet's success demonstrated the power of deep convolutional networks for large-scale visual recognition, directly influencing subsequent architectures such as [ResNet](https://www.wikiprompt.org/wiki/residual-network) and [U-Net](https://www.wikiprompt.org/wiki/u-net). It also popularized the use of GPUs for training deep networks, a practice that became standard in the field. The paper's emphasis on depth and the effectiveness of ReLU, dropout, and data augmentation shaped modern deep learning practice. AlexNet is often cited as the catalyst for the deep learning revolution, leading to breakthroughs in [generative AI](https://www.wikiprompt.org/wiki/generative-ai) and the development of [large language models](https://www.wikiprompt.org/wiki/large-language-model) and [transformers](https://www.wikiprompt.org/wiki/transformer). Its influence extended beyond academia, spurring investment in AI research at companies like [OpenAI](https://www.wikiprompt.org/wiki/openai), [Google DeepMind](https://www.wikiprompt.org/wiki/google-deepmind), and [Anthropic](https://www.wikiprompt.org/wiki/anthropic).

---
Source: https://www.wikiprompt.org/wiki/alexnet-paper-2012
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-09T02:02:28.293948+00:00