LeNet-5 is a convolutional neural network architecture developed by a research group at AT&T Bell Laboratories, centered around Yann LeCun, and published in 1998. It is the most well-known version of the LeNet series, designed for reading small grayscale images of handwritten digits and letters. LeNet-5 is historically important as one of the earliest convolutional neural networks and a foundational contribution to the development of deep learning, influencing many modern architectures.
Convolutional neural networks are a kind of feed-forward neural network whose artificial neurons can respond to a part of the surrounding cells in the coverage range, performing well in large-scale image processing. LeNet-5 exemplifies this by using convolutional layers to extract spatial features, pooling layers for subsampling, and fully connected layers for classification. Its success in practical applications, such as reading millions of checks per day, demonstrated the viability of neural networks in real-world tasks.
Development History
The origins of LeNet trace back to 1988 when Yann LeCun joined the Adaptive Systems Research Department at AT&T Bell Laboratories in Holmdel, New Jersey, headed by Lawrence D. Jackel. In late November 1988, John S. Denker and colleagues published a neural network design for recognizing handwritten zip codes, though its early layers were hand-tuned. In 1989, LeCun and his team first applied the backpropagation algorithm to practical applications, combining convolutional neural networks with backpropagation to read handwritten numbers. This prototype, later called LeNet-1, was successfully applied to identify handwritten zip codes from the US Postal Service, achieving an error rate of only 1% with a rejection rate of about 9%.
Over the next four years, the research continued, leading to the development of the MNIST database in 1994. LeNet-1 was too small for this new dataset, prompting the creation of LeNet-4. By 1998, Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner described LeNet-5 in a paper that also showcased practical applications, including systems for recognizing handwritten characters and models that could read millions of checks per day. This work sparked widespread interest in neural network research, and although modern architectures differ, LeNet served as a starting point for many subsequent designs.
Architecture
LeNet-5 incorporates several motifs common in modern convolutional neural networks, including convolutional layers, pooling layers, and fully connected layers. Each convolutional layer includes three parts: convolution, pooling, and a nonlinear activation function, typically the tanh function. The network uses convolution to extract spatial features, with the concept of receptive fields, and employs average pooling for subsampling. The final layers are fully connected for classification, and sparse connections between layers reduce computational complexity.
The 1989 report by LeCun et al. introduced networks labeled Net-1 through Net-5, with various refinements leading to the 1998 LeNet-5. The naming is inconsistent, and it is unclear what LeNet-2 and LeNet-3 might refer to. LeNet-1, LeNet-4, and LeNet-5 are well-documented, but the intermediate versions are not.
Early Prototypes
The first neural network published by the LeCun group in 1988 was a hybrid approach. Its first stage scaled, deskewed, and skeletonized the input image. The second stage used a convolutional layer with 18 hand-designed kernels, and the third stage was a fully connected network with one hidden layer. The dataset consisted of handwritten digit images extracted from actual U.S. Mail, the same dataset used in the famed 1989 report.
Net-1 to Net-5
The 1989 report detailed Net-1 through Net-5, all with fully connected last layers. The original paper does not explain the padding strategy, and all cells have independent biases, including output cells of convolutional layers.
- Net-1: No hidden layer, fully connected, mapping (16×16) to 10 outputs.
- Net-2: One hidden fully connected layer with 12 hidden units, mapping (16×16) to 12 to 10.
- Net-3: Two hidden convolutional layers, mapping (16×16) to (8×8) to (4×4) to 10, with locally connected layers using 3×3 input shapes and stride 2.
- Net-4: Two hidden layers, the first convolutional and the second locally connected, mapping (16×16) to (8×8×2) to (4×4) to 10, with the convolution layer having 2 kernels of shape 3×3 and stride 2.
- Net-5: The final version, LeNet-5, refined these concepts into a deeper architecture with multiple convolutional and pooling layers, culminating in fully connected layers for classification.
Impact and Legacy
LeNet-5's design principles, such as convolutional feature extraction and subsampling, became foundational in deep learning. It demonstrated that minimizing the number of free parameters in a neural network could enhance generalization, a key insight for later architectures. The network's success in reading handwritten checks in ATMs and postal codes influenced both academic research and commercial applications. While modern neural networks have evolved significantly, LeNet-5 remains a historical milestone, often cited as a precursor to contemporary Deep learning models and a catalyst for the broader adoption of Neural network approaches in Machine learning and Artificial intelligence.