The neocognitron is a hierarchical, multilayered artificial neural network proposed by Kunihiko Fukushima in 1979. It was designed for pattern recognition tasks, such as Japanese handwritten character recognition, and is widely recognized as a direct inspiration for convolutional neural networks (CNNs). The model is grounded in neurophysiological findings about the visual cortex and introduced key concepts like local feature extraction and tolerance to positional shifts that remain central to modern computer vision.
The neocognitron's origins trace back to earlier work by Fukushima. In 1969, he published a similar architecture with hand-designed kernels based on convolutions observed in mammalian vision. He refined this into the Cognitron in 1975, and then into the neocognitron in 1979, which learned all convolutional kernels through unsupervised learning, which he described as "self-organized by 'learning without a teacher'." This evolution marked a shift from manually specified features to learned representations.
Biological Inspiration
The neocognitron was inspired by the model of the visual system proposed by David Hubel and Torsten Wiesel in 1959. Their research on the cat's striate cortex identified two types of cells: simple cells, which respond to oriented edges and bars, and complex cells, which respond to similar features but with greater spatial invariance. They also proposed a cascading model of these cell types for pattern recognition. The neocognitron is a natural extension of this cascading idea, implementing a layered architecture that mimics the hierarchical processing in the visual pathway.
Architecture and Mechanisms
The neocognitron consists of multiple layers of cells, with the most important being S-cells and C-cells. S-cells extract local features, such as edges or corners, from their receptive fields. C-cells tolerate small deformations, such as local shifts, in the features detected by S-cells. As signals propagate through the network, local features are gradually integrated into more complex and global representations, ultimately enabling classification in the higher layers. This principle of local feature integration is also found in other models, including convolutional neural networks, the SIFT method, and the HoG method.
A notable feature of the neocognitron is its ability to learn without supervision. The convolutional kernels are self-organized through a competitive learning process, allowing the network to discover relevant features from input patterns without labeled data. This contrasts with later supervised training methods used in modern CNNs.
Variants and Extensions
Various types of neocognitron have been developed to address different tasks. Some variants can detect multiple patterns in the same input by using backward signals to achieve selective attention, enabling the model to focus on specific objects or regions. Other extensions have incorporated improvements in learning rules, scaling, and rotation invariance, though the core architecture remains hierarchical and biologically inspired.
Legacy and Influence
The neocognitron is considered a foundational model in artificial intelligence and machine learning. Its hierarchical, convolutional design directly influenced the development of CNNs, which became a cornerstone of modern computer vision and deep learning. While the neocognitron itself is less commonly used today, its principles - local receptive fields, shared weights, and hierarchical feature extraction - are embedded in virtually all contemporary convolutional architectures. The model also contributed to the broader understanding of how biological vision can inform computational models.