A capsule neural network (CapsNet) is a type of artificial neural network used in Machine learning to better model hierarchical relationships in data, particularly in image recognition. The approach attempts to more closely mimic biological neural organization by introducing structures called capsules, which are groups of neurons that collectively encode both the probability of an entity's presence and its pose (such as position, size, and orientation). CapsNets were developed to address limitations of traditional convolutional neural networks (CNNs), especially their reliance on pooling layers and their difficulty in handling spatial relationships among object parts.
CapsNets exploit the fact that while viewpoint changes have nonlinear effects at the pixel level, they have linear effects at the part/object level. This property allows the network to recognize objects regardless of their pose, similar to inverting the rendering of an object composed of multiple parts. Among other benefits, CapsNets address the "Picasso problem" in image recognition, where images contain all the correct parts but in incorrect spatial relationships (e.g., a face with a switched eye and mouth).
History
The concept of capsule networks traces back to 2000, when Geoffrey Hinton and colleagues described an imaging system that combined segmentation and recognition into a single inference process using parse trees. This system, called credibility networks, modeled the joint distribution over latent variables and possible parse trees and proved useful on the MNIST handwritten digit database. In Hinton's original idea, a minicolumn would represent and detect one multidimensional entity.
A dynamic routing mechanism for capsule networks was introduced by Hinton and his team in 2017. This approach was claimed to reduce error rates on MNIST and to reduce training set sizes, with results considerably better than a CNN on highly overlapped digits. Since then, CapsNets have been studied for various applications, though they have not been as widely adopted as other architectures like transformers.
Transformations
In computer vision, understanding transformations is crucial. An invariant is an object property that does not change under a transformation, such as the area of a circle when shifted. An equivariant is a property that changes predictably, like the center of a circle moving with the circle. A nonequivariant is a property that does not change predictably, such as the perimeter of a circle when transformed into an ellipse.
For object recognition, the class of an object is typically invariant over many transformations (a cat remains a cat when shifted or scaled), but other properties are equivariant, such as the volume of a cat when scaled. Equivariant properties, including spatial relationships, are captured in a pose, which describes translation, rotation, scale, and reflection.
Unsupervised CapsNets learn a global linear manifold between an object and its pose as a matrix of weights. This allows the network to identify an object independent of its pose, rather than learning to recognize the object with spatial relationships included. The pose can also incorporate non-spatial properties, such as color. Multiplying the object by the manifold poses the object in space.
Pooling
CapsNets reject the pooling layer strategy of conventional CNNs, which reduces the amount of detail processed at higher layers. Pooling provides translational invariance and allows more feature types, but CapsNet proponents argue that pooling has several drawbacks: it violates biological shape perception by lacking an intrinsic coordinate frame; it provides invariance (discarding positional information) instead of equivariance (disentangling that information); it ignores the linear manifold underlying image variations; it routes statically rather than communicating potential "finds" to features that can appreciate them; and it damages nearby feature detectors by deleting information they rely upon.
Instead of pooling, CapsNets use routing-by-agreement to dynamically direct information between layers, preserving detailed spatial information.
Capsules
A capsule is a set of neurons that individually activate for various properties of a type of object, such as position, size, and hue. Formally, a capsule produces an activity vector with one element per neuron, holding that neuron's instantiation value (e.g., hue). Graphics programs use instantiation values to draw objects, and CapsNets attempt to derive these from input. The probability of an entity's presence is the vector's length, while the vector's orientation quantifies the capsule's properties.
Traditional artificial neurons output a scalar activation, but CapsNets replace scalar-output feature detectors with vector-output capsules and max-pooling with routing-by-agreement. Because capsules are independent, when multiple capsules agree, the probability of correct detection is much higher. For example, a minimal cluster of two capsules considering a six-dimensional entity would agree within 10% by chance only once in a million trials; the likelihood of chance agreement decreases exponentially with more dimensions.
Higher-layer capsules take outputs from lower-layer capsules and accept those whose outputs cluster. A cluster causes the higher capsule to output a high probability of entity presence and a high-dimensional (20-50+) pose. Higher-level capsules ignore outliers, concentrating on clusters, similar to the Hough transform, RHT, and RANSAC from classic digital image processing.
Routing by agreement
Routing by agreement is the mechanism by which outputs from one capsule (child) are routed to capsules in the next layer (parent) based on the child's ability to predict the parents' outputs. Over a few iterations, each parent's output may converge with predictions from some children and diverge from others, indicating whether that parent is present or absent.
For each possible parent, each child computes a prediction vector by multiplying its output by a weight matrix trained by backpropagation. The parent's output is then computed as a weighted sum of predictions, where the weights represent the probability that a child belongs to that parent. A child whose predictions are close to the resulting output increases its coupling coefficient with that parent, while decreasing it for others. This iterative process allows the network to dynamically allocate resources and improve recognition accuracy.