In artificial neural networks, the activation function of a node is a function that calculates the output of the node based on its individual inputs and their weights. It is a core component of neural networks, determining whether a neuron should be activated and introducing the nonlinearity necessary for the network to learn complex patterns. Without a nonlinear activation function, a multi-layer network would collapse into a single linear transformation, severely limiting its expressive power. Activation functions are used in various architectures, including transformers, residual networks, and convolutional neural networks, and are central to disciplines such as Machine learning and Deep learning.
The choice of activation function significantly influences training dynamics and performance. Modern examples include the logistic (sigmoid) function, used in the 2012 speech recognition model developed by Hinton et al.; the rectified linear unit (ReLU), used in the 2012 AlexNet computer vision model and the 2015 ResNet model; and the Gaussian error linear unit (GELU), a smooth version of ReLU used in the 2018 BERT model. These functions differ in mathematical properties such as nonlinearity, range, and differentiability, which affect their behavior in gradient-based optimization.
Mathematical Properties
Activation functions can be characterized by several key mathematical properties that influence their effectiveness in training neural networks.
Nonlinearity
When the activation function is nonlinear, a two-layer neural network can be proven to be a universal function approximator, meaning it can approximate any continuous function on a compact domain to arbitrary accuracy. This is known as the Universal Approximation Theorem. The identity activation function does not satisfy this property; if all layers use identity activation, the entire network is equivalent to a single-layer model, incapable of learning complex mappings.
Range
When the range of the activation function is finite, gradient-based training methods tend to be more stable because pattern presentations significantly affect only limited weights. In contrast, when the range is infinite, training is generally more efficient because pattern presentations affect most weights, but smaller learning rates are typically necessary to prevent divergence. For example, the sigmoid function has a finite range (0, 1), while ReLU has an infinite range [0, ∞).
Continuously Differentiable
Continuously differentiable activation functions are desirable for enabling gradient-based optimization methods. ReLU is not continuously differentiable at zero, yet gradient-based methods still work in practice, though it can cause issues such as dead neurons. The binary step activation function is not differentiable at 0 and has zero gradient elsewhere, so gradient-based methods make no progress with it. These properties do not conclusively determine performance but are important considerations.
Categories of Activation Functions
Most common activation functions can be divided into three categories: ridge functions, radial functions, and fold functions.
Ridge Activation Functions
Ridge functions are multivariate functions acting on a linear combination of input variables. Examples include:
- Linear activation: \(\phi(\mathbf{v}) = a + \mathbf{v}'\mathbf{b}\)
- ReLU activation: \(\phi(\mathbf{v}) = \max(0, a + \mathbf{v}'\mathbf{b})\)
- Heaviside activation: \(\phi(\mathbf{v}) = 1_{a+\mathbf{v}'\mathbf{b}>0}\)
- Logistic activation: \(\phi(\mathbf{v}) = (1 + \exp(-a - \mathbf{v}'\mathbf{b}))^{-1}\)
In biologically inspired neural networks, the activation function often represents the firing rate of a neuron. The simplest form is binary: a neuron either fires or it does not, modeled by the Heaviside step function. Real neurons cannot fire faster than a certain rate, motivating sigmoid functions with finite output ranges. A linear function with positive slope can represent increase in firing rate with input current.
Radial Activation Functions
A special class of activation functions known as radial basis functions (RBFs) are used in radial basis function networks. These functions depend on the distance from a center vector \(\mathbf{c}\). Common forms include:
- Gaussian: \(\phi(\mathbf{v}) = \exp(-\|\mathbf{v} - \mathbf{c}\|^2 / (2\sigma^2))\)
- Multiquadratics: \(\phi(\mathbf{v}) = \sqrt{\|\mathbf{v} - \mathbf{c}\|^2 + a^2}\)
- Inverse multiquadratics: \(\phi(\mathbf{v}) = (\|\mathbf{v} - \mathbf{c}\|^2 + a^2)^{-1/2}\)
- Polyharmonic splines
Here, \(\sigma\) and \(a\) control the spread of the function.
Folding Activation Functions
Folding activation functions aggregate inputs, such as taking the mean, minimum, or maximum, and are extensively used in Pooling layers and output layers for multiclass classification. In the latter, the Softmax Function activation is commonly used to produce a probability distribution over classes.
Other Examples
Periodic functions can serve as activation functions, typically the sinusoid, since any periodic function can be decomposed into sinusoids via the Fourier transform. Quadratic activation functions map \(x \mapsto x^2\).
Saturating and Non-Saturating Functions
An activation function \(f\) is saturating if the limit of the gradient norm goes to zero as the input magnitude goes to infinity, i.e., \(\lim_{|v| \to \infty} |\nabla f(v)| = 0\). It is nonsaturating if this limit is not zero. Non-saturating functions like ReLU are often preferred because they are less likely to suffer from the vanishing gradient problem, which hinders training in deep networks.
Quantum Activation Functions
In quantum neural networks programmed on gate-model quantum computers, based on quantum perceptrons rather than variational quantum circuits, the nonlinearity of activation functions can be implemented without measuring the output of each perceptron per layer. By creating Taylor series expansions of the argument computed by the perceptron, and using suitable quantum circuits to compute powers up to a desired degree, the superposition property can be preserved. These quantum circuits can be designed to approximate any arbitrary classical activation function.
See Also
- Logistic function
- Rectifier (neural networks)
- Stability (learning theory)
- Softmax function