# Activation Functions

Activation functions in artificial neural networks compute a node's output from its weighted inputs, introducing non-linearity that enables solving complex problems. Common examples include sigmoid, ReLU, tanh, GELU, and Swish.

In artificial neural networks, the activation function of a node is a function that calculates the output of the node based on its individual inputs and their weights. Nontrivial problems can be solved using only a few nodes if the activation function is nonlinear. Activation functions are a core component of [neural networks](https://www.wikiprompt.org/wiki/neural-network), determining how signals propagate through layers and enabling the network to learn complex patterns in data. They are used in virtually every modern [machine learning](https://www.wikiprompt.org/wiki/machine-learning) model, including [deep learning](https://www.wikiprompt.org/wiki/deep-learning) systems such as [large language models](https://www.wikiprompt.org/wiki/large-language-model) and [transformers](https://www.wikiprompt.org/wiki/transformer).

Activation functions can be categorized by their mathematical properties, such as nonlinearity, range, and differentiability, which influence training stability and efficiency. Over the years, researchers have developed numerous activation functions, each with specific advantages and trade-offs. The choice of activation function can significantly affect a model's performance, convergence speed, and ability to avoid issues like vanishing gradients.

## Historical Development

The concept of activation functions dates back to early artificial neuron models, such as the perceptron introduced by Frank Rosenblatt in 1958, which used a binary step activation. However, the step function is not differentiable, limiting its use in gradient-based optimization. In the 1980s, the logistic sigmoid function became popular because it is smooth and differentiable, enabling backpropagation. In 1986, David Rumelhart, Geoffrey Hinton, and Ronald Williams popularized backpropagation, which relies on differentiable activation functions.

The 2010s saw a shift toward rectified linear units (ReLU), which helped mitigate the vanishing gradient problem. ReLU was used in the 2012 AlexNet computer vision model, which won the ImageNet competition, and in the 2015 ResNet model, which enabled training of very deep networks. In 2018, the Gaussian Error Linear Unit (GELU) was introduced and used in the BERT model, a foundational [transformer](https://www.wikiprompt.org/wiki/transformer) architecture for natural language processing. More recently, Swish (also known as SiLU) was proposed in 2017 by researchers at Google, offering a smooth, non-monotonic alternative.

## Nonlinearity and Universal Approximation

A key property of activation functions is nonlinearity. When the activation function is non-linear, a two-layer neural network can be proven to be a universal function approximator, meaning it can approximate any continuous function on a compact domain to arbitrary accuracy. This is known as the Universal Approximation Theorem. The identity activation function does not satisfy this property; when multiple layers use the identity activation function, the entire network is equivalent to a single-layer model, losing the ability to represent complex functions. Nonlinear activation functions allow neural networks to learn non-linear relationships in data, which is essential for tasks like image recognition, speech processing, and language understanding.

## Range and Training Stability

Activation functions have different ranges, which affect gradient-based training. When the range of the activation function is finite, gradient-based training methods tend to be more stable, because pattern presentations significantly affect only limited weights. For example, the sigmoid function outputs values between 0 and 1, and the tanh function outputs between -1 and 1. When the range is infinite, training is generally more efficient because pattern presentations significantly affect most of the weights, but smaller learning rates are typically necessary to avoid instability. ReLU has an unbounded positive range, which can lead to faster learning but also potential issues like dead neurons.

## Differentiability and Gradient-Based Optimization

Continuously differentiable activation functions are desirable for enabling gradient-based optimization methods, as they allow the computation of gradients for backpropagation. ReLU is not continuously differentiable at zero, but it is still possible to use subgradients or define the derivative at zero as 0 or 1. The binary step activation function is not differentiable at 0 and differentiates to 0 for all other values, so gradient-based methods can make no progress with it. Non-saturating activation functions, such as ReLU, may be better than saturating activation functions because they are less likely to suffer from the vanishing gradient problem, where gradients become extremely small in deep networks.

## Common Activation Functions

### Sigmoid (Logistic)

The sigmoid function, also known as the logistic function, is defined as \( \sigma(x) = \frac{1}{1 + e^{-x}} \). It outputs values between 0 and 1, making it useful for binary classification and as an output layer for probabilities. However, it suffers from saturation at extreme values, leading to vanishing gradients. The sigmoid was used in the 2012 speech recognition model developed by Hinton et al.

### Tanh (Hyperbolic Tangent)

The tanh function is defined as \( \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} \), and it outputs values between -1 and 1. It is zero-centered, which often makes optimization easier than sigmoid. Tanh is commonly used in recurrent neural networks and as a hidden layer activation.

### ReLU (Rectified Linear Unit)

ReLU is defined as \( \text{ReLU}(x) = \max(0, x) \). It is computationally efficient and helps mitigate vanishing gradients, as its derivative is 1 for positive inputs. However, it can suffer from dead neurons, where units become inactive permanently. ReLU was used in AlexNet and ResNet.

### Leaky ReLU and Parametric ReLU

Leaky ReLU allows a small positive slope for negative inputs, such as \( \text{LeakyReLU}(x) = \max(0.01x, x) \), to avoid dead neurons. Parametric ReLU (PReLU) learns the slope parameter during training, offering more flexibility.

### GELU (Gaussian Error Linear Unit)

GELU is a smooth approximation of ReLU, defined as \( \text{GELU}(x) = x \cdot \Phi(x) \), where \( \Phi \) is the cumulative distribution function of the standard normal distribution. It was used in BERT and many modern transformers, providing smoother gradients and often better performance.

### Swish / SiLU

Swish, also known as SiLU (Sigmoid Linear Unit), is defined as \( \text{Swish}(x) = x \cdot \sigma(x) \). It is smooth and non-monotonic, which has been shown to improve performance in some deep networks. Swish was proposed by researchers at Google in 2017.

### Softplus

Softplus is a smooth approximation of ReLU, defined as \( \text{softplus}(x) = \ln(1 + e^x) \). It has a strictly positive range, making it suitable for predicting variances in variational autoencoders.

## Mathematical Details

The most common activation functions can be divided into three categories: ridge functions, radial functions, and fold functions. An activation function \( f \) is saturating if \( \lim_{|v| \to \infty} |\nabla f(v)| = 0 \), and nonsaturating if the limit is not zero. Non-saturating functions, such as ReLU, are less prone to vanishing gradients.

### Ridge Activation Functions

Ridge functions are multivariate functions acting on a linear combination of the input variables. Examples include:
- Linear activation: \( \phi(\mathbf{v}) = a + \mathbf{v}'\mathbf{b} \)
- ReLU activation: \( \phi(\mathbf{v}) = \max(0, a + \mathbf{v}'\mathbf{b}) \)
- Heaviside activation: \( \phi(\mathbf{v}) = 1_{a + \mathbf{v}'\mathbf{b} > 0} \)
- Logistic activation: \( \phi(\mathbf{v}) = (1 + \exp(-a - \mathbf{v}'\mathbf{b}))^{-1} \)

In biologically inspired neural networks, the activation function is usually an abstraction representing the rate of action potential firing in the cell. In its simplest form, this function is binary, meaning the neuron is either firing or not. Neurons cannot fire faster than a certain rate, motivating sigmoid activation functions whose output range is confined to a finite interval.

### Radial Activation Functions

A special class of activation functions known as radial basis functions (RBFs) are used in RBF networks. These activation functions can take many forms, but they are usually found as Gaussian functions: \( \phi(\mathbf{v}) = \exp(-\|\mathbf{v} - \mathbf{c}\|^2 / (2\sigma^2)) \), where \( \mathbf{c} \) is the center and \( \sigma \) controls the width. RBF networks use these local receptive fields for function approximation.

## Practical Considerations

When choosing an activation function, practitioners consider factors such as computational cost, gradient behavior, and the specific task. For deep networks, ReLU and its variants are often preferred due to their efficiency and ability to avoid vanishing gradients. For output layers, sigmoid is used for binary classification, softmax for multi-class classification, and linear activation for regression. In [transformers](https://www.wikiprompt.org/wiki/transformer), GELU is commonly used in feed-forward layers, as seen in models like BERT and GPT. The choice of activation function can also interact with other techniques such as [batch normalization](https://www.wikiprompt.org/wiki/batch-normalization) and [dropout](https://www.wikiprompt.org/wiki/dropout).

## Conclusion

Activation functions are fundamental to the operation of neural networks, providing the nonlinearity that enables learning complex patterns. From early step functions to modern smooth variants like GELU and Swish, they have evolved to address challenges in training deep models. Understanding their mathematical properties helps researchers and engineers select appropriate functions for specific applications, contributing to the success of [artificial intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) systems across domains.

---
Source: https://www.wikiprompt.org/wiki/activation-functions
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-13T03:59:38.607013+00:00