A radial basis function network (RBF network) is an artificial neural network that uses radial basis functions as activation functions. The output of the network is a linear combination of radial basis functions of the inputs and neuron parameters. RBF networks have many uses, including function approximation, time series prediction, classification, and system control. They were first formulated in a 1988 paper by Broomhead and Lowe, both researchers at the Royal Signals and Radar Establishment.
RBF networks are a type of Neural network that differ from more common feedforward networks in their hidden layer structure and training approach. They are closely related to Machine learning methods such as kernel regression and Gaussian process models, and they have been applied in fields ranging from Artificial intelligence to control engineering.
Network Architecture
RBF networks typically have three layers: an input layer, a hidden layer with a non-linear radial basis function activation, and a linear output layer. The input is a vector of real numbers, and the output is a scalar function of that input. The output is computed as a weighted sum of radial basis functions, each centered at a specific point in the input space.
Each hidden neuron has a center vector and a width parameter. The activation of a neuron depends only on the distance between the input and its center, making the function radially symmetric. The Euclidean distance is commonly used, though the Mahalanobis distance can perform better for pattern recognition. The most common radial basis function is the Gaussian, which decays smoothly to zero as the distance from the center increases. This locality means that changing the parameters of one neuron has little effect on inputs far from its center.
Given mild conditions on the activation function, RBF networks are universal approximators on compact subsets of the input space. With enough hidden neurons, they can approximate any continuous function on a closed and bounded set with arbitrary precision.
The network parameters (weights, centers, and widths) are determined to optimize the fit between the network output and the training data.
Normalization
RBF networks can be normalized, in which case the output is a weighted average of the neuron outputs rather than a simple sum. This normalized architecture ensures that the output lies within a bounded range and can improve numerical stability. Normalization is particularly useful when the network is used for probability estimation or when the input distribution varies.
Training Methods
Training an RBF network typically involves two stages. First, the centers and widths of the hidden neurons are selected, often using unsupervised methods such as k-means clustering or random sampling from the training data. Second, the output weights are learned, usually by solving a linear least-squares problem, which is fast and has a unique solution.
This two-stage approach contrasts with the gradient-based training used in Deep learning networks, where all parameters are adjusted simultaneously. RBF networks can also be trained with gradient descent, but the linear output layer makes the weight estimation particularly efficient.
Applications
RBF networks are used in a variety of tasks. In function approximation, they can model complex nonlinear relationships. In time series prediction, they have been applied to financial and meteorological data. In classification, they serve as a type of Machine learning classifier, particularly in pattern recognition problems. They are also used in system control, where their fast training and smooth outputs are advantageous.
Compared to Deep learning models, RBF networks are often simpler and require less data, but they can struggle with high-dimensional inputs due to the curse of dimensionality. They have been used in fields such as Amazon Web Services for certain predictive tasks, though large-scale modern applications often favor deeper architectures.
Limitations and Extensions
One limitation of RBF networks is that the number of hidden neurons can grow exponentially with the input dimension, making them impractical for very high-dimensional problems. Extensions include adaptive selection of centers, use of different basis functions, and hybrid approaches that combine RBF layers with other network types.
Despite the rise of Transformer (architecture)-based models in Generative AI, RBF networks remain a fundamental tool in the neural network toolbox, valued for their theoretical clarity and ease of training.