A multilayer perceptron (MLP) is a type of feedforward neural network composed of fully connected neurons organized into layers, with nonlinear activation functions. Unlike single-layer perceptrons, which can only classify linearly separable data, MLPs can distinguish data that is not linearly separable, making them a foundational architecture in deep learning. Modern MLPs are trained using backpropagation and are often colloquially referred to as "vanilla" neural networks.
The MLP grew out of efforts to overcome the limitations of the single-layer perceptron, which used a Heaviside step function as its activation. Backpropagation, however, requires continuous activation functions such as sigmoid or rectified linear unit (ReLU). MLPs form the basis of deep learning and are applicable across a vast set of domains, including image recognition, natural language processing, and time-series forecasting.
Historical Development
The concept of artificial neurons dates to 1943, when Warren McCulloch and Walter Pitts proposed a binary neuron as a logical model of biological neural networks. In 1958, Frank Rosenblatt introduced the multilayered perceptron model, which had an input layer, a hidden layer with randomized weights that did not learn, and an output layer with learnable connections. Rosenblatt's 1962 book Principles of Neurodynamics described variants with up to two trainable layers by "back-propagating errors," but this was not the modern backpropagation algorithm, and no general method for training multiple layers existed then.
In 1965, Alexey Grigorevich Ivakhnenko and Valentin Lapa published the Group Method of Data Handling, one of the first deep learning methods, used to train an eight-layer neural net in 1971. In 1967, Shun'ichi Amari reported the first multilayered neural network trained by stochastic gradient descent, capable of classifying non-linearly separable pattern classes; his student Saito conducted computer experiments using a five-layered feedforward network with two learning layers.
Backpropagation was independently developed multiple times in the early 1970s. The earliest published instance was Seppo Linnainmaa's master's thesis in 1970. Paul Werbos developed it independently in 1971, though he had difficulty publishing until 1982. In 1986, David E. Rumelhart and colleagues popularized backpropagation, leading to widespread adoption. Interest in backpropagation networks resurged in 2003 due to deep learning successes in language modeling by Yoshua Bengio and co-authors.
Architecture and Activation Functions
An MLP consists of three or more layers: an input layer, an output layer, and one or more hidden layers. Each node in one layer connects with a certain weight to every node in the following layer, making the network fully connected. If all neurons used linear activation functions, linear algebra shows that any number of layers could be reduced to a two-layer input-output model. Therefore, MLPs use nonlinear activation functions, which were developed to model the firing frequency of biological neurons.
Historically, two common activation functions were sigmoids: the hyperbolic tangent, which ranges from -1 to 1, and the logistic function, which ranges from 0 to 1. More recently, the rectified linear unit (ReLU) has become prevalent in deep learning, as it helps overcome numerical problems associated with sigmoids. Other alternatives include softplus and radial basis functions, the latter used in radial basis networks.
Learning via Backpropagation
Learning in an MLP occurs by adjusting connection weights after processing each data point, based on the error between the output and the expected result. This is supervised learning, carried out through backpropagation, a generalization of the least mean squares algorithm used in linear perceptrons. The error for an output node j on the nth training example is defined as e_j(n) = d_j(n) - y_j(n), where d_j(n) is the desired target and y_j(n) is the actual output. The algorithm propagates this error backward through the network to update weights, typically using optimization methods like stochastic gradient descent.
Modern Variants and Impact
In 2021, researchers designed MLP-Mixer, a simple architecture combining two deep MLPs with skip connections and layer normalizations. Its realizations, with 19 to 431 million parameters, performed comparably to vision transformers of similar size on ImageNet and similar image classification tasks. This showed that MLPs remain competitive in modern deep learning, despite the rise of Transformer (architecture) models. MLPs are also integral to many Neural network systems, including components of Large language model architectures and Deep learning frameworks. Their simplicity and effectiveness make them a standard baseline in Machine learning research and applications.
Limitations and Extensions
While MLPs are powerful, they have limitations. They are fully connected, leading to many parameters and computational cost for high-dimensional inputs like images. They also do not inherently capture spatial or sequential structure, which motivated architectures like Residual Network (ResNet) and Sequence-to-Sequence (Seq2Seq) models. Techniques such as Dropout, Batch Normalization, and Layer Normalization are often applied to improve training stability and generalization. Despite these challenges, MLPs remain a fundamental building block in Artificial intelligence and continue to inform the design of more complex architectures.