ADALINE (Adaptive Linear Neuron, later Adaptive Linear Element) is an early single-layer artificial neural network and the name of the physical device that implemented it. It was developed by professor Bernard Widrow and his doctoral student Marcian Hoff at Stanford University in 1960. It is based on the perceptron and consists of weights, a bias, and a summation function. The weights and biases were implemented by rheostats (as seen in the "knobby ADALINE"), and later, memistors. It found extensive use in adaptive signal processing, especially of adaptive noise filtering.
The difference between ADALINE and the standard (Rosenblatt) perceptron is in how they learn. ADALINE unit weights are adjusted to match a teacher signal, before applying the Heaviside function, but the standard perceptron unit weights are adjusted to match the correct output, after applying the Heaviside function. A multilayer network of ADALINE units is known as a MADALINE.
Architecture and Operation
ADALINE is a single-layer neural network with multiple nodes, where each node accepts multiple inputs and generates one output. Given the input vector x, the weight vector w, the number of inputs N, and a bias b, the output o is computed as the weighted sum: o = sum_{n=1}^{N} x_n w_n + b. If we further assume that x_0 = 1 and w_0 = b, then the output reduces to o = sum_{n=0}^{N} x_n w_n.
The network's linear summation function distinguishes it from later models that introduced nonlinear activation functions within individual units. The physical implementation used adjustable resistors (rheostats) for weights, allowing real-time adaptation during training. Later versions employed memistors, a type of resistor with memory, which enabled more compact and stable weight storage.
Learning Rule
The learning rule used by ADALINE is the LMS ("least mean squares") algorithm, a special case of gradient descent. Given a learning rate eta, the model output o, and the desired target y, the square of the error is defined as E = (y - o)^2. The LMS algorithm updates the weights as: w <- w + eta (y - o) x. This update rule minimizes E, the square of the error, and is in fact the stochastic gradient descent update for linear regression.
This approach was a significant departure from the perceptron's error-correction rule, which adjusted weights based on the binary output after thresholding. By minimizing the continuous error before applying the Heaviside function, ADALINE could achieve smoother convergence and better performance in noisy environments, making it particularly suitable for adaptive filtering applications.
MADALINE
MADALINE (Many ADALINE) is a three-layer (input, hidden, output), fully connected, feedforward neural network architecture for classification that uses ADALINE units in its hidden and output layers. Its activation function is the sign function. The three-layer network uses memistors. As the sign function is non-differentiable, backpropagation cannot be used to train MADALINE networks. Hence, three different training algorithms have been suggested, called Rule I, Rule II and Rule III.
Despite many attempts, researchers never succeeded in training more than a single layer of weights in a MADALINE model. This was until Widrow saw the backpropagation algorithm at a 1985 conference in Snowbird, Utah.
MADALINE Rule 1 (MRI), dating back to 1962, consists of two layers: the first is made of ADALINE units; the second layer has two units. One is a majority-voting unit that takes in all outputs and outputs +1 if there are more positives than negatives, and vice versa. Another is a "job assigner": if the desired output differs from the majority-voted output, it calculates the minimal number of ADALINE units that must change their outputs, picks those closest to being negative, and updates their weights according to the ADALINE learning rule. This was thought of as a form of "minimal disturbance principle."
The largest MADALINE machine built had 1000 weights, each implemented by a memistor. It was built in 1963 and used MRI for learning. Some MADALINE machines were demonstrated to perform tasks including inverted pendulum balancing, weather forecasting, and speech recognition.
MADALINE Rule 2 (MRII), described in 1988, improved on Rule I. The Rule II training algorithm is based on a principle called "minimal disturbance." It proceeds by looping over training examples, and for each example, it finds the hidden layer unit with the lowest confidence in its prediction, tentatively flips the sign of the unit, accepts or rejects the change based on whether the network's error is reduced, and stops when the error is zero. Additionally, when flipping single units' signs does not drive the error to zero for a particular example, the training algorithm starts flipping pairs of units' signs, then triples of units, etc.
MADALINE Rule 3 applied to a modified network with sigmoid activations instead of sign; it was later found to be equivalent to backpropagation.
Legacy and Influence
ADALINE and its successor MADALINE were foundational in the development of neural networks and machine learning. The LMS algorithm introduced by Widrow became a cornerstone of adaptive signal processing and influenced later developments in deep learning. The concept of minimizing continuous error before thresholding anticipated modern approaches in training neural networks with differentiable loss functions.
Although ADALINE itself was limited to linear problems, its principles of weight adaptation and error minimization were carried forward into more complex architectures. The work at Stanford AI Lab in the 1960s helped establish the field of artificial neural networks, alongside contributions from Xerox PARC and other research institutions. The minimal disturbance principle later resurfaced in various forms in online learning algorithms.
The physical implementation of ADALINE using rheostats and memistors also demonstrated the feasibility of analog computing for neural networks, a concept that has seen renewed interest in modern neuromorphic hardware. Widrow's contributions are recognized as pivotal in the history of artificial intelligence, bridging early perceptron research and later transformer-based models.