# Connectionism

Connectionism is an approach to studying cognition using artificial neural networks, where mental phenomena emerge from interconnected simple units. It has evolved through multiple waves, from early perceptrons to modern deep learning and large language models.

Connectionism is an approach to the study of human mental processes and cognition that utilizes mathematical models known as connectionist networks or artificial neural networks. The central principle holds that mental phenomena can be described by interconnected networks of simple and often uniform units, where the form of connections and units can vary from model to model. For example, units in the network could represent neurons and connections could represent synapses, as in the human brain. This approach has been seen as an alternative to classical theories of mind based on symbolic computation, often referred to as good old-fashioned AI (GOFAI), though the extent to which the two approaches are compatible has been debated since their inception.

Connectionism has experienced several distinct waves of development since its beginnings. The first wave emerged in 1943 with the work of Warren Sturgis McCulloch and Walter Pitts, who focused on comprehending neural circuitry through a formal and mathematical approach. The second wave blossomed in the late 1980s with the introduction of hidden layers and sigmoid activation functions. The current third wave has been marked by advances in [deep-learning](https://www.wikiprompt.org/wiki/deep-learning), which have made possible the creation of [large language models](https://www.wikiprompt.org/wiki/large-language-model). The success of deep-learning networks in recent decades has greatly increased the popularity of this approach, but the complexity and scale of such networks has brought increased interpretability problems.

## Basic principle

The central connectionist principle is that mental phenomena can be described by interconnected networks of simple and often uniform units. The form of the connections and the units can vary from model to model. For example, units in the network could represent neurons and the connections could represent synapses, as in the human brain. This principle has been seen as an alternative to GOFAI and the classical theories of mind based on symbolic computation, but the extent to which the two approaches are compatible has been the subject of much debate since their inception.

Neural networks follow two basic principles. Any mental state can be described as an n-dimensional vector of numeric activation values over neural units in a network. Memory and learning are created by modifying the weights of the connections between neural units, generally represented as an n×m matrix. The weights are adjusted according to some learning rule or algorithm, such as Hebbian learning.

Most of the variety among connectionist models comes from three sources. The interpretation of units can vary, with units interpreted as neurons or groups of neurons. The definition of activation can differ; for example, in a Boltzmann machine, activation is interpreted as the probability of generating an action potential spike and is determined via a logistic function on the sum of inputs to a unit. The learning algorithm also varies, with any mathematically defined change in connection weights over time referred to as the learning algorithm.

## Activation and learning

Internal states of any network change over time due to neurons sending a signal to a succeeding layer of neurons in the case of a feedforward network, or to a previous layer in the case of a recurrent network. Discovery of non-linear activation functions enabled the second wave of connectionism. Modern networks commonly use functions such as the sigmoid, ReLU, or variants thereof, which allow networks to model complex non-linear relationships.

Learning in connectionist networks typically involves adjusting connection weights according to a learning rule. The most influential of these has been backpropagation, which computes gradients of error with respect to weights and updates them accordingly. Optimization techniques such as [adam-optimizer](https://www.wikiprompt.org/wiki/adam-optimizer) and other [sgd-variants](https://www.wikiprompt.org/wiki/sgd-variants) have become standard in training deep networks. Other techniques like [dropout](https://www.wikiprompt.org/wiki/dropout), [batch-normalization](https://www.wikiprompt.org/wiki/batch-normalization), and [layer-normalization](https://www.wikiprompt.org/wiki/layer-normalization) help stabilize and regularize training.

## Biological realism

Connectionist work in general does not need to be biologically realistic. One area where connectionist models are thought to be biologically implausible is with respect to error-propagation networks that are needed to support learning. However, error propagation can explain some of the biologically-generated electrical activity seen at the scalp in event-related potentials such as the N400 and P600, providing some biological support for one of the key assumptions of connectionist learning procedures.

Many recurrent connectionist models also incorporate dynamical systems theory. Many researchers, such as the connectionist Paul Smolensky, have argued that connectionist models will evolve toward fully continuous, high-dimensional, non-linear, dynamic systems approaches. The field of [neural network](https://www.wikiprompt.org/wiki/neural-network) research continues to draw inspiration from neuroscience while also pursuing engineering goals.

## Precursors

Precursors of the connectionist principles can be traced to early work in psychology, such as that of William James. Psychological theories based on knowledge about the human brain were fashionable in the late 19th century. As early as 1869, the neurologist John Hughlings Jackson argued for multi-level, distributed systems. Following from this lead, Herbert Spencer's Principles of Psychology, 3rd edition (1872), and Sigmund Freud's Project for a Scientific Psychology (composed 1895) propounded connectionist or proto-connectionist theories. These tended to be speculative theories. By the early 20th century, Edward Thorndike was writing about human learning that posited a connectionist type network.

Hopfield networks had precursors in the Ising model due to Wilhelm Lenz (1920) and Ernst Ising (1925), though the Ising model conceived by them did not involve time. Monte Carlo simulations of the Ising model required the advent of computers in the 1950s.

## The first wave

The first wave began in 1943 with Warren Sturgis McCulloch and Walter Pitts, who focused on comprehending neural circuitry through a formal and mathematical approach. Their classic paper "A Logical Calculus of Ideas Immanent in Nervous Activity" (1943) showed how neural systems could implement first-order logic. They were influenced by the work of Nicolas Rashevsky in the 1930s.

Frank Rosenblatt published the 1958 paper "The Perceptron: A Probabilistic Model For Information Storage and Organization in the Brain" in Psychological Review while working at the Cornell Aeronautical Laboratory. The perceptron was a simple single-layer network that could classify linearly separable patterns. The first wave ended with the 1969 book Perceptrons about limitations of the original perceptron idea, written by Marvin Minsky and Seymour Papert, which contributed to discouraging major funding agencies in the US from investing in connectionist research. With a few noteworthy deviations, most connectionist research entered a period of inactivity until the mid-1980s. The term connectionist model was reintroduced in a 1982 paper in the journal Cognitive Science by Jerome Feldman and Dana Ballard.

## The second wave

The second wave blossomed in the late 1980s, following a 1987 book Parallel Distributed Processing by James L. McClelland, David E. Rumelhart, and colleagues, which introduced improvements to the simple perceptron idea. These included intermediate processors now known as hidden layers alongside input and output units, and the use of a sigmoid activation function instead of the old all-or-nothing function. Their work built upon that of John Hopfield, who was a key figure investigating the mathematical characteristics of sigmoid activation functions.

From the late 1980s to the mid-1990s, connectionism took on an almost revolutionary tone when researchers such as Schneider, Terence Horgan, and Tienson posed the question of whether connectionism represented a fundamental shift in psychology and so-called good old-fashioned AI. Some advantages of the second wave connectionist approach included its applicability to a broad array of functions, structural approximation to biological neurons, low requirements for innate structure, and capacity for graceful degradation. Its disadvantages included the difficulty in deciphering how artificial neural networks process information or account for the compositionality of mental representations, and a resultant difficulty explaining phenomena at a higher level.

## The third wave and deep learning

The current third wave has been marked by advances in deep learning, which have made possible the creation of large language models. The success of deep-learning networks in the past decade has greatly increased the popularity of this approach, but the complexity and scale of such networks has brought with them increased interpretability problems. Modern architectures such as the [transformer](https://www.wikiprompt.org/wiki/transformer) have driven progress in [generative-ai](https://www.wikiprompt.org/wiki/generative-ai), with models developed at organizations including [openai](https://www.wikiprompt.org/wiki/openai), [anthropic](https://www.wikiprompt.org/wiki/anthropic), and [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind). These systems build on connectionist principles but operate at scales far beyond early networks, with billions of parameters trained on vast datasets using specialized hardware from companies like [nvidia](https://www.wikiprompt.org/wiki/nvidia) and [amd](https://www.wikiprompt.org/wiki/amd).

## External links

- [Wikipedia: Connectionism](https://en.wikipedia.org/wiki/Connectionism)

---
Source: https://www.wikiprompt.org/wiki/connectionism
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-14T04:27:35.343721+00:00