A restricted Boltzmann machine (RBM) is a generative stochastic artificial neural network that can learn a probability distribution over its set of inputs. It is a variant of the Boltzmann machine, with the restriction that its neurons form a bipartite graph: connections exist only between two groups of units, commonly called visible and hidden units, and there are no connections within each group. This restriction allows for more efficient training algorithms than are available for general Boltzmann machines, particularly the gradient-based contrastive divergence algorithm.
RBMs were initially proposed under the name Harmonium by Paul Smolensky in 1986, and rose to prominence after Geoffrey Hinton and collaborators developed fast learning algorithms for them in the mid-2000s. They have found applications in dimensionality reduction, classification, collaborative filtering, feature learning, topic modelling, immunology, and many-body quantum mechanics. RBMs can be trained in either supervised or unsupervised ways, depending on the task, and are a key component in deep learning networks, where deep belief networks can be formed by stacking RBMs and optionally fine-tuning the resulting network with gradient descent and backpropagation.
Structure
The standard type of RBM has binary-valued (Boolean) hidden and visible units. It consists of a matrix of weights \(W\) of size \(m \times n\), where each weight element \(w_{i,j}\) is associated with the connection between visible unit \(v_i\) and hidden unit \(h_j\). Additionally, there are bias weights \(a_i\) for visible units and \(b_j\) for hidden units. The energy of a configuration (pair of Boolean vectors) \((v, h)\) is defined as:
\[ E(v, h) = -\sum_i a_i v_i - \sum_j b_j h_j - \sum_i \sum_j v_i w_{i,j} h_j \]
or in matrix notation:
\[ E(v, h) = -a^{\mathrm{T}} v - b^{\mathrm{T}} h - v^{\mathrm{T}} W h. \]
This energy function is analogous to that of a Hopfield network. As with general Boltzmann machines, the joint probability distribution for the visible and hidden vectors is defined in terms of the energy function:
\[ P(v, h) = \frac{1}{Z} e^{-E(v, h)} \]
where \(Z\) is a partition function, a normalizing constant that ensures probabilities sum to 1. The marginal probability of a visible vector is the sum of \(P(v, h)\) over all possible hidden layer configurations, and vice versa.
Training and Contrastive Divergence
Because the underlying graph structure of an RBM is bipartite, hidden unit activations are mutually independent given the visible unit activations, and conversely, visible unit activations are mutually independent given the hidden unit activations. This property simplifies the computation of conditional probabilities, enabling efficient training. The most common training algorithm is contrastive divergence, which approximates the gradient of the log-likelihood by performing a short Markov chain Monte Carlo run, typically with one Gibbs sampling step. This approach avoids the intractable computation of the partition function \(Z\), which sums over all possible configurations.
RBMs can be trained in both supervised and unsupervised modes. In unsupervised learning, the model learns to represent the input data distribution, often for feature extraction or dimensionality reduction. In supervised tasks, the RBM can be adapted to predict labels by incorporating them into the visible layer or by using a separate output layer.
Applications in Deep Learning
Restricted Boltzmann machines are a foundational component in deep learning. A deep belief network can be formed by stacking multiple RBMs, where the hidden layer of one RBM serves as the visible layer for the next. This layer-wise pretraining allows the network to learn hierarchical features from data. After pretraining, the entire network can be fine-tuned using gradient descent and backpropagation, a technique that was influential in the development of modern deep learning architectures.
RBMs have been applied to collaborative filtering, where they model user-item interactions for recommendation systems, and to topic modelling, where they learn latent topics from text corpora. In immunology, they have been used to model immune system responses, and in quantum mechanics, they have been employed to represent many-body quantum states.
Relationship to Other Models
RBMs are closely related to other neural network models. They are a special case of Boltzmann machines, which allow connections between hidden units, but the bipartite restriction makes training more tractable. They also share similarities with Hopfield networks in their energy function, but RBMs are generative and stochastic, whereas Hopfield networks are typically deterministic and used for associative memory. In the broader context of Machine learning, RBMs are considered a type of generative model, distinct from discriminative models like standard feedforward neural networks.
Legacy and Influence
The development of RBMs contributed significantly to the resurgence of Deep learning in the mid-2000s. Geoffrey Hinton, who later became a prominent figure in Artificial intelligence, used RBMs to demonstrate that deep networks could be trained effectively, overcoming earlier difficulties with gradient-based methods. This work influenced subsequent architectures, including transformers and large language models, although those models use different training paradigms. RBMs remain an important theoretical model in the study of Generative AI and are still used in specialized applications where their probabilistic nature is advantageous.