A deep belief network (DBN) is a generative graphical model, or alternatively a class of deep neural network, composed of multiple layers of latent variables ("hidden units"), with connections between layers but not between units within each layer. When trained without supervision on a set of examples, a DBN can learn to probabilistically reconstruct its inputs, with layers acting as feature detectors. After this learning step, a DBN can be further trained with supervision to perform classification.
DBNs can be viewed as a composition of simple, unsupervised networks such as restricted Boltzmann machines (RBMs) or autoencoders, where each sub-network's hidden layer serves as the visible layer for the next. An RBM is an undirected, generative energy-based model with a visible input layer and a hidden layer, with connections between but not within layers. This composition leads to a fast, layer-by-layer unsupervised training procedure, where contrastive divergence is applied to each sub-network in turn, starting from the lowest pair of layers (the lowest visible layer is a training set).
The observation that DBNs can be trained greedily, one layer at a time, led to one of the first effective deep learning algorithms. Overall, there are many attractive implementations and uses of DBNs in real-life applications and scenarios, such as electroencephalography and drug discovery.
Historical Context
Deep belief networks emerged in the mid-2000s as a breakthrough in training deep architectures. Prior to their introduction, training multi-layer neural networks was notoriously difficult due to issues like vanishing gradients and slow convergence. The greedy layer-wise pretraining strategy introduced with DBNs provided a practical way to initialize deep networks, which could then be fine-tuned for supervised tasks. This work, primarily associated with Geoffrey Hinton and colleagues at the University of Toronto, revitalized interest in Deep learning and laid groundwork for later advances in Neural network architectures.
The development of DBNs is often cited as a key milestone in the history of Artificial intelligence, bridging earlier work on probabilistic models and Machine learning with modern deep learning. The success of DBNs in tasks like handwritten digit recognition and document classification demonstrated the power of hierarchical feature learning, influencing subsequent models such as deep autoencoders and convolutional networks.
Training with Contrastive Divergence
The training method for RBMs proposed by Geoffrey Hinton for use with training "Product of Experts" models is called contrastive divergence (CD). CD provides an approximation to the maximum likelihood method that would ideally be applied for learning the weights. In training a single RBM, weight updates are performed with gradient descent using the equation:
w_ij(t+1) = w_ij(t) + η * ∂log(p(v))/∂w_ij
where p(v) is the probability of a visible vector, given by p(v) = (1/Z) * Σ_h e^(-E(v,h)), with Z as the partition function for normalization and E(v,h) as the energy function assigned to the state of the network. A lower energy indicates a more "desirable" configuration. The gradient ∂log(p(v))/∂w_ij has the simple form ⟨v_i h_j⟩_data - ⟨v_i h_j⟩_model, where ⟨...⟩_p represents averages with respect to distribution p.
The challenge arises in sampling ⟨v_i h_j⟩_model because it requires extended alternating Gibbs sampling. CD replaces this step by running alternating Gibbs sampling for n steps (values of n=1 perform well). After n steps, the data are sampled and that sample is used in place of ⟨v_i h_j⟩_model. The CD procedure works as follows:
- Initialize the visible units to a training vector.
- Update the hidden units in parallel given the visible units: p(h_j=1|V) = σ(b_j + Σ_i v_i w_ij), where σ is the sigmoid function and b_j is the bias of hidden unit j.
- Reconstruct the visible units from the hidden units, then update the hidden units again based on the reconstruction.
- Perform the weight update using the difference between the original and reconstructed correlations.
This procedure is applied greedily, layer by layer, to train a DBN. After pretraining, the entire network can be fine-tuned using backpropagation or other supervised methods.
Applications and Impact
Deep belief networks have been applied across various domains. In healthcare, DBNs have been used for analyzing electroencephalography (EEG) signals, aiding in the detection of neurological conditions. In drug discovery, they have been employed to predict molecular properties and identify potential drug candidates, leveraging their ability to learn hierarchical representations from chemical data.
DBNs also found use in speech recognition, where they served as acoustic models, and in image recognition tasks, where they learned features from raw pixels. Their success in these areas demonstrated the versatility of generative pretraining, influencing later developments in Generative AI and large-scale models.
Despite being overshadowed by newer architectures like Transformer (architecture) models, DBNs remain an important educational tool for understanding deep learning principles. They are often studied in courses on Machine learning and Deep learning as a foundational example of unsupervised pretraining and hierarchical feature learning.
Limitations and Evolution
DBNs have several limitations. Training can be computationally intensive, especially for large datasets, and the contrastive divergence approximation can lead to biased estimates. Additionally, as generative models, they are less scalable than discriminative models for certain tasks. The rise of Transformer (architecture)-based models, particularly in natural language processing, shifted focus away from DBNs, as these new architectures offered better performance with end-to-end training.
Nevertheless, the principles underlying DBNs - such as layer-wise pretraining and energy-based modeling - continue to influence modern research. Concepts like autoencoders and variational autoencoders build on similar ideas, and the greedy pretraining strategy has been adapted in various forms in contemporary deep learning pipelines.
Legacy and Further Reading
Deep belief networks represent a pivotal moment in the evolution of Artificial intelligence. They demonstrated that deep architectures could be trained effectively, paving the way for the deep learning revolution. Researchers at institutions like MIT CSAIL, Stanford AI Lab, and Berkeley AI Research have built upon these foundations, contributing to the rapid progress in the field.
For those interested in exploring DBNs further, the original papers by Hinton and his collaborators provide detailed derivations and experimental results. Textbooks on Deep learning often include chapters on DBNs and RBMs, offering both theoretical and practical perspectives. As the field continues to evolve, the lessons learned from DBNs remain relevant, particularly in areas like unsupervised learning and generative modeling.