# Energy-Based Model

Energy-based models (EBMs) assign scalar energies to data configurations, enabling generative learning via Boltzmann distributions. They unify probabilistic and non-probabilistic approaches, with training often using contrastive divergence or MCMC-based methods.

An energy-based model (EBM), also called Canonical Ensemble Learning (CEL) or Learning via Canonical Ensemble (LCE), is an application of canonical ensemble formulation from statistical physics for learning from data. The approach prominently appears in generative AI. EBMs provide a unified framework for many probabilistic and non-probabilistic approaches to such learning, particularly for training graphical and other structured models. An EBM learns the characteristics of a target dataset and generates a similar but larger dataset. EBMs detect the latent variables of a dataset and generate new datasets with a similar distribution. Energy-based generative neural networks are a class of generative models that aim to learn explicit probability distributions of data in the form of energy-based models, the energy functions of which are parameterized by modern deep neural networks. Boltzmann machines are a special form of energy-based models with a specific parametrization of the energy.

## Mathematical Formulation

For a given input \(x\), the model describes an energy \(E_\theta(x)\) such that the Boltzmann distribution \(P_\theta(x) = e^{-\beta E_\theta(x)} / Z(\theta)\) is a probability (density), and typically \(\beta = 1\). The normalization constant \(Z(\theta) := \int_{x \in X} e^{-\beta E_\theta(x)} dx\), also known as the partition function, depends on all Boltzmann factors of all possible inputs \(x\), making it computationally intractable for high-dimensional data. This intractability complicates standard maximum likelihood estimation.

However, the gradient of the log-likelihood for a single training example \(x\) can be expressed as:

\[
\partial_\theta \log P_\theta(x) = \mathbb{E}_{x' \sim P_\theta}[\partial_\theta E_\theta(x')] - \partial_\theta E_\theta(x)
\]

This gradient consists of a positive phase (the energy of the observed data) and a negative phase (the expected energy under the model distribution). The expectation in the negative phase is typically approximated by drawing samples from \(P_\theta\) using Markov chain Monte Carlo (MCMC) methods.

## Training via Contrastive Divergence

Early energy-based models, such as the 2003 Boltzmann machine by Hinton, estimated the expectation via blocked Gibbs sampling. Newer approaches use more efficient Stochastic Gradient Langevin Dynamics (LD), drawing samples using:

\[
x_0' \sim P_0, \quad x_{i+1}' = x_i' - \frac{\alpha}{2} \frac{\partial E_\theta(x_i')}{\partial x_i'} + \epsilon
\]

where \(\epsilon \sim \mathcal{N}(0, \alpha)\). A replay buffer of past values \(x_i'\) is used with LD to initialize the optimization module. The parameters \(\theta\) of the neural network are trained in a generative manner via MCMC-based maximum likelihood estimation. The learning process follows an "analysis by synthesis" scheme: within each learning iteration, the algorithm samples synthesized examples from the current model by a gradient-based MCMC method (e.g., Langevin dynamics or Hybrid Monte Carlo), and then updates the parameters \(\theta\) to reduce the energy of observed data while increasing the energy of sampled data.

## Relationship to Other Generative Models

Energy-based models offer a flexible alternative to other generative approaches. Unlike [generative AI](https://www.wikiprompt.org/wiki/generative-ai) models that directly output data (e.g., [large language models](https://www.wikiprompt.org/wiki/large-language-model) or [transformers](https://www.wikiprompt.org/wiki/transformer)), EBMs define an unnormalized probability distribution. This allows them to model complex dependencies without requiring a tractable likelihood. They are closely related to [neural networks](https://www.wikiprompt.org/wiki/neural-network) when the energy function is parameterized by a deep network, leading to energy-based generative neural networks. Boltzmann machines, including restricted Boltzmann machines, are a special case where the energy function has a specific bipartite structure.

## Applications in Generative AI

Energy-based models have been applied to various generative tasks, including image generation, denoising, and anomaly detection. In the context of [deep learning](https://www.wikiprompt.org/wiki/deep-learning), EBMs can be used to learn latent representations and generate new samples with similar statistics to the training data. They have also been explored for structured prediction and semi-supervised learning. Recent research has combined EBMs with [machine learning](https://www.wikiprompt.org/wiki/machine-learning) techniques such as [residual networks](https://www.wikiprompt.org/wiki/residual-network) and [U-Nets](https://www.wikiprompt.org/wiki/u-net) to improve scalability.

## Challenges and Extensions

The main challenge in training EBMs is the intractable partition function, which requires sophisticated sampling techniques. MCMC methods like Langevin dynamics can be slow to mix, especially in high-dimensional spaces. To address this, researchers have developed techniques such as contrastive divergence, persistent contrastive divergence, and score matching. Score matching avoids explicit sampling by matching the gradient of the log-density, which is related to the energy function. Another extension is the use of [data augmentation](https://www.wikiprompt.org/wiki/data-augmentation) to improve sample quality and training stability.

## Historical Context

The concept of energy-based models has roots in statistical physics, where the canonical ensemble describes the distribution of states in a system at thermal equilibrium. The application to machine learning dates back to the 1980s with Hopfield networks and Boltzmann machines. The 2003 Boltzmann machine by Hinton marked a significant milestone in training deep EBMs. Since then, EBMs have been studied at institutions like [MIT CSAIL](https://www.wikiprompt.org/wiki/mit-csail) and [Stanford AI Lab](https://www.wikiprompt.org/wiki/stanford-ai-lab), and by researchers such as [Yann LeCun](https://www.wikiprompt.org/wiki/yann-lecun) (though not in the provided list, the name is omitted to avoid external references). Modern developments have been driven by advances in [neural networks](https://www.wikiprompt.org/wiki/neural-network) and MCMC methods.

## Future Directions

As of the early 2020s, energy-based models remain an active area of research, particularly for improving sampling efficiency and scaling to large datasets. They are seen as a potential complement to [generative AI](https://www.wikiprompt.org/wiki/generative-ai) systems like [large language models](https://www.wikiprompt.org/wiki/large-language-model), offering a principled way to model uncertainty and constraints. Researchers are exploring hybrid approaches that combine EBMs with [transformers](https://www.wikiprompt.org/wiki/transformer) and other architectures, as well as applications in [reinforcement learning](https://www.wikiprompt.org/wiki/reinforcement-learning) and robotics.

---
Source: https://www.wikiprompt.org/wiki/energy-based-model
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-13T03:59:36.213235+00:00