Diffusion models, also known as diffusion-based generative models or score-based generative models, are a class of latent variable generative models in Machine learning. They consist of two main components: a forward diffusion process that gradually adds noise to data, and a reverse sampling process that learns to denoise. The goal is to learn a diffusion process for a given dataset such that new elements can be generated that are distributed similarly to the original data. A trained diffusion model can be sampled in various ways, with different trade-offs in efficiency and quality.
Diffusion models were introduced in 2015, drawing on techniques from non-equilibrium thermodynamics. The core idea is to model the distribution of data as a "cloud" in a high-dimensional space. By repeatedly adding noise, this cloud diffuses outward until it becomes nearly indistinguishable from a Gaussian distribution. A model trained to approximately undo this diffusion can then generate new samples from the original distribution by starting from random noise and iteratively denoising.
Formalisms and Training
There are several equivalent formalisms for diffusion models, including Markov chains, denoising diffusion probabilistic models (DDPM), noise-conditioned score networks, and stochastic differential equations. They are typically trained using variational inference. The model responsible for denoising is called the "backbone," which can be any architecture but is commonly a U-Net or a Transformer (architecture). The 2020 DDPM paper improved upon earlier methods by applying variational inference to the diffusion process.
Applications in Computer Vision
As of 2024, diffusion models are mainly used for computer vision tasks, including image denoising, inpainting, super-resolution, image generation, and video generation. These tasks typically involve training a neural network to sequentially denoise images blurred with Gaussian noise. After training to convergence, the model can generate images by starting with random noise and applying the network iteratively to remove noise. Diffusion-based image generators have seen widespread commercial interest, with notable examples including Stable Diffusion and DALL-E. These systems often combine diffusion models with text encoders and cross-attention modules to enable text-conditioned generation.
Other Domains
Beyond computer vision, diffusion models have found applications in natural language processing, such as text generation and summarization, as well as in sound generation and reinforcement learning. Their versatility has made them a key area of research in Generative AI.
Relationship to Other Generative Models
Diffusion models are part of a broader landscape of generative models, which also includes Large language models and other approaches. While large language models typically rely on autoregressive or transformer-based architectures, diffusion models offer an alternative paradigm that is particularly effective for continuous data like images and audio. The choice of backbone, whether a U-Net or a transformer, allows diffusion models to leverage advances in Deep learning and Neural network research.
Future Directions
Research continues to improve the efficiency and quality of diffusion models, exploring new sampling methods, backbone architectures, and applications. The integration of diffusion models with other AI systems, such as those developed by OpenAI, Google DeepMind, and Anthropic, is likely to drive further innovation in Artificial intelligence.