A diffusion model is a generative model that learns to create data, most commonly images, video, or audio, by reversing a gradual noising process: it is trained to progressively remove noise from a corrupted sample, and once trained, can generate new data by starting from pure random noise and iteratively denoising it into a coherent output. Diffusion models are the technology behind most leading Text-to-image generation and Text-to-video generation systems as of the mid-2020s, having displaced Generative adversarial networks as the dominant approach for high-fidelity image synthesis.
History
The theoretical foundation was laid in a 2015 paper by Jascha Sohl-Dickstein and colleagues, which drew an analogy to nonequilibrium thermodynamics to describe a process of gradually destroying structure in data with noise and then learning to reverse it. The approach remained a research curiosity until Jonathan Ho, Ajay Jain, and Pieter Abbeel published Denoising Diffusion Probabilistic Models, or DDPM, in 2020, showing that a relatively simple training objective could produce image quality competitive with GANs. The technique reached mainstream attention through OpenAI's DALL-E 2 in April 2022, and became broadly accessible a few months later with the August 2022 open-source release of Stable Diffusion, built on latent diffusion, a 2022 technique introduced by Robin Rombach and colleagues that ran the denoising process in a compressed Latent space rather than directly on raw pixels, drastically cutting the compute needed to train and run diffusion models.
How it works
Training a diffusion model involves a forward process that adds small amounts of Gaussian noise to a real data sample over many steps until it becomes indistinguishable from pure noise, and a neural network, typically a U-Net or, increasingly, a Transformer (architecture)-based architecture, trained to predict and remove the noise added at each step, effectively learning the reverse process. Generation runs this reverse process from scratch: starting with random noise, the model repeatedly predicts and subtracts noise over a number of steps, gradually revealing a coherent image or video. Conditioning mechanisms, most commonly by encoding a text Prompt and feeding it into the denoising network via cross-attention, let users guide what the model generates, and techniques like classifier-free guidance strengthen how closely the output follows that conditioning signal.
Applications and ecosystem
Diffusion models power essentially every major image generator in wide use, including Stable Diffusion, DALL-E, Midjourney, and Flux, and underlie leading video generators such as Sora, Google's Veo, and Chinese systems including Kling and Seedance. Extensions such as ControlNet, introduced in 2023, add structural conditioning, letting users guide generation with sketches, poses, or depth maps, while ecosystems built around open checkpoints support extensive customization through techniques like LoRA adapters trained by the community. Diffusion techniques have also been applied beyond media generation, including to protein structure prediction and molecule design, and researchers have explored diffusion-based approaches to text generation as an alternative to the token-by-token autoregressive approach used by most Large language models, though autoregressive generation remains dominant for text as of 2025.
Limitations
Compared to GANs, diffusion models are computationally expensive at inference time because generating a single sample can require dozens or hundreds of sequential denoising steps, though techniques such as distillation into few-step models and improved samplers have substantially reduced this cost since 2022. Diffusion models trained on large, uncurated web-scraped datasets have also faced AI and copyright disputes over whether generating images in the style of, or resembling, artists in the Training data constitutes infringement, and have drawn scrutiny over their capacity to reproduce memorized training images verbatim under certain conditions.