# Diffusion model

A diffusion model is a generative model that creates data by learning to reverse a gradual noising process, and is the core technique behind most modern text-to-image and text-to-video systems.

A diffusion model is a generative model that learns to create data, most commonly images, video, or audio, by reversing a gradual noising process: it is trained to progressively remove noise from a corrupted sample, and once trained, can generate new data by starting from pure random noise and iteratively denoising it into a coherent output. Diffusion models are the technology behind most leading [text-to-image](https://www.wikiprompt.org/wiki/text-to-image) and [text-to-video](https://www.wikiprompt.org/wiki/text-to-video) systems as of the mid-2020s, having displaced [generative-adversarial-network](https://www.wikiprompt.org/wiki/generative-adversarial-network)s as the dominant approach for high-fidelity image synthesis.

## History

The theoretical foundation was laid in a 2015 paper by Jascha Sohl-Dickstein and colleagues, which drew an analogy to nonequilibrium thermodynamics to describe a process of gradually destroying structure in data with noise and then learning to reverse it. The approach remained a research curiosity until Jonathan Ho, Ajay Jain, and Pieter Abbeel published Denoising Diffusion Probabilistic Models, or DDPM, in 2020, showing that a relatively simple training objective could produce image quality competitive with GANs. The technique reached mainstream attention through OpenAI's [dall-e](https://www.wikiprompt.org/wiki/dall-e) 2 in April 2022, and became broadly accessible a few months later with the August 2022 open-source release of [stable-diffusion](https://www.wikiprompt.org/wiki/stable-diffusion), built on latent diffusion, a 2022 technique introduced by [robin-rombach](https://www.wikiprompt.org/wiki/robin-rombach) and colleagues that ran the denoising process in a compressed [latent-space](https://www.wikiprompt.org/wiki/latent-space) rather than directly on raw pixels, drastically cutting the compute needed to train and run diffusion models.

## How it works

Training a diffusion model involves a forward process that adds small amounts of Gaussian noise to a real data sample over many steps until it becomes indistinguishable from pure noise, and a neural network, typically a U-Net or, increasingly, a [transformer](https://www.wikiprompt.org/wiki/transformer)-based architecture, trained to predict and remove the noise added at each step, effectively learning the reverse process. Generation runs this reverse process from scratch: starting with random noise, the model repeatedly predicts and subtracts noise over a number of steps, gradually revealing a coherent image or video. Conditioning mechanisms, most commonly by encoding a text [prompt](https://www.wikiprompt.org/wiki/prompt) and feeding it into the denoising network via cross-attention, let users guide what the model generates, and techniques like classifier-free guidance strengthen how closely the output follows that conditioning signal.

## Applications and ecosystem

Diffusion models power essentially every major image generator in wide use, including [stable-diffusion](https://www.wikiprompt.org/wiki/stable-diffusion), [dall-e](https://www.wikiprompt.org/wiki/dall-e), Midjourney, and Flux, and underlie leading video generators such as Sora, Google's Veo, and Chinese systems including Kling and Seedance. Extensions such as [controlnet](https://www.wikiprompt.org/wiki/controlnet), introduced in 2023, add structural conditioning, letting users guide generation with sketches, poses, or depth maps, while ecosystems built around open checkpoints support extensive customization through techniques like [lora](https://www.wikiprompt.org/wiki/lora) adapters trained by the community. Diffusion techniques have also been applied beyond media generation, including to protein structure prediction and molecule design, and researchers have explored diffusion-based approaches to text generation as an alternative to the token-by-token autoregressive approach used by most [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)s, though autoregressive generation remains dominant for text as of 2025.

## Limitations

Compared to GANs, diffusion models are computationally expensive at inference time because generating a single sample can require dozens or hundreds of sequential denoising steps, though techniques such as distillation into few-step models and improved samplers have substantially reduced this cost since 2022. Diffusion models trained on large, uncurated web-scraped datasets have also faced [ai-copyright](https://www.wikiprompt.org/wiki/ai-copyright) disputes over whether generating images in the style of, or resembling, artists in the [training-data](https://www.wikiprompt.org/wiki/training-data) constitutes infringement, and have drawn scrutiny over their capacity to reproduce memorized training images verbatim under certain conditions.

---
Source: https://www.wikiprompt.org/wiki/diffusion-model
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-02T20:29:57.623435+00:00