# Latent Diffusion Model A latent diffusion model (LDM) is a diffusion model architecture that performs the diffusion process in a compressed latent space, enabling efficient text-to-image generation. It is the core architecture behind Stable Diffusion. A latent diffusion model (LDM) is a type of [generative model](https://www.wikiprompt.org/wiki/generative-ai) that extends the diffusion model framework by operating in a lower-dimensional latent space rather than directly in pixel space. Developed by the Computer Vision & Learning (CompVis) group at LMU Munich, the LDM was introduced in a paper published on arXiv on December 20, 2021. It improves upon standard diffusion models by reducing computational cost while maintaining high-quality image synthesis, and it supports conditioning through self-attention and cross-attention mechanisms. LDMs are widely used in practical diffusion models; for instance, Stable Diffusion versions 1.1 to 2.1 were based on the LDM architecture. The LDM architecture consists of three main components: a variational autoencoder (VAE), a modified U-Net, and a text encoder. The VAE compresses images into a latent space, the U-Net performs denoising in that latent space, and the text encoder provides conditioning information. This design allows the model to generate images from text prompts efficiently, making it a cornerstone of modern [AI](https://www.wikiprompt.org/wiki/artificial-intelligence) image generation. ## Background and Development Diffusion models were first introduced in 2015 as a method for learning to sample from complex probability distributions, using principles from non-equilibrium thermodynamics. A 2019 paper proposed the noise conditional score network (NCSN), also known as score-matching with Langevin dynamics (SMLD), which was implemented in [PyTorch](https://www.wikiprompt.org/wiki/deep-learning). In 2020, the denoising diffusion probabilistic model (DDPM) improved upon these methods using variational inference, with a reference implementation in TensorFlow. The LDM paper was published on arXiv on December 20, 2021, and both the Stable Diffusion and LDM repositories were released on GitHub. The architecture gained prominence when Stable Diffusion versions 1.1 through 2.1 adopted it. Stable Diffusion 1.1 was trained on the LAION-2B-en dataset, and subsequent versions were fine-tuned for improved aesthetics and classifier-free guidance. Stable Diffusion 1.5 was released by RunwayML in October 2022. ## Architecture Overview The LDM operates in a compressed latent space, which reduces the dimensionality of the data and speeds up training and inference. The process begins with a VAE encoder that compresses an input image from pixel space into a latent representation. Gaussian noise is then iteratively applied to this latent representation during forward diffusion. A U-Net, composed of a ResNet backbone, denoises the latent representation in reverse, and finally a VAE decoder converts the denoised latent back into pixel space. The denoising step can be conditioned on various modalities, such as text, images, or other data. For text conditioning, a pretrained CLIP ViT-L/14 text encoder transforms text prompts into an embedding space, which is exposed to the U-Net via a cross-attention mechanism. This allows the model to generate images that align with the provided textual description. ### Variational Autoencoder The VAE is trained on a dataset of images to learn a compressed latent representation. The encoder takes an RGB image of shape (3, 512, 512) and outputs a latent tensor of shape (4, 64, 64), after applying a scaling factor of 0.18215 to roughly whiten the latent vector. The decoder reverses this process, taking a latent tensor and producing an image, with a scaling factor of 0.18125 and clipping to the range [0, 1]. The encoder is a convolutional neural network (CNN) with a single self-attention mechanism near the end. It outputs a concatenation of predicted mean and variance for the latent vector, each of shape (4, H/8, W/8). During training, the variance is used, but at inference, only the mean is typically retained. The decoder is also a CNN with a similar self-attention structure, converting latent tensors back to images. ### U-Net The U-Net backbone is the core denoising component. It takes as input a latent image array produced by the VAE encoder, along with conditioning information such as text embeddings. The U-Net is designed to predict the noise that was added during forward diffusion, allowing the model to iteratively remove noise and recover the original latent representation. The architecture includes residual connections and attention mechanisms, enabling it to handle complex dependencies in the data. ## Applications and Impact Latent diffusion models have become the foundation for many text-to-image systems, most notably Stable Diffusion. The efficiency gained by operating in latent space allows these models to run on consumer hardware, democratizing access to AI image generation. The architecture has also been adapted for other tasks, such as inpainting, super-resolution, and video generation, demonstrating its versatility. ## Limitations and Future Directions Despite their success, LDMs face challenges such as computational requirements for training, potential biases in generated content, and the need for careful tuning of conditioning mechanisms. Research continues to improve the architecture, including more efficient attention mechanisms and better latent space representations. As of the early 2020s, LDMs remain a key area of study in [machine learning](https://www.wikiprompt.org/wiki/machine-learning) and [computer vision](https://www.wikiprompt.org/wiki/computer-vision), with ongoing contributions from academic and industrial labs. --- Source: https://www.wikiprompt.org/wiki/latent-diffusion-model License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-09T02:00:36.864957+00:00