# Latent Diffusion Latent diffusion models (LDMs) are a diffusion model architecture that performs denoising in a compressed latent space, enabling efficient high-resolution image generation. Developed by the CompVis group at LMU Munich, LDMs underpin Stable Diffusion versions 1.1 to 2.1. The latent diffusion model (LDM) is a diffusion model architecture developed by the Computer Vision & Learning (CompVis) group at LMU Munich. It improves upon standard diffusion models by performing the diffusion process in a lower-dimensional latent space rather than directly in pixel space, which reduces computational cost and allows for more efficient generation of high-resolution images. LDMs also incorporate self-attention and cross-attention conditioning, enabling flexible control over the generated output based on text, images, or other modalities. Diffusion models were introduced in 2015 as a class of generative models that learn to reverse a gradual noising process. The LDM architecture was published on arXiv on December 20, 2021, and both the Stable Diffusion and LDM repositories were released on GitHub. LDMs are widely used in practical diffusion models; for instance, Stable Diffusion versions 1.1 to 2.1 were based on the LDM architecture. ## Architecture The LDM consists of three main components: a variational autoencoder (VAE), a modified U-Net, and a text encoder. The VAE encoder compresses an input image from pixel space into a smaller latent space, capturing semantic meaning while reducing dimensionality. Gaussian noise is iteratively applied to this compressed latent representation during forward diffusion. The U-Net, built on a ResNet backbone, denoises the latent representation through a reverse process. Finally, the VAE decoder converts the denoised latent back into pixel space to produce the final image. The denoising step can be conditioned on text, images, or other modalities. For text conditioning, a pretrained CLIP ViT-L/14 text encoder transforms text prompts into an embedding space, which is then exposed to the U-Net via a cross-attention mechanism. This allows the model to generate images that align with the provided textual description. ### Variational Autoencoder The VAE is first trained on a dataset of images. Its encoder takes an image as input and outputs a lower-dimensional latent representation, which serves as input to the U-Net. After training, the encoder is used to compress images, and the decoder reconstructs images from latent representations. In the implemented version, the encoder is a convolutional neural network (CNN) with a single self-attention mechanism near the end. It takes a tensor of shape (3, H, W) and outputs a tensor of shape (8, H/8, W/8), which concatenates the predicted mean and variance of the latent vector, each of shape (4, H/8, W/8). During training, the variance is used, but at inference typically only the mean is retained. The decoder is also a CNN with a single self-attention mechanism, mapping a (4, H/8, W/8) tensor back to (3, H, W). ### U-Net The U-Net backbone processes several inputs: a latent image array from the VAE encoder, a timestep embedding indicating the current noise level, and conditioning embeddings from the text encoder. The U-Net is trained to predict the noise that was added to the latent representation, allowing it to iteratively denoise. The architecture incorporates residual blocks and attention layers, enabling it to handle both local and global context. ## Training and Conditioning LDMs are trained with a denoising objective: given a noisy latent, the model learns to predict the original noise. The training process uses a loss function that measures the difference between predicted and actual noise. Conditioning is integrated through cross-attention layers, where the text embedding is used to modulate the denoising process. This approach supports classifier-free guidance, where the model is trained both with and without conditioning, and at inference the conditioning is amplified to improve adherence to the prompt. ## Applications and Impact LDMs have become a foundational architecture for text-to-image generation. Stable Diffusion, a prominent open-source model, uses the LDM architecture. Versions 1.1 to 1.4 were released by CompVis in August 2022, with each version finetuned on progressively more aesthetic images. Stable Diffusion 1.5, released by RunwayML in October 2022, incorporated a 10% dropout of text conditioning to improve classifier-free guidance. The efficiency of latent space diffusion has made it possible to run these models on consumer hardware, democratizing access to generative AI. ## Related Developments Diffusion models evolved from earlier work on non-equilibrium thermodynamics. A 2019 paper introduced the noise conditional score network (NCSN), also known as score matching with Langevin dynamics (SMLD), with a PyTorch implementation. A 2020 paper proposed the denoising diffusion probabilistic model (DDPM), which improved training via variational inference and was released in TensorFlow. The LDM builds on these foundations by shifting the diffusion process to a latent space, a key innovation that reduces memory and compute requirements while maintaining high output quality. LDMs are part of the broader field of [generative-ai](https://www.wikiprompt.org/wiki/generative-ai), which also includes [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)s and other [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) architectures. The use of [cross-attention](https://www.wikiprompt.org/wiki/cross-attention) and [u-net](https://www.wikiprompt.org/wiki/u-net) backbones connects LDMs to advances in [neural-network](https://www.wikiprompt.org/wiki/neural-network) design. As of 2025, latent diffusion remains a standard approach in commercial and open-source image generation systems, with ongoing research focusing on improving efficiency, controllability, and fidelity. --- Source: https://www.wikiprompt.org/wiki/latent-diffusion License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-12T16:28:48.649402+00:00