Stable Diffusion is a deep learning, text-to-image model released in August 2022, based on diffusion techniques. It is the premier product of Stability AI and is considered part of the ongoing AI boom. The model generates detailed images conditioned on text descriptions, and it also supports tasks such as inpainting, outpainting, and image-to-image translation guided by text prompts. Its development involved researchers from the CompVis Group at LMU Munich and Runway, with a computational donation from Stability AI and training data from non-profit organizations.
Stable Diffusion is a latent diffusion model, a type of deep generative artificial neural network. Its code and model weights were released publicly, and an optimized version runs on most consumer hardware with a modest GPU using as little as 2.4 GB VRAM. This departure from proprietary text-to-image models like DALL-E and Midjourney, which were accessible only via cloud services, made Stable Diffusion widely available to individual users and researchers.
Development
Stable Diffusion originated from a project called latent diffusion, developed in Germany by researchers at LMU Munich and Heidelberg University. Four of the original five authors - Robin Rombach, Andreas Blattmann, Patrick Esser, and Dominik Lorenz - later joined Stability AI and released subsequent versions. The technical license was released by the CompVis group at LMU Munich. Development was led by Patrick Esser of Runway and Robin Rombach of CompVis, who had earlier invented the latent diffusion architecture. Stability AI also credited EleutherAI and LAION, a German nonprofit that assembled the training dataset, as project supporters.
Technology
Architecture
Diffusion models, introduced in 2015, are trained to remove successive applications of Gaussian noise on training images, functioning as a sequence of denoising autoencoders. The name comes from thermodynamic diffusion, as the technique was inspired by thermodynamics. Models in the Stable Diffusion series before SD 3 used a variant called latent diffusion model (LDM), developed in 2021 by the CompVis group at LMU Munich.
Stable Diffusion consists of three parts: a variational autoencoder (VAE), a U-Net, and an optional text encoder. The VAE encoder compresses images from pixel space to a smaller dimensional latent space, capturing semantic meaning. Gaussian noise is iteratively applied to the compressed latent representation during forward diffusion. The U-Net block, composed of a ResNet backbone, denoises the output backward to obtain a latent representation. Finally, the VAE decoder generates the final image by converting the representation back to pixel space.
The denoising step can be conditioned on text, an image, or another modality. Encoded conditioning data is exposed to denoising U-Nets via a Cross-Attention mechanism. For text conditioning, a fixed, pretrained CLIP ViT-L/14 text encoder transforms prompts into an embedding space. Researchers cite increased computational efficiency for training and generation as an advantage of LDMs.
With 860 million parameters in the U-Net and 123 million in the text encoder, Stable Diffusion is relatively lightweight by 2022 standards. Unlike other diffusion models, it runs on consumer GPUs, and even CPU-only with the OpenVINO version.
#### SD XL
The XL version uses the same LDM architecture but larger: a larger UNet backbone, larger cross-attention context, two text encoders instead of one, and training on multiple aspect ratios rather than square only. The SD XL Refiner, released simultaneously, has the same architecture but was trained for adding fine details to preexisting images via text-conditional img2img.
#### SD 3.0
The 3.0 version completely changes the backbone. It uses a Rectified Flow Transformer, implementing the rectified flow method with a Transformer (architecture) architecture. The architecture has three tracks: original text encoding, transformed text encoding, and image encoding in latent space. The transformed text encoding and image encoding are mixed during each transformer block. This is named "multimodal diffusion transformer (MMDiT)", where multimodal means it mixes text and image encodings inside its operations, unlike previous DiT versions where text encoding affected image encoding but not vice versa.
Training Data
Stable Diffusion was trained on image-caption pairs from LAION-5B, a publicly available dataset derived from Common Crawl web scrapes. The 5 billion image-text pairs were classified by language and filtered into datasets by resolution, predicted watermark likelihood, and predicted "aesthetic" score. LAION, a German nonprofit receiving funding from Stability AI, created the dataset. The model was trained on three subsets: laion2B-en, laion-high-resolution, and laion-aesthetics v2 5+.
A third-party analysis of a smaller subset of 12 million images found approximately 47% came from 100 domains, with Pinterest taking 8.5%, followed by WordPress, Blogspot, Flickr, DeviantArt, and Wikimedia Commons. An investigation by Bayerischer Rundfunk showed LAION's datasets, hosted on Hugging Face, contain large amounts of private and sensitive data.
Training Procedures
The model was initially trained on laion2B-en and laion-high-resolution, with final rounds on LAION-Aesthetics v2 5+, a subset of 600 million captioned images predicted to receive at least 5 out of 10 in human aesthetic ratings. This subset excluded low-resolution images and those with a watermark detection probability above 80%. Final training rounds dropped 10% of text conditioning to improve Classifier-Free Diffusion Guidance. Training used 256 Nvidia A100 GPUs on Amazon Web Services for 150,000 GPU-hours, costing $600,000.
Limitations
Stable Diffusion has issues with degradation and artifacts, particularly in complex scenes, human anatomy, and text rendering. It can reproduce biases from training data, including stereotypes and harmful content. The model struggles with fine details like hands and faces, often producing distorted results. It also has difficulty with compositional prompts involving multiple objects or specific spatial relationships. As of 2024, newer versions address some limitations but still face challenges in these areas.
Impact and Reception
The public release of Stable Diffusion in August 2022 sparked widespread discussion about generative AI ethics, copyright, and artistic labor. Its open-source nature allowed developers to create tools for machine learning experimentation, data augmentation, and creative workflows. The model's accessibility on consumer hardware contributed to a surge in AI art communities and commercial applications. Stability AI released subsequent versions, including SD XL in 2023 and SD 3.0 in 2024, each improving quality and capabilities. The model also influenced policy debates, leading to discussions about AI regulation and content provenance.