Stable Diffusion is a deep learning, text-to-image model released in August 2022, developed by researchers from the CompVis Group at LMU Munich and Runway, with computational support from Stability AI and training data from non-profit organizations. It is a latent diffusion model, a type of deep generative artificial neural network, and is considered a premier product of Stability AI, contributing to the ongoing AI boom. The model's code and weights were released publicly, allowing it to run on consumer hardware with modest GPUs (as little as 2.4 GB VRAM), a departure from proprietary models like DALL-E and Midjourney that were only accessible via cloud services.
The primary use case is generating detailed images conditioned on text descriptions, but it also supports tasks such as inpainting, outpainting, and image-to-image translation guided by text prompts. Its open-source nature and efficiency led to rapid adoption across various applications, from digital art to research.
Development
Stable Diffusion originated from a project called latent diffusion, developed in Germany by researchers at LMU Munich and Heidelberg University. Four of the original five authors (Robin Rombach, Andreas Blattmann, Patrick Esser, and Dominik Lorenz) later joined Stability AI and released subsequent versions. The technical license was released by the CompVis group at LMU Munich. Development was led by Patrick Esser of Runway and Robin Rombach of CompVis, who were among the inventors of the latent diffusion architecture. Stability AI also credited EleutherAI and LAION, a German non-profit that assembled the training dataset, as supporters.
Technology
Architecture
Diffusion models, introduced in 2015, are trained to remove successive applications of Gaussian noise on training images, functioning as a sequence of denoising autoencoders. The name derives from thermodynamic diffusion, as they were inspired by thermodynamics. Stable Diffusion uses a variant called latent diffusion model (LDM), developed in 2021 by the CompVis group.
The model consists of three parts: a variational autoencoder (VAE), a U-Net, and an optional text encoder. The VAE encoder compresses images from pixel space to a smaller latent space, capturing semantic meaning. Gaussian noise is iteratively applied to the compressed latent representation during forward diffusion. The U-Net, composed of a ResNet backbone, denoises the output to obtain a latent representation. Finally, the VAE decoder generates the final image by converting the representation back to pixel space.
The denoising step can be conditioned on text, images, or other modalities via a cross-attention mechanism. For text conditioning, a fixed, pretrained CLIP ViT-L/14 text encoder transforms prompts into an embedding space. LDMs offer increased computational efficiency for training and generation. With 860 million parameters in the U-Net and 123 million in the text encoder, Stable Diffusion is relatively lightweight by 2022 standards, capable of running on consumer GPUs and even CPU-only with the OpenVINO version.
SD XL
The XL version uses the same LDM architecture but larger: a larger UNet backbone, larger cross-attention context, two text encoders instead of one, and training on multiple aspect ratios. The SD XL Refiner, released simultaneously, has the same architecture but was trained for adding fine details to existing images via text-conditional img2img.
SD 3.0
The 3.0 version changes the backbone entirely, using a Rectified Flow Transformer instead of a UNet. The Transformer architecture has three tracks: original text encoding, transformed text encoding, and image encoding (in latent space). The transformed text encoding and image encoding are mixed during each transformer block. This architecture is named "multimodal diffusion transformer (MMDiT)", where "multimodal" means it mixes text and image encodings inside its operations, differing from previous DiT versions where text encoding only affected image encoding.
Training Data
Stable Diffusion was trained on image-caption pairs from LAION-5B, a publicly available dataset derived from Common Crawl data, containing 5 billion image-text pairs filtered by language, resolution, watermark likelihood, and predicted aesthetic score. The dataset was created by LAION, a German non-profit funded by Stability AI. The model was trained on three subsets: laion2B-en, laion-high-resolution, and laion-aesthetics v2 5+. A third-party analysis of a smaller subset of 12 million images found that approximately 47% came from 100 different domains, with Pinterest taking 8.5%, followed by WordPress, Blogspot, Flickr, DeviantArt, and Wikimedia Commons. An investigation by Bayerischer Rundfunk revealed that LAION's datasets, hosted on Hugging Face, contain large amounts of private and sensitive data.
Training Procedures
The model was initially trained on laion2B-en and laion-high-resolution, with final rounds on LAION-Aesthetics v2 5+, a subset of 600 million captioned images predicted to receive a score of at least 5 out of 10 for aesthetic quality. This subset excluded low-resolution images and those with a watermark probability greater than 80%. Final training rounds dropped 10% of text conditioning to improve Classifier-Free Diffusion Guidance. The model was trained using 256 Nvidia A100 GPUs on Amazon Web Services for a total of 150,000 GPU-hours, at a cost of $600,000.
Limitations
Stable Diffusion has known limitations, including difficulty with complex scenes, anatomical inaccuracies (e.g., hands), and biases in training data. It can also generate images that reflect societal biases present in the dataset. The model may struggle with text rendering and fine details, and its performance varies across different prompts. As of 2024, ongoing research addresses these issues through fine-tuning and architectural improvements.
Impact and Adoption
The public release of Stable Diffusion in August 2022 marked a significant shift in generative AI, making high-quality text-to-image generation accessible to a wide audience. Its open-source license enabled developers to integrate it into various applications, from art tools to educational platforms. The model's efficiency allowed it to run on consumer hardware, fostering a community of hobbyists and researchers. This release also sparked discussions about copyright, ethics, and the potential misuse of AI-generated content. Stability AI continued to release improved versions, such as SD XL and SD 3.0, maintaining its position in the rapidly evolving field of artificial intelligence.
Future Directions
Subsequent versions of Stable Diffusion have explored different architectures, such as the Rectified Flow Transformer in SD 3.0, aiming to improve quality and efficiency. Research continues on reducing biases, enhancing controllability, and integrating with other modalities. The model's success has influenced other machine learning projects and contributed to the broader AI boom, with implications for deep learning and neural networks. As of 2025, Stable Diffusion remains a benchmark in text-to-image generation, with ongoing community and corporate development.