Stable Diffusion is a deep learning, text-to-image model released in 2022 by Stability AI, based on diffusion techniques. It is primarily used to generate detailed images conditioned on text descriptions, though it can also be applied to tasks such as inpainting, outpainting, and image-to-image translations guided by a text prompt. The model is considered a part of the ongoing generative AI boom and marked a departure from previous proprietary text-to-image models like DALL-E and Midjourney, which were accessible only via cloud services.
Stable Diffusion is a latent diffusion model, a kind of deep generative artificial neural network. Its code and model weights were released publicly, and an optimized version can run on most consumer hardware equipped with a modest GPU with as little as 2.4 GB VRAM. This accessibility distinguished it from earlier models and contributed to its widespread adoption.
Development
Stable Diffusion originated from a project called latent diffusion, developed in Germany by researchers at LMU Munich and Heidelberg University. Four of the original five authors - Robin Rombach, Andreas Blattmann, Patrick Esser, and Dominik Lorenz - later joined Stability AI and released subsequent versions of the model. The technical license was released by the CompVis group at LMU Munich, with development led by Patrick Esser of Runway and Robin Rombach of CompVis, who had earlier invented the latent diffusion architecture. Stability AI also credited EleutherAI and LAION, a German nonprofit that assembled the training dataset, as supporters.
Technology
Architecture
Diffusion models, introduced in 2015, are trained to remove successive applications of Gaussian noise on training images, akin to a sequence of denoising autoencoders. The name derives from thermodynamic diffusion, as the initial development was inspired by thermodynamics. Models in the Stable Diffusion series before SD 3 used a variant called latent diffusion model (LDM), developed in 2021 by the CompVis group at LMU Munich.
Stable Diffusion consists of three parts: a variational autoencoder (VAE), a U-Net, and an optional text encoder. The VAE encoder compresses images from pixel space to a smaller latent space, capturing fundamental semantic meaning. Gaussian noise is iteratively applied to the compressed latent representation during forward diffusion. The U-Net block, composed of a ResNet backbone, denoises the output to obtain a latent representation, and the VAE decoder generates the final image by converting the representation back to pixel space.
The denoising step can be conditioned on text, an image, or another modality via a cross-attention mechanism. For text conditioning, a fixed, pretrained CLIP ViT-L/14 text encoder transforms prompts into an embedding space. Researchers point to increased computational efficiency for training and generation as an advantage of LDMs. With 860 million parameters in the U-Net and 123 million in the text encoder, Stable Diffusion is relatively lightweight by 2022 standards and can run on consumer GPUs, or even CPU-only with the OpenVINO version.
#### SD XL
The XL version uses the same LDM architecture but larger: a larger UNet backbone, larger cross-attention context, two text encoders instead of one, and training on multiple aspect ratios. The SD XL Refiner, released simultaneously, has the same architecture but was trained for adding fine details to preexisting images via text-conditional img2img.
#### SD 3.0
The 3.0 version changes the backbone entirely, using a Rectified Flow Transformer rather than a UNet. The architecture has three tracks for original text encoding, transformed text encoding, and image encoding in latent space, with the latter two mixed during each transformer block. It is named "multimodal diffusion transformer" (MMDiT), reflecting that it mixes text and image encodings internally, unlike previous DiT versions where text affects image but not vice versa.
Training Data
Stable Diffusion was trained on image-caption pairs from LAION-5B, a publicly available dataset derived from Common Crawl web data. LAION-5B contains 5 billion image-text pairs classified by language and filtered by resolution, watermark likelihood, and predicted aesthetic score. The dataset was created by LAION, a German nonprofit funded by Stability AI. The model was trained on three subsets: laion2B-en, laion-high-resolution, and laion-aesthetics v2 5+. A third-party analysis of a smaller subset of 12 million images found that approximately 47% came from 100 domains, with Pinterest taking 8.5%, followed by WordPress, Blogspot, Flickr, DeviantArt, and Wikimedia Commons. An investigation by Bayerischer Rundfunk showed that LAION's datasets, hosted on Hugging Face, contain large amounts of private and sensitive data.
Training Procedures
The model was initially trained on laion2B-en and laion-high-resolution, with final rounds on LAION-Aesthetics v2 5+, a subset of 600 million captioned images predicted to receive a score of at least 5 out of 10 from human raters. This subset excluded low-resolution images and those with a predicted watermark probability greater than 80%. Final training rounds dropped 10% of text conditioning to improve Classifier-Free Diffusion Guidance. The model was trained using 256 Nvidia A100 GPUs on Amazon Web Services for 150,000 GPU-hours, at a cost of $600,000.
Limitations
Stable Diffusion has known issues with degradation, such as difficulty rendering hands and text, and can produce biased or harmful content due to training data. The public release raised concerns about misuse, including deepfakes and copyright infringement, leading to ongoing debates about regulation and ethical use in the machine learning community.
Impact
Stable Diffusion's release in 2022 significantly accelerated the deep learning art movement, enabling hobbyists and artists to generate high-quality images on personal hardware. It influenced subsequent text-to-image models and contributed to the broader AI boom, with applications in art, design, and content creation. The model's open-source nature fostered a large ecosystem of community tools and fine-tuned variants, though it also sparked discussions about intellectual property and the future of creative work.
Reception and Legacy
The model received widespread attention for its technical achievements and accessibility. It was praised for its efficiency and quality, but also criticized for potential misuse. As of 2025, Stable Diffusion remains a foundational reference in generative AI, with ongoing development by Stability AI and the open-source community. Its architecture and training approach have been adapted in numerous subsequent models, cementing its legacy in the field.