Stable Diffusion 2.0 is a deep learning, text-to-image model released in 2022 by Stability AI, based on diffusion techniques. It is part of the ongoing AI boom (2022-present) in generative AI and is primarily used to generate detailed images conditioned on text descriptions, though it can also handle tasks such as Inpainting, outpainting, and image-to-image translations guided by a text prompt. The model is a latent diffusion model, a kind of deep generative artificial neural network, and its code and model weights were released publicly, allowing it to run on most consumer hardware with a modest GPU of as little as 2.4 GB VRAM. This marked a departure from previous proprietary text-to-image models such as DALL-E and Midjourney, which were accessible only via cloud services.
The development of Stable Diffusion involved researchers from the CompVis Group at LMU Munich and Runway, with a computational donation from Stability AI and training data from non-profit organizations. The model represents a significant step in making advanced generative AI accessible to a wider audience, as it can be run locally rather than requiring cloud-based access.
Development History
Stable Diffusion originated from a project called latent diffusion, developed in Germany by researchers at LMU Munich and Heidelberg University. Four of the original five authors - Robin Rombach, Andreas Blattmann, Patrick Esser, and Dominik Lorenz - later joined Stability AI and released subsequent versions of the model. The technical license for the model was released by the CompVis group at LMU Munich. Development was led by Patrick Esser of Runway and Robin Rombach of CompVis, who were among the researchers who had earlier invented the latent diffusion model architecture used by Stable Diffusion. Stability AI also credited EleutherAI and LAION, a German nonprofit that assembled the dataset on which the model was trained, as supporters of the project.
The release of Stable Diffusion 2.0 followed the initial public release of Stable Diffusion in August 2022, which had already garnered significant attention for its ability to generate high-quality images from text prompts. The 2.0 version introduced several improvements and refinements to the architecture and training process.
Technology
Architecture
Diffusion models, introduced in 2015, are trained with the objective of removing successive applications of Gaussian noise on training images, which can be thought of as a sequence of denoising autoencoders. The name "diffusion" comes from thermodynamic diffusion, as the models were first developed with inspiration from thermodynamics. Models in the Stable Diffusion series before SD 3 all used a variant of diffusion models called latent diffusion model (LDM), developed in 2021 by the CompVis group at LMU Munich.
Stable Diffusion consists of three parts: the variational autoencoder (VAE), a U-Net, and an optional text encoder. The VAE encoder compresses the image from pixel space to a smaller dimensional latent space, capturing a more fundamental semantic meaning of the image. Gaussian noise is iteratively applied to the compressed latent representation during forward diffusion. The U-Net block, composed of a ResNet backbone, denoises the output from forward diffusion backwards to obtain a latent representation. Finally, the VAE decoder generates the final image by converting the representation back into pixel space.
The denoising step can be flexibly conditioned on a string of text, an image, or another modality. The encoded conditioning data is exposed to denoising U-Nets via a cross-attention mechanism. For conditioning on text, the fixed, pretrained CLIP ViT-L/14 text encoder is used to transform text prompts to an embedding space. Researchers point to increased computational efficiency for training and generation as an advantage of LDMs.
With 860 million parameters in the U-Net and 123 million in the text encoder, Stable Diffusion is considered relatively lightweight by 2022 standards. Unlike other diffusion models, it can run on consumer GPUs, and even CPU-only if using the OpenVINO version of Stable Diffusion.
Training Data
Stable Diffusion was trained on pairs of images and captions taken from LAION-5B, a publicly available dataset derived from Common Crawl data scraped from the web. The dataset contains 5 billion image-text pairs that were classified based on language and filtered into separate datasets by resolution, a predicted likelihood of containing a watermark, and predicted "aesthetic" score (e.g., subjective visual quality). The dataset was created by LAION, a German non-profit that receives funding from Stability AI.
The Stable Diffusion model was trained on three subsets of LAION-5B: laion2B-en, laion-high-resolution, and laion-aesthetics v2 5+. A third-party analysis of the model's training data identified that out of a smaller subset of 12 million images taken from the original wider dataset used, approximately 47% of the sample size of images came from 100 different domains, with Pinterest taking up 8.5% of the subset, followed by websites such as WordPress, Blogspot, Flickr, DeviantArt, and Wikimedia Commons. An investigation by Bayerischer Rundfunk showed that LAION's datasets, hosted on Hugging Face, contain large amounts of private and sensitive data.
Training Procedures
The model was initially trained on the laion2B-en and laion-high-resolution subsets, with the last few rounds of training done on LAION-Aesthetics v2 5+, a subset of 600 million captioned images which the LAION-Aesthetics Predictor V2 predicted that humans would, on average, give a score of at least 5 out of 10 when asked to rate how much they liked them. The LAION-Aesthetics v2 5+ subset also excluded low-resolution images and images which LAION-5B-WatermarkDetection identified as carrying a watermark with greater than 80% probability. Final rounds of training additionally dropped 10% of text conditioning to improve Classifier-Free Diffusion Guidance.
The model was trained using 256 Nvidia A100 GPUs on Amazon Web Services for a total of 150,000 GPU-hours, at a cost of $600,000.
Improvements in Version 2.0
Stable Diffusion 2.0 introduced several key improvements over the original release. The model incorporated a new text encoder, using a larger CLIP model that improved the alignment between text prompts and generated images. The training data was also refined, with a greater emphasis on high-resolution and aesthetically pleasing images, which helped reduce the occurrence of artifacts and improve overall image quality.
Additionally, the 2.0 version included better support for non-square aspect ratios, allowing users to generate images in a wider variety of formats. The model also showed improved performance on tasks such as inpainting and outpainting, where parts of an image are filled in or extended based on surrounding context.
Limitations
Stable Diffusion has issues with degradation and artifacts in certain scenarios. The model can struggle with generating accurate representations of complex scenes, particularly those involving multiple objects or intricate details. It may also produce biased or inappropriate content, reflecting biases present in the training data. The model's reliance on the CLIP text encoder means that its understanding of language is limited to what CLIP has learned, which can lead to misinterpretations of certain prompts.
Furthermore, the training data, sourced from the web, contains a wide range of content, some of which may be private or sensitive. This has raised concerns about privacy and the ethical implications of using such data for training generative models. The computational cost of training, while lower than some alternatives, still requires significant resources, and the model's performance can vary depending on the hardware used for inference.
Impact and Reception
The release of Stable Diffusion 2.0 was part of the broader Stable Diffusion public release that contributed to the generative AI boom. The model's open-source nature and ability to run on consumer hardware made it widely accessible, spurring a wave of creative and commercial applications. It also influenced subsequent developments in the field, including the release of other open-source models and the integration of diffusion-based techniques into various products.
The model's accessibility also raised discussions about copyright, artistic ownership, and the potential for misuse, leading to ongoing debates about the regulation of generative AI technologies. Despite these challenges, Stable Diffusion 2.0 remains a significant milestone in the evolution of text-to-image generation, demonstrating the feasibility of high-quality generative models that are not confined to cloud services.