Stable Diffusion XL (SDXL) is a deep learning, text-to-image model released by Stability AI in July 2023. It is an enhanced version of the original Stable Diffusion model, designed to generate higher-resolution images with greater detail and compositional accuracy. SDXL is part of the broader Generative AI boom and represents a significant advancement in diffusion models, making high-quality image synthesis accessible to a wider audience.
The model is primarily used to generate detailed images conditioned on text descriptions, but it also supports tasks such as Inpainting, outpainting, and image-to-image translation guided by a text prompt. Unlike earlier proprietary systems like DALL-E and Midjourney, which were accessible only via cloud services, SDXL's code and weights were released publicly, allowing it to run on consumer hardware with modest GPUs. This openness marked a departure from previous text-to-image models and contributed to the rapid adoption of Stable Diffusion technologies.
Development
SDXL was developed by Stability AI in collaboration with researchers from the CompVis Group at LMU Munich and Runway, with computational support from Stability AI and training data from non-profit organizations. The project built on the latent diffusion model (LDM) architecture, which was invented in 2021 by researchers at LMU Munich and Heidelberg University, including Robin Rombach, Andreas Blattmann, Patrick Esser, and Dominik Lorenz. These researchers later joined Stability AI to develop subsequent versions of Stable Diffusion.
The technical license for the model was released by the CompVis group, and development was led by Patrick Esser of Runway and Robin Rombach of CompVis. Stability AI also credited EleutherAI and LAION, a German non-profit that assembled the training dataset, as supporters. The release of SDXL followed the success of the original Stable Diffusion in 2022 and aimed to address limitations in resolution and detail.
Technology
Architecture
SDXL uses a variant of diffusion models called latent diffusion models (LDMs), which were developed in 2021. Diffusion models, introduced in 2015, are trained to remove successive applications of Gaussian noise from training images, functioning as a sequence of denoising autoencoders. The name "diffusion" comes from thermodynamic diffusion, as the models were inspired by thermodynamics.
SDXL consists of three main parts: a variational autoencoder (VAE), a U-Net, and an optional text encoder. The VAE encoder compresses an image from pixel space to a smaller dimensional latent space, capturing the semantic meaning of the image. Gaussian noise is iteratively applied to the compressed latent representation during forward diffusion. The U-Net block, composed of a ResNet backbone, denoises the output to obtain a latent representation. Finally, the VAE decoder generates the final image by converting the representation back into pixel space.
The denoising step can be conditioned on text, an image, or another modality. The encoded conditioning data is exposed to the denoising U-Net via a cross-attention mechanism. For text conditioning, SDXL uses two text encoders instead of one, which improves the model's ability to understand complex prompts. The larger cross-attention context and multi-aspect ratio training allow SDXL to generate images at various resolutions, not just the square format of previous versions.
With 860 million parameters in the U-Net and 123 million in the text encoder, SDXL is relatively lightweight compared to other models of its era, enabling it to run on consumer GPUs with as little as 2.4 GB VRAM. An optimized version can even run on CPU-only using OpenVINO.
SD XL Refiner
Alongside SDXL, Stability AI released the SD XL Refiner, which has the same architecture but was trained specifically for adding fine details to preexisting images via text-conditional image-to-image translation. This refiner model allows users to enhance generated images iteratively.
Training Data
SDXL was trained on pairs of images and captions from LAION-5B, a publicly available dataset derived from Common Crawl data scraped from the web. The dataset contains 5 billion image-text pairs, filtered by language, resolution, watermark likelihood, and aesthetic score. The training used three subsets: laion2B-en, laion-high-resolution, and laion-aesthetics v2 5+. The last subset includes 600 million images with predicted aesthetic scores of at least 5 out of 10, excluding low-resolution images and those with watermarks.
A third-party analysis of the training data revealed that approximately 47% of a smaller subset of 12 million images came from 100 different domains, with Pinterest accounting for 8.5%, followed by WordPress, Blogspot, Flickr, DeviantArt, and Wikimedia Commons. An investigation by Bayerischer Rundfunk found that LAION's datasets contained large amounts of private and sensitive data, raising privacy concerns.
Training Procedures
The model was initially trained on laion2B-en and laion-high-resolution, with final rounds on LAION-Aesthetics v2 5+. The training dropped 10% of text conditioning to improve classifier-free diffusion guidance. SDXL was trained using 256 Nvidia A100 GPUs on Amazon Web Services for a total of 150,000 GPU-hours, at a cost of $600,000.
Limitations
Like other diffusion models, SDXL has limitations, including issues with degradation in certain scenarios. It may struggle with fine details, such as hands or text, and can produce artifacts. The model's training data also introduces biases and potential privacy issues, as noted in investigations. As of its release, SDXL was considered a significant improvement over earlier versions but still required careful prompt engineering for optimal results.
Impact
SDXL's release contributed to the AI boom by democratizing high-quality text-to-image generation. Its open availability allowed developers and artists to integrate it into various applications, from art creation to design. The model's architecture influenced subsequent developments in latent diffusion and transformer-based models, including the later SD 3.0, which replaced the U-Net with a rectified flow transformer.
SDXL also sparked discussions about copyright, ethics, and the societal impact of generative AI, as it enabled widespread creation of realistic images. Its training data, sourced from the web, raised questions about consent and data usage, leading to ongoing debates in the AI community.
See Also
- Stable Diffusion Public Release
- Diffusion Models Paper
- CLIP
- Variational Autoencoder
- U-Net
- ResNet
- Attention Mechanism
- LAION
- EleutherAI
- Runway
- Robin Rombach
- Stability AI