DALL-E 1, stylised as DALL·E, is a text-to-image model developed by OpenAI using deep learning methodology. It generates digital images from natural language descriptions, known as prompts. The first version of DALL-E was announced in January 2021, making it one of the first prominent models to combine large language models with image generation. The software's name is a portmanteau of the names of the animated robot WALL-E and the Spanish surrealist artist Salvador DalÃ.
Its successor, DALL-E 2, was released in April 2022, with a public beta in July 2022. DALL-E 3, released in October 2023, was integrated natively into ChatGPT for Plus and Enterprise customers. DALL-E 3 was later replaced in ChatGPT by GPT Image's native image-generation capabilities in March 2025.
History and background
DALL-E was revealed by OpenAI in a blog post on 5 January 2021. The model used a version of gpt-partnr? but for images, a Transformer (architecture) architecture adapted for image generation. At the time, OpenAI was known for large language models like GPT-3, but DALL-E marked a shift into multimodal artificial intelligence - handling both text and image domains. The model's name followed the trend of playful acronyms and puns common in OpenAI's naming.
In February 2024, OpenAI began adding watermarking to DALL-E generated images, containing metadata in the C2PA (Coalition for Content Provenance and Authenticity) standard, promoted by the Content Authenticity Initiative, to help trace image origins.
Technology
The first generative pre-trained transformer (GPT) model was initially developed by OpenAI in 2018, using a Transformer (architecture) architecture. The first iteration, GPT-1, was scaled up to produce GPT-2 in 2019; in 2020, it was scaled up again to produce GPT-3, with 175 billion parameters. DALL-E is a distinct variant of GPT-3, modified to generate images.
DALL-E 1
DALL-E has three components: a discrete variational-encoder?, an autoregressive decoder-only Transformer (architecture) model (12 billion parameters) similar to GPT-3, and a CLIP pair of image encoder and text encoder.
The discrete variational autoencoder (VAE) converts an image to a sequence of tokens, and conversely, converts a sequence of tokens back to an image. This is necessary as the transformer model does not directly process image data. The input to the transformer is a sequence of tokenised image caption followed by tokenised image patches. The caption is in English, tokenised by byte-pair-encoding (vocabulary size 16384), and can be up to 256 tokens long. Each image is a 256x256 RGB image, divided into 32x32 patches of 4x4 each patch. Each patch is then converted to a token (vocabulary size 8192) by a discrete VAE.
DALL-E is developed and released in conjunction with CLIP (Contrastive Language-Image Pre-training). CLIP is a separate model based on contrastive learning, trained on 400 million pairs of images with text captions scraped from the internet. Its role is to understand and rank DALL-E's output by predicting which caption from a list of 32,768 randomly selected captions (of which one is correct) is most appropriate for an image. A trained CLIP pair filters a larger initial list of images generated by DALL-E to select the image closest to the text prompt.
Successors
DALL-E 2 uses 3.5 billion parameters, a smaller number than its predecessor. Instead of an autoregressive transformer, it uses a Diffusion model conditioned on CLIP image embeddings, which, during inference, are generated from CLIP text embeddings by a prior model. This is the same architecture as that of Stable-Diffusion, released a few months later.
DALL-E 3, released in September 2023, follows complex prompts with more accuracy and detail than prior iterations, and can generate more coherent text. While a technical report was written for it, the report does not include training or implementation details, focusing on improved prompt following.
Capabilities
DALL-E can generate imagery in multiple styles, including photorealistic imagery, paintings, and emoji. It can manipulate and rearrange objects in its images, and can correctly place design elements in novel compositions without explicit instruction. It demonstrates creative-reasoning abilities, such as filling in plausible details for prompts - for example, adding Christmas imagery to prompts associated with the celebration, or appropriately placed shadows even when not mentioned. DALL-E exhibits a broad understanding of visual design trends.
DALL-E can produce images for a wide variety of arbitrary descriptions from various viewpoints with only rare failures. Mark Riedl, an associate professor at the Georgia Tech School of Interactive Computing, found that DALL-E could blend concepts, a key element of human creativity. Its visual reasoning ability is sufficient to solve Raven's Matrices, visual tests often administered to measure intelligence.
Image modification
Given an existing image, DALL-E 2 and DALL-E 3 can produce variations of the image, and edit it to modify or expand the scene. These models' inpainting and outpainting abilities use context from the image to fill in missing areas with a style consistent with the original, following a given prompt. This can be used to insert a new subject or extend an image beyond its borders. According to OpenAI, outpainting takes into account the image's existing visual elements, including shadows, reflections, and textures.
Applications and impact
DALL-E 1 opened a new era of generative AI image creation, and influenced later models and applications. Its successor DALL-E 2 was made available via an API in November 2022, allowing developers to integrate the model into applications, with cost-per-image pricing. Microsoft incorporated DALL-E 2 into its Designer app and Image Creator tool in Bing and Edge. DALL-E 3 was integrated into ChatGPT for Plus and Enterprise customers in October 2023, with APIs and Labs availability in November. In February 2024, OpenAI added watermarks to DALL-E images following the C2PA standard.
References
This article is based on information from public reports and OpenAI's own publications.