Muse is a text-to-image generative model developed by Google, first announced in a research paper in January 2023. It belongs to the family of Transformer (architecture)-based models and is designed to generate high-resolution images from textual descriptions. Unlike diffusion-based models such as Stable Diffusion or DALL-E, Muse operates on discrete tokens in a compressed image space, using a masked generative approach inspired by masked language modeling in Large language models.
The model was developed by researchers at Google and Google DeepMind, building on prior work in vector quantization and transformer architectures. It was publicly detailed in the paper "Muse: Text-To-Image Generation via Masked Generative Transformers," which demonstrated state-of-the-art results on benchmarks like MS-COCO and zero-shot generation capabilities. Google positioned Muse as a more efficient alternative to diffusion models, claiming faster inference times while maintaining competitive image quality.
Architecture
Muse uses a two-stage pipeline. First, a pretrained variational autoencoder (VAE) compresses images into a discrete token grid, similar to the approach used in VQGAN or VQ-VAE. Second, a transformer model is trained to predict masked tokens in this grid, conditioned on text embeddings produced by a frozen Large language model (specifically T5). During generation, the model starts with all tokens masked and iteratively predicts the most confident tokens, filling in the rest over several steps. This masked modeling technique allows for parallel decoding, which speeds up generation compared to autoregressive models.
The text encoder is a frozen T5-XXL model, which provides rich semantic representations. The image tokenizer is a VAE trained on a large dataset of images, producing a 256x256 or 512x512 token grid depending on the configuration. The transformer itself is a standard encoder-decoder architecture, with the text embeddings as the encoder input and the masked image tokens as the decoder input.
Training and data
Muse was trained on a large corpus of image-text pairs, including the publicly available LAION-5B dataset and internal Google datasets. The training process involved two stages: first, the VAE was trained separately on images alone; then, the transformer was trained to predict masked tokens given text and unmasked context. The model was trained on TPU v4 pods, with the largest variant having 3 billion parameters. The paper reported that Muse achieved a zero-shot FID score of 7.9 on MS-COCO, outperforming earlier diffusion models like GLIDE and DALL-E 2 at the time.
Capabilities
The model supports several tasks beyond text-to-image generation. It can perform image editing by partially masking an input image and providing a text prompt, allowing for localized changes. It also supports outpainting (extending an image beyond its borders) and inpainting (filling in missing regions). Additionally, Muse can generate images with a variable aspect ratio and can be fine-tuned for specific styles or domains. The paper demonstrated that Muse could produce high-quality 512x512 images in about 1.5 seconds on a single TPU, which is significantly faster than diffusion models that require many denoising steps.
Comparison with other models
Muse is often compared to DALL-E 2 and Imagen, both of which use diffusion processes. Diffusion models gradually add and remove noise, requiring hundreds of steps. Muse, in contrast, uses a discrete token approach with iterative unmasking, typically requiring only 10-20 steps. This makes it more computationally efficient during inference. However, diffusion models have been more widely adopted in production systems, partly because of their simplicity and robustness. Muse's reliance on a discrete token space can sometimes lead to artifacts, especially with complex textures or fine details.
Reception and impact
Muse was well received in the research community for its innovative use of masked modeling in the image domain. It influenced subsequent work on discrete diffusion and token-based generation. However, Google did not release Muse as a public product or API, unlike its later models such as Imagen or Gemini. The code and weights were not open-sourced, limiting reproducibility. Nevertheless, the ideas from Muse have been incorporated into later Google models and have inspired open-source efforts like Parti and MuseGAN.
See also
References
- Chang, H., et al. (2023). "Muse: Text-To-Image Generation via Masked Generative Transformers." arXiv:2301.00704.
- Google AI Blog. (2023). "Muse: A New Text-to-Image Model."