# PaLM

PaLM (Pathways Language Model) is a 540 billion-parameter transformer-based large language model developed by Google AI, announced in April 2022. It excels in reasoning, code generation, and translation, and was succeeded by PaLM 2 in 2023.

PaLM (Pathways Language Model) is a large language model developed by Google AI, first announced in April 2022. It is a dense decoder-only transformer-based model with 540 billion parameters, making it one of the largest language models at the time of its release. The model was trained using Google's Pathways system, which enables efficient training across multiple TPU pods. Researchers also trained smaller versions with 8 and 62 billion parameters to study the effects of model scale on performance.

PaLM demonstrated strong capabilities across a wide range of tasks, including commonsense reasoning, arithmetic reasoning, joke explanation, code generation, and translation. When combined with chain-of-thought prompting, PaLM achieved significantly better performance on datasets requiring multi-step reasoning, such as word problems and logic-based questions. The model remained private until March 2023, when Google launched an API for PaLM, initially available to a limited number of developers through a waitlist before public release.

## Architecture and Training

PaLM is based on the [transformer](https://www.wikiprompt.org/wiki/transformer) architecture, specifically a decoder-only design. The 540B version was pre-trained on a corpus of 780 billion tokens comprising filtered webpages, books, Wikipedia articles, news articles, source code from GitHub repositories, and social media conversations. The dataset was derived from the one used to train Google's LaMDA model, with social media conversations making up 50% of the corpus to enhance conversational capabilities.

Training the 540B model required two TPU v4 Pods, each with 3,072 TPU v4 chips attached to 768 hosts, connected using a combination of model and data parallelism. This configuration, totaling 6,144 chips, was the largest TPU setup at the time and achieved a hardware FLOPs utilization of 57.8%, a record for training efficiency at that scale. The training process leveraged the Pathways system, which allowed for efficient scaling across multiple pods.

## Variants and Applications

Google and [DeepMind](https://www.wikiprompt.org/wiki/google-deepmind) developed Med-PaLM, a version of PaLM 540B fine-tuned on medical data. Med-PaLM outperformed previous models on medical question-answering benchmarks and was the first to obtain a passing score on U.S. medical licensing questions. It provides reasoning for its answers and can evaluate its own responses.

Google also extended PaLM with a vision transformer to create PaLM-E, a vision-language model for robotic manipulation that does not require retraining or fine-tuning for new tasks. In June 2023, Google announced AudioPaLM for speech-to-speech translation, which uses the PaLM-2 architecture and initialization.

## Successor: PaLM 2

In May 2023, Google announced PaLM 2 at the annual Google I/O keynote. PaLM 2 is reported to be a 340 billion-parameter model trained on 3.6 trillion tokens. It serves as the successor to PaLM and powers various Google products and services. PaLM 2 improved upon its predecessor in areas such as multilingual capabilities, reasoning, and coding, and was made available through the PaLM API and other Google Cloud services.

## Impact and Legacy

PaLM represented a significant step in the development of [large language models](https://www.wikiprompt.org/wiki/large-language-model), demonstrating the feasibility of training models at unprecedented scale using distributed systems. Its success influenced subsequent models, including Gemini, which succeeded PaLM as Google's flagship language model. PaLM's contributions to chain-of-thought prompting and multimodal extensions like PaLM-E also advanced research in reasoning and embodied AI.

## See Also

- [LaMDA](https://www.wikiprompt.org/wiki/lamda) - PaLM's predecessor
- [Gemini](https://www.wikiprompt.org/wiki/gemini) - PaLM's successor
- [Chinchilla](https://www.wikiprompt.org/wiki/chinchilla) - a related language model

---
Source: https://www.wikiprompt.org/wiki/palm
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-07T21:35:40.737111+00:00