PaLM 2 is a large language model (LLM) developed by Google and announced at the annual Google I/O keynote in May 2023. It is the successor to the Pathways Language Model (PaLM), a 540 billion-parameter dense decoder-only transformer-based LLM developed by Google AI. PaLM 2 is reported to be a 340 billion-parameter model trained on 3.6 trillion tokens, representing a significant advancement in scale and capability over its predecessor.
The model is part of Google's broader efforts in Generative AI and Large language model research, building on the transformer architecture that underpins modern Deep learning systems. PaLM 2 was developed alongside Google DeepMind, reflecting the integration of Google's AI research divisions.
Architecture and Capabilities
PaLM 2 follows the decoder-only transformer design established by its predecessor, utilizing Multi-Head Attention mechanisms and Positional Encoding to process sequential data. The model is capable of a wide range of tasks, including commonsense reasoning, arithmetic reasoning, joke explanation, code generation, and translation. When combined with chain-of-thought prompting, PaLM 2 achieves significantly better performance on datasets requiring multi-step reasoning, such as word problems and logic-based questions.
The model's training on 3.6 trillion tokens - a substantial increase from the 780 billion tokens used for the original PaLM - enables enhanced performance across diverse domains. This training corpus includes filtered webpages, books, Wikipedia articles, news articles, source code from open source repositories, and social media conversations, with the social media portion comprising 50% of the corpus to support conversational capabilities.
Training Infrastructure
The original PaLM 540B was trained over two TPU v4 Pods with 3,072 TPU v4 chips in each Pod attached to 768 hosts, connected using a combination of model and data parallelism. This configuration, using 6,144 chips total, marked a record for the highest training efficiency achieved for LLMs at that scale, with a hardware FLOPs utilization of 57.8%. PaLM 2's training infrastructure follows similar principles, though specific details about its hardware configuration have not been fully disclosed.
Related Models and Applications
Google extended the PaLM architecture to create several specialized variants. Med-PaLM, a version fine-tuned on medical data, outperforms previous models on medical question-answering benchmarks and was the first to obtain a passing score on U.S. medical licensing questions. It provides reasoning and can evaluate its own responses in addition to answering multiple choice and open-ended questions accurately.
PaLM-E, another extension using a vision transformer, serves as a vision-language model for robotic manipulation without the need for retraining or fine-tuning. In June 2023, Google announced AudioPaLM for speech-to-speech translation, which uses the PaLM-2 architecture and initialization.
Availability and Impact
The original PaLM was first announced in April 2022 and remained private until March 2023, when Google launched an API initially available to a limited number of developers through a waitlist. PaLM 2's announcement at Google I/O in May 2023 signaled Google's commitment to competing in the rapidly evolving Artificial intelligence landscape, alongside other major players such as OpenAI and Anthropic. The model's capabilities have implications for Google Cloud services and broader enterprise applications, positioning Google as a significant force in the development of advanced Machine learning systems.