PaLM 2 (Pathways Language Model 2) is a large language model developed by Google AI, announced at the annual Google I/O keynote in May 2023. It is an updated version of the original PaLM model, with a reported 340 billion parameters and training on 3.6 trillion tokens. PaLM 2 is designed to excel in a wide range of tasks, including commonsense reasoning, arithmetic reasoning, code generation, and translation, with a particular emphasis on improved multilingual skills.
The original PaLM model, introduced in April 2022, was a 540 billion-parameter dense decoder-only transformer-based large language model. Researchers also trained smaller versions with 8 and 62 billion parameters to study the effects of model scale. PaLM 2 builds on this foundation, offering enhanced performance and broader language coverage.
Architecture and Training
PaLM 2 is a dense decoder-only transformer model, similar to its predecessor. It is pre-trained on a high-quality corpus of 780 billion tokens, which includes filtered webpages, books, Wikipedia articles, news articles, source code from open-source repositories on GitHub, and social media conversations. The dataset is based on the one used to train Google's LaMDA model, with social media conversations making up 50% of the corpus to enhance conversational abilities.
The original PaLM 540B was trained over two TPU v4 Pods, each with 3,072 TPU v4 chips attached to 768 hosts, using a combination of model and data parallelism. This configuration, totaling 6,144 chips, achieved a record hardware FLOPs utilization of 57.8%, marking the highest training efficiency for large language models at that scale.
Capabilities and Variants
PaLM 2 is capable of a wide range of tasks, including commonsense reasoning, arithmetic reasoning, joke explanation, code generation, and translation. When combined with chain-of-thought prompting, it achieves significantly better performance on datasets requiring multi-step reasoning, such as word problems and logic-based questions.
Google and DeepMind developed Med-PaLM, a version of PaLM 540B fine-tuned on medical data, which outperforms previous models on medical question-answering benchmarks. Med-PaLM was the first to obtain a passing score on U.S. medical licensing questions, providing reasoning and self-evaluation in addition to accurate answers.
Google also extended PaLM using a vision transformer to create PaLM-E, a vision-language model for robotic manipulation without the need for retraining or fine-tuning. In June 2023, Google announced AudioPaLM for speech-to-speech translation, which uses the PaLM-2 architecture and initialization.
Release and Access
The original PaLM remained private until March 2023, when Google launched an API for PaLM and several other technologies. The API was initially available to a limited number of developers who joined a waitlist before public release. PaLM 2 was announced in May 2023 and is expected to be integrated into various Google products and services.
Impact and Legacy
PaLM 2 represents a significant advancement in large language models, particularly in multilingual understanding and reasoning. Its development aligns with ongoing efforts in the field of artificial intelligence and machine learning, contributing to the broader landscape of generative AI. PaLM 2 is considered a successor to LaMDA and a predecessor to Gemini, reflecting Google's continuous innovation in this domain.