Google PaLM 2 is a large language model (LLM) developed by Google AI, announced at the annual Google I/O keynote in May 2023. It is the successor to the Pathways Language Model (PaLM), a 540 billion-parameter dense decoder-only transformer-based model first introduced in April 2022. PaLM 2 is reported to be a 340 billion-parameter model trained on 3.6 trillion tokens, a significant increase from its predecessor's 780 billion tokens. The model powers Bard, Google's conversational AI service, and other Google AI products, with enhanced multilingual capabilities, improved reasoning, and better coding proficiency.
PaLM 2 builds on the architectural foundations of PaLM, which was designed as a dense decoder-only transformer. The original PaLM model demonstrated strong performance across tasks such as commonsense reasoning, arithmetic reasoning, joke explanation, code generation, and translation. When combined with chain-of-thought prompting, PaLM achieved significantly better results on datasets requiring multi-step reasoning, including word problems and logic-based questions. PaLM 2 extends these capabilities, particularly in multilingual contexts, allowing it to understand and generate text across a wider range of languages with greater accuracy.
Development and Announcement
The original PaLM model was first announced in April 2022 but remained private until March 2023, when Google launched an API for PaLM and several other technologies. The API was initially available to a limited number of developers who joined a waitlist before it was released to the public. PaLM 2 was unveiled at Google I/O in May 2023, marking a major milestone in Google's Generative AI efforts. The announcement highlighted PaLM 2's improved efficiency and performance over its predecessor, with a focus on practical applications in Google's product ecosystem.
Google and DeepMind also developed specialized versions of PaLM. Med-PaLM, a version of PaLM 540B fine-tuned on medical data, outperformed previous models on medical question-answering benchmarks. It was the first model to obtain a passing score on U.S. medical licensing questions, providing reasoning and self-evaluation alongside accurate answers. Another variant, PaLM-E, extended PaLM using a vision transformer to create a vision-language model for robotic manipulation without retraining or fine-tuning. In June 2023, Google announced AudioPaLM for speech-to-speech translation, which uses the PaLM-2 architecture and initialization.
Architecture and Training
PaLM 2 is a dense decoder-only Transformer (architecture) model, similar to its predecessor but with a larger training corpus. The original PaLM was pre-trained on a high-quality corpus of 780 billion tokens, including filtered webpages, books, Wikipedia articles, news articles, source code from open-source repositories on GitHub, and social media conversations. The social media conversation portion made up 50% of the corpus, aiding conversational capabilities. PaLM 2's training data expanded to 3.6 trillion tokens, enabling better generalization and multilingual performance.
The original PaLM 540B was trained over two TPU v4 Pods, each with 3,072 TPU v4 chips attached to 768 hosts, using a combination of model and data parallelism. This configuration, totaling 6,144 chips, marked a record for the highest training efficiency achieved for LLMs at that scale, with a hardware FLOPs utilization of 57.8%. PaLM 2's training details are less publicly documented, but it likely used similar infrastructure, reflecting Google's investment in Google Cloud and specialized hardware.
Capabilities and Applications
PaLM 2 is designed to excel in multilingual tasks, reasoning, and coding. It powers Bard, Google's conversational AI, and is integrated into other products like Google Workspace and Google Cloud's AI services. The model's improved multilingual abilities allow it to handle translation, summarization, and question-answering across languages more effectively than earlier models. Its reasoning capabilities are enhanced through techniques like chain-of-thought prompting, enabling multi-step problem solving.
In coding, PaLM 2 supports code generation and debugging across multiple programming languages, making it a tool for developers. The model's versatility is demonstrated in applications ranging from Machine learning research to enterprise solutions. Google has positioned PaLM 2 as a foundational model for its AI ecosystem, competing with offerings from OpenAI and Anthropic.
Comparison with Other Models
PaLM 2's 340 billion parameters place it in the upper tier of Large language models, though smaller than some competitors like OpenAI's GPT-4, which is reported to have over a trillion parameters in a mixture-of-experts architecture. However, PaLM 2's training on 3.6 trillion tokens and its focus on efficiency allow it to achieve competitive performance on benchmarks. The model's architecture, based on the Transformer (architecture) framework, is similar to that of other LLMs, but Google's proprietary training techniques and infrastructure give it distinct advantages in scalability.
Compared to its predecessor PaLM, PaLM 2 offers significant improvements in multilingual understanding and reasoning, as well as reduced computational requirements for inference. This makes it more accessible for deployment in real-world applications. Google's integration of PaLM 2 into products like Bard and Google Cloud's Vertex AI has broadened its reach, positioning it as a key player in the Artificial intelligence landscape.
Impact and Reception
The launch of PaLM 2 in May 2023 was met with attention from the AI community and industry observers. It demonstrated Google's commitment to advancing Deep learning and Neural network technologies, particularly in the competitive field of generative AI. The model's improved multilingual capabilities were seen as a step forward in making AI accessible to non-English speakers, and its coding abilities appealed to developers.
However, PaLM 2 also raised questions about the environmental impact of training large models, though Google has emphasized efficiency improvements. The model's integration into Bard sparked discussions about the ethics of AI, including issues of bias and misinformation, which are common concerns for Generative AI systems. Google has implemented safety measures, but the broader implications remain a topic of debate.
Future Directions
Following PaLM 2, Google continued to evolve its AI models. In late 2023, Google announced Gemini, a successor to PaLM 2, which integrates multimodal capabilities and is designed to be more powerful and versatile. PaLM 2's architecture and training methods have influenced subsequent developments, including improvements in Multi-Head Attention and Positional Encoding techniques. The model's legacy is evident in Google's ongoing efforts to push the boundaries of Artificial intelligence, with applications in fields ranging from healthcare to robotics.
As of 2025, PaLM 2 remains a significant milestone in the history of LLMs, representing a bridge between earlier models like LaMDA and more advanced systems like Gemini. Its contributions to multilingual AI and efficient training have had a lasting impact on the field, and its principles continue to inform research at Google DeepMind and beyond.
See Also
- LaMDA, PaLM's predecessor
- Gemini, PaLM's successor
- Chinchilla, a related LLM
- Transformer (architecture), the underlying architecture
- Google Cloud, infrastructure for training
References
- Google AI Blog, May 2023, PaLM 2 announcement
- Wikipedia, PaLM (Pathways Language Model), accessed 2025
- Google Research, PaLM: Scaling Language Modeling with Pathways, 2022