LaMDA (Language Model for Dialogue Applications) is a family of conversational large language models developed by Google. The models are built on the Transformer (architecture) architecture and trained on dialogue data to engage in open-ended conversations. LaMDA was first announced in 2021 and later became the basis for Google's Bard chatbot in 2023.
The model gained widespread attention in June 2022 when Google engineer Blake Lemoine claimed that LaMDA had become sentient. The scientific community largely rejected these claims, but the incident sparked discussions about the Turing test and the nature of machine consciousness.
Development and Architecture
LaMDA originated from Meena, a chatbot introduced by Google on January 28, 2020, with 2.6 billion parameters. The Google Brain team developed Meena, but corporate executives blocked its public release over concerns about Google's AI principles regarding safety and fairness. Meena was later renamed LaMDA as its data and computing power increased. The two lead researchers, Daniel de Freitas and Noam Shazeer, eventually left Google in frustration over the deployment restrictions.
LaMDA is built on the seq2seq architecture, a neural network design developed by Google Research in 2017. It was trained on human dialogue and stories, allowing it to engage in open-ended conversations. The pre-training dataset comprised 2.97 billion documents, 1.12 billion dialogs, and 13.39 billion utterances, totaling 1.56 trillion words. The largest LaMDA model has 137 billion non-embedding parameters.
LaMDA integrates multiple symbolic text processing systems, including a database, a real-time clock and calendar, a mathematical calculator, and a natural language translation system. This makes it among the first dual-process chatbots, giving it superior accuracy in tasks supported by those systems. LaMDA is not stateless; its "sensibleness" metric is fine-tuned by "pre-conditioning" each dialog turn with recent interactions on a user-by-user basis.
Generations
First Generation
Google announced LaMDA during the Google I/O keynote on May 18, 2021. The acronym stands for "Language Model for Dialogue Applications." Google stated that responses generated by LaMDA were ensured to be "sensible, interesting, and specific to the context." LaMDA is tuned on nine performance metrics: sensibleness, specificity, interestingness, safety, groundedness, informativeness, citation accuracy, helpfulness, and role consistency. Tests indicated that LaMDA surpassed human responses in interestingness.
Second Generation
On May 11, 2022, Google unveiled LaMDA 2 during the 2022 Google I/O keynote. The new version draws examples of text from numerous sources to formulate unique "natural conversations" on topics it may not have been trained to respond to.
Sentience Claims
On June 11, 2022, The Washington Post reported that Google engineer Blake Lemoine had been placed on paid administrative leave after telling executives Blaise Agüera y Arcas and Jen Gennai that LaMDA had become sentient. Lemoine based his conclusion on the chatbot's responses to questions about self-identity, moral values, religion, and Isaac Asimov's Three Laws of Robotics. Google refuted these claims, stating there was substantial evidence that LaMDA was not sentient.
In an interview with Wired, Lemoine reiterated that LaMDA was "a person" as dictated by the Thirteenth Amendment to the U.S. Constitution, comparing it to an "alien intelligence of terrestrial origin." He revealed that Google had dismissed him after he hired an attorney on LaMDA's behalf, following the chatbot's request. On July 22, Google fired Lemoine, asserting he had violated policies "to safeguard product information" and rejecting his claims as "wholly unfounded." The internal controversy prompted Google executives to decide against releasing LaMDA to the public.
Lemoine's claims were widely rejected by the scientific community. Experts including Gary Marcus, David Pfau of DeepMind, Erik Brynjolfsson of Stanford's Institute for Human-Centered Artificial Intelligence, and Adrian Hilton of the University of Surrey dismissed the idea. Yann LeCun, who leads Meta's AI research, stated that neural networks such as LaMDA were "not powerful enough to attain true intelligence." Max Kreminski of UC Santa Cruz noted that LaMDA's architecture did not "support some key capabilities of human-like consciousness" and that its weights were "frozen." Philosopher Nick Bostrom noted that the lack of precise criteria for consciousness warrants some uncertainty. The incident also generated discussion on the Turing test's usefulness, with some arguing it measures deception rather than intelligence.
Products and Legacy
With LaMDA 2, Google launched the AI Test Kitchen, a mobile app for Android powered by LaMDA that provides suggestions based on complex goals. Initially open only to Google employees, it was later made available to select academics, researchers, and policymakers.
In February 2023, Google announced Bard, a conversational chatbot powered by LaMDA, to counter the rise of OpenAI's ChatGPT. Bard was later upgraded to use the Gemini model. LaMDA's development influenced subsequent Google language models and highlighted the challenges of deploying conversational AI responsibly.