Google Neural Machine Translation (GNMT) was a neural machine translation system developed by Google and introduced in November 2016. It used an artificial neural network to increase fluency and accuracy in Google Translate, replacing the previous statistical machine translation methods. The system was designed to translate whole sentences at a time, leveraging deep learning to improve translation quality. By 2020, GNMT had been replaced by a newer deep learning system based on a Transformer encoder and an RNN decoder.
The GNMT architecture consisted of two main blocks: an encoder and a decoder, both based on LSTM (Long Short-Term Memory) networks. Each block had 8 layers with 1024 units per layer, and a simple 1-layer 1024-wide feedforward attention mechanism connected them. The total number of parameters was reported variously as over 160 million, approximately 210 million, 278 million, or 380 million. The system used a WordPiece tokenizer and a beam search decoding strategy, and it ran on Tensor Processing Units (TPUs).
History
The Google Brain project, which laid the groundwork for GNMT, was established in 2011 in the Google X research lab by Google Fellow Jeff Dean, Google Researcher Greg Corrado, and Stanford University professor Andrew Ng. In November 2016, GNMT was introduced, and Google Translate began using neural machine translation in preference to its previous statistical methods, which had been in use since October 2007.
Training GNMT was a significant computational effort. By a 2018 estimate from OpenAI, it required on the order of 79 petaFLOP-days (or 7e21 FLOPs) of compute, which was 1.5 orders of magnitude larger than the Seq2seq model of 2014 but about 2x smaller than GPT-J-6B in 2021.
The new engine was first enabled for eight languages: English and French, German, Spanish, Portuguese, Chinese, Japanese, Korean, and Turkish. In March 2017, support for Russian, Hindi, Vietnamese, and Thai was added. Hebrew and Arabic were added later that month with help from the Google Translate Community. In mid-April 2017, Dutch and other European languages related to English were supported. At the end of April 2017, nine Indian languages were added: Hindi, Bengali, Marathi, Gujarati, Punjabi, Tamil, Telugu, Malayalam, and Kannada.
By 2020, Google had phased out GNMT, replacing it with a transformer-based neural network system.
Technology
GNMT used an example-based machine translation method, learning from millions of examples of language translation. The system's architecture was first tested on over a hundred languages supported by Google Translate. With a large end-to-end framework, the system learned over time to create better, more natural translations. GNMT attempted to translate whole sentences at a time, rather than piece by piece, and could undertake interlingual machine translation by encoding the semantics of the sentence, rather than memorizing phrase-to-phrase translations.
The neural network was trained using deep learning techniques, and its design drew on insights from psychology and linguistics to find commonalities between languages. It did not create a universal interlingua but aimed to encode meaning in a language-independent way.
Evaluation
GNMT represented an improvement over the former Google Translate in that it could handle "zero-shot translation," directly translating one language into another without passing through English. For example, if trained on Japanese-English and Korean-English pairs, it could perform Japanese-Korean translation. The system appeared to learn a language-independent intermediate representation, enabling this capability. Previously, Google Translate first translated the source language into English and then into the target language.
A July 2019 study in Annals of Internal Medicine found that "Google Translate is a viable, accurate tool for translating non–English-language trials." Only one disagreement between reviewers reading machine-translated trials was due to a translation error. Since many medical studies are excluded from systematic reviews due to language barriers, GNMT had the potential to reduce bias and improve accuracy in such reviews.
Legacy
GNMT was a milestone in the application of deep learning to machine translation, demonstrating the effectiveness of Neural network based approaches over traditional statistical methods. Its architecture influenced subsequent developments in Transformer (architecture) models, which later became foundational in Large language models. The system's use of tensor-processing-units also highlighted the importance of specialized hardware for AI workloads. By 2020, the shift to transformer-based models marked the next evolution in translation technology, building on the groundwork laid by GNMT.