BLEU (bilingual evaluation understudy) is an algorithm for evaluating the quality of text that has been machine-translated from one natural language to another. Quality is considered to be the correspondence between a machine's output and that of a human: "the closer a machine translation is to a professional human translation, the better it is" - this is the central idea behind BLEU. Invented at IBM in 2001, BLEU was one of the first metrics to claim a high correlation with human judgements of quality, and remains one of the most popular automated and inexpensive metrics.
Scores are calculated for individual translated segments - generally sentences - by comparing them with a set of good quality reference translations. Those scores are then averaged over the whole corpus to reach an estimate of the translation's overall quality. Intelligibility or grammatical correctness are not taken into account. BLEU's output is always a number between 0 and 1, indicating how similar the candidate text is to the reference texts, with values closer to 1 representing more similar texts. Few human translations will attain a score of 1, since this would indicate that the candidate is identical to one of the reference translations. For this reason, it is not necessary to attain a score of 1. Because there are more opportunities to match, adding additional reference translations will increase the BLEU score.
Historical Context
BLEU was developed by researchers at IBM, including Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, who published the metric in a 2002 paper titled "BLEU: a Method for Automatic Evaluation of Machine Translation." The work emerged from the growing need for automated evaluation in the field of Machine learning and Artificial intelligence, particularly as machine translation systems became more sophisticated. Before BLEU, evaluation of machine translation typically relied on human judges, which was expensive, time-consuming, and difficult to replicate across different studies. BLEU offered a cheap, fast, and language-independent alternative that could be applied consistently to any translation system.
The metric was introduced during a period when statistical machine translation was gaining traction, and it quickly became a standard benchmark in the field. Its development was closely tied to the work of IBM's research division, which had been a major player in natural language processing since the 1980s. The name "bilingual evaluation understudy" reflects its role as a stand-in for human evaluation, much like an understudy actor who fills in for a lead performer.
Mathematical Definition
Basic Setup
A basic, first attempt at defining the BLEU score would take two arguments: a candidate string and a list of reference strings. The idea is that the BLEU score should be close to 1 when the candidate is similar to the references, and close to 0 if not. As an analogy, the BLEU score is like a language teacher trying to score the quality of a student translation by checking how closely it follows the reference answers.
Since in natural language processing, one should evaluate a large set of candidate strings, one must generalize the BLEU score to the case where one has a list of M candidate strings (called a "corpus"), and for each candidate string, a list of reference candidate strings. Given any string and any integer n ≥ 1, the set of its n-grams is defined as the set of unique contiguous sequences of n words. For example, the 2-grams of the string "the cat sat" would be {"the cat", "cat sat"}.
Precision and Modified Precision
The core of BLEU is a modified form of precision. Standard precision would count the number of n-grams in the candidate that appear in any reference, divided by the total number of n-grams in the candidate. However, this can be inflated by repeating common words. BLEU uses a clipped precision: for each n-gram in the candidate, its count is clipped to the maximum number of times it appears in any single reference. This prevents overcounting and ensures that the score reflects genuine correspondence.
For each n-gram order n (typically 1 through 4), the modified precision is computed as the sum of clipped counts over all candidate sentences, divided by the sum of total n-gram counts in all candidates. This is often denoted as p_n.
Brevity Penalty
A critical component of BLEU is the brevity penalty, which addresses the problem of short translations. A system could achieve high precision by outputting only a few words that happen to match references, but such a translation would be incomplete. The brevity penalty is calculated based on the ratio of the total length of the candidate corpus to the total length of the reference corpus. If the candidate is shorter than the reference, the penalty reduces the score; if it is longer, no penalty is applied. The formula is: BP = 1 if c > r, and BP = exp(1 - r/c) if c ≤ r, where c is the candidate length and r is the effective reference length (the sum of the reference lengths closest to each candidate's length).
Final Score
The final BLEU score is the product of the brevity penalty and the geometric mean of the modified precisions for n-grams of orders 1 through N (usually N=4). The geometric mean is weighted equally, and the result is a value between 0 and 1. In practice, scores are often reported as percentages (e.g., BLEU 30 means 0.30). The formula is: BLEU = BP exp(Σ_{n=1}^{N} (1/N) log p_n).
Applications in Machine Translation
BLEU was originally designed for evaluating machine translation systems, and it remains a standard metric in that domain. Research groups and companies developing translation systems, from early statistical models to modern Neural network-based approaches, have used BLEU to compare their outputs against reference translations. For example, systems from Google DeepMind and other organizations have reported BLEU scores in academic papers to demonstrate improvements over baselines.
The metric is particularly useful for rapid iteration during development. Developers can run BLEU on a test set after each change to a model, allowing them to track progress without needing human annotators. This has made BLEU a de facto standard in the field, despite its known limitations.
Use in Text Generation and Large Language Models
Beyond machine translation, BLEU has been adapted for evaluating other text generation tasks, including summarization, dialogue systems, and image captioning. With the rise of Large language models such as those developed by OpenAI and Anthropic, BLEU has been used as one of several automated metrics to assess output quality. However, its applicability to open-ended generation is debated, since BLEU rewards exact n-gram overlap with references, which may not capture semantic equivalence.
In tasks like summarization, BLEU is often used alongside other metrics such as ROUGE. For dialogue systems, BLEU can be misleading because there are many valid responses to a given prompt, and a single reference may not represent all possibilities. Despite these issues, BLEU remains a quick and inexpensive way to get a rough estimate of quality, and it is often reported in research papers as a baseline metric.
Limitations and Criticisms
BLEU has several well-documented limitations. First, it does not account for intelligibility or grammatical correctness; a translation could score high if it shares many n-grams with references but is still ungrammatical. Second, it is sensitive to the choice of references; different reference sets can yield very different scores for the same candidate. Third, BLEU tends to favor translations that are lexically similar to references, which can penalize valid translations that use different wording or sentence structure.
Another criticism is that BLEU does not handle synonyms or paraphrases well. For instance, if a reference says "the cat sat on the mat" and a candidate says "the feline rested on the rug," the BLEU score would be low despite the semantic equivalence. This has led to the development of alternative metrics, such as METEOR and ROUGE, which incorporate stemming and synonym matching. In the context of Generative AI, researchers have also noted that BLEU correlates poorly with human judgment for creative or open-ended outputs.
Relationship to Other Metrics
BLEU is part of a broader family of automated evaluation metrics in natural language processing. It is precision-oriented, focusing on how much of the candidate's content appears in references. In contrast, ROUGE is recall-oriented, measuring how much of the reference content is captured by the candidate. METEOR combines precision and recall with synonym matching and a penalty for word order differences. These metrics are often used together to provide a more comprehensive evaluation.
In the context of Transformer (architecture)-based models and Deep learning systems, BLEU has been used as a training objective in some reinforcement learning setups, though this is less common than using it purely for evaluation. The metric's simplicity and computational efficiency make it attractive for large-scale benchmarking, such as in shared tasks organized by academic conferences.
Practical Considerations
When using BLEU, several practical factors affect the score. The number of reference translations matters: adding more references generally increases the score because there are more opportunities for n-gram matches. The tokenization method also influences results; for example, whether punctuation is stripped or case is normalized can change the n-gram counts. Different implementations may produce slightly different scores for the same data, so it is important to report the exact settings used.
BLEU scores are not comparable across different corpora or languages. A score of 0.40 on one test set may be excellent, while on another it may be mediocre. Therefore, BLEU is best used for relative comparisons between systems on the same data, rather than as an absolute measure of quality. Researchers often report BLEU alongside human evaluation to validate findings.
Future Directions
As Artificial intelligence and Machine learning continue to evolve, the role of BLEU is being reassessed. Newer metrics, such as BERTScore and COMET, use embeddings from pre-trained models to capture semantic similarity more effectively. These metrics often correlate better with human judgment but are more computationally expensive. BLEU, however, remains a useful baseline due to its simplicity and interpretability.
In the era of Large language models, evaluation is increasingly focused on tasks like instruction following and factual accuracy, where BLEU is less relevant. Nevertheless, BLEU's legacy as one of the first automated evaluation metrics has shaped how the field approaches quality assessment. Its principles - comparing machine output to human references - continue to inform the design of new metrics.