# BLEU

BLEU (bilingual evaluation understudy) is an algorithm for evaluating machine translation quality by comparing a candidate translation to reference human translations, producing a score from 0 to 1. It was invented at IBM in 2001 and remains a popular automated metric.

BLEU (bilingual evaluation understudy) is an algorithm for evaluating the quality of text that has been machine-translated from one natural language to another. The metric operates on the principle that a machine translation is better the closer it matches a professional human translation. Developed at IBM in 2001, BLEU was among the first automated metrics to show a high correlation with human judgments of translation quality retains its popularity due to its low cost and speed.

BLEU scores are calculated for individual translated segments - generally sentences - by comparing them against a set of high-quality reference translations. These segment scores are then averaged over the entire corpus to estimate overall translation quality. The metric does not consider intelligibility or grammatical correctness directly. Its output is always a number between 0 and 1, where values closer to 1 indicate greater similarity to the reference translations. Few human translations achieve a perfect 1.0, as that would require an exact match with a reference, and adding more reference translations increases the chance of matches and thus the score.

## Mathematical Definition

The BLEU score compares a candidate string ŷ with a list of reference strings (y^(1), ..., y^(N)). The score approaches 1 when the candidate closely resembles the references and 0 when it does not. A useful analogy is that of a language teacher scoring a student's translation by comparing it to reference answers.

For a corpus of M candidate strings, each with its own set of reference translations, BLEU computes a modified precision score based on n-grams. An n-gram is a contiguous sequence of n words from a text. The metric counts how many n-grams in the candidate translation appear in any of the reference translations, but clips the count to avoid overcounting repeated n-grams.

### Modified precision

The modified precision for n-grams (typically n = 1 to 4) is calculated by taking the number of candidate n-grams that match any reference n-gram and dividing by the total number of candidate n-grams. The brevity penalty addresses the tendency of shorter translations to achieve artificially high precision scores. It penalizes candidates shorter than their references.

The final BLEU score is the geometric mean of the n-gram precisions multiplied by the brevity penalty. Common practice uses a uniform weight over n-grams of orders 1 through 4.

== History ==

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu developed BLEU while working at IBM's Thomas J. Watson Research Center. They published it in 2002 at the Annual Meeting of the Association for Computational Linguistics (ACL). The metric emerged as one of the first automated evaluation methods with claimed high correlation to human judgment of translation quality. Its introduction addressed the need for a fast, inexpensive alternative to human evaluation, which was costly and slow. BLEU quickly became a standard benchmark in [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) and natural language processing research, adopted widely in academic papers and industry evaluations.

## Mathematical basis

BLEU scores are calculated per segment, typically a sentence, by comparing the candidate translation against one or more reference translations. The algorithm computes a modified n-gram precision: it counts the number of n-grams (contiguous sequences of n words) in the candidate that appear in any reference translation, with a clipping mechanism that prevents over-counting when a candidate repeats an n-gram more times than it occurs in a reference. This precision is computed for a range of n-gram lengths, commonly 1 through 4.

A brevity penalty is applied to the score to discourage candidates that are too short relative to the referencesesley because short outputs could otherwise achieve high precision by including only words that match. The brevity penalty is calculated from the ratio of the candidate length to the effective reference length, ensuring that incomplete translations are penalized. The final BLEU score is a geometric mean of the modified n-gram precisions multiplied by this brevity penalty factor.

The output always falls between 0 and 1retains its popularity due to the low cost and speed of computation. Few human translations attain a perfect score of 1, since that would indicate the candidate is identical to one of the reference translations. Because there are more opportunities for n-gram matches, including more reference translations will generally increase the BLEU score.

## History and Development

BLEU was introduced in 2001 by researchers at IBM, including Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. The metric was designed to address the need for an inexpensive, language-independent evaluation method for machine translation, replacing costly and slow human evaluation. The name, bilingual evaluation understudy, reflects its role as a stand-in for human judges.

The metric's introduction coincided with growing interest in statistical machine translation, where rapid iteration required an automated way to compare system outputs. BLEU's simplicity and strong correlation with human judgment made it a standard benchmark in the field for years.

## N-gram Precision and Brevity Penalty

BLEU computes precision for n-grams of varying lengths, typically from 1 to 4. The modified precision for each n-gram order counts the maximum number of times an n-gram appears in any single reference, capping the candidate count at that maximum. This prevents overly long or repetitive translations from gaming the score.

A brevity penalty is applied to candidate translations shorter than their reference counterparts. If the candidate is longer than the reference, the penalty is 1; otherwise, it decreases exponentially with the length ratio. This penalty discourages incomplete translations that might otherwise achieve high precision.

## Geometric Mean and Score Range

The final BLEU score combines the modified precisions across n-gram orders using a geometric mean, typically for n = 1 to 4. The brevity penalty multiplies this mean. BLEU scores typically fall between 0 and 1, though they are often reported as percentages on a 0 to 100 scale. Scores above 0.4 are generally considered high-quality translations, but the scale is not absolute and depends heavily on the language pair and domain.

## Applications and Limitations

BLEU remains one of the most widely used metrics in machine translation research. It has been employed to evaluate systems in academic papers, industry benchmarks, and shared tasks. The metric is also used in fields beyond translation, such as text summarization and image captioning, where reference texts serve as ground truth.

Critics note that BLEU does not directly measure intelligibility or grammatical correctness)Skip a candidate that is fluent but semantically different from the reference can score low, while one that is awkward but shares n-grams can score higher. The metric's n-gram matching is purely lexical mirroring, so paraphrasing and synonymy are penalized even when meaning is preserved. Correlation with human judgment is strong at the corpus level but weaker for individual sentences.

## Applications and Limitations

BLEU remains one of the most popular automated metrics in [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) research and in the development of [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)-based translation systems. It is frequently used to compare model iterations in [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) development and to tune hyperparameters in [neural-network](https://www.wikiprompt.org/wiki/neural-network) models. Its low computational cost makes it suitable for large-scale evaluation during training and development cycles.

Limitations include sensitivity to reference translation quality, no consideration of semantic equivalence, and the need for multiple references for reliable scores. BLEU also tends to favor translations that closely mirror reference phrasing, potentially penalizing valid alternative translations. Subsequent metrics, such as METEOR and ROUGE, have attempted to address some of these shortcomings, though BLEU remains a widely cited baseline in [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) research.

## Applications and Influence

BLEU is used across the field of [natural language processing](https://www.wikiprompt.org/wiki/natural-language-processing), particularly in training and evaluating translation systems developed with [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) approaches such as [transformer](https://www.wikiprompt.org/wiki/transformer)-based models. It has also been adapted for other generation tasks, including text summarization and image captioning, despite its original design for translation. The metric's influence extends beyond academia; major [AI](https://www.wikiprompt.org/wiki/artificial-intelligence) laboratories and companies, including [openai](https://www.wikiprompt.org/wiki/openai), [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind), and [anthropic](https://www.wikiprompt.org/wiki/anthropic), have used BLEU or its variants when benchmarking generative models.

Standard implementations, such as those in the NLTK (Natural Language Toolkit) library, provide easy-to-use functions for computing BLEU scores flatter. This accessibility has cemented its role as a default metric in natural language processing evaluations. However, researchers have noted that BLEU does not capture semantic equivalence well; two translations with identical meaning but different word choices can receive low scores despite being high quality. This limitation has led to the development of alternatives, but BLEU remains a foundational baseline in machine translation evaluation.

## Limitations and Criticism

BLEU has several known limitations. It ignores grammatical correctness and does not explicitly account for word order beyond n-gram matching. It also fails to recognize synonyms or paraphrases that do not match reference wording. These weaknesses can lead to cases where a translation is accurate but scores poorly, or where a flawed translation with high lexical overlap scores well. 

Additionally, BLEU scores are sensitive to the number and quality of reference translations. Systems can optimize for BLEU by favoring common phrases, sometimes at the expense of fluency or adequacy. Despite these issues, BLEU correlates reasonably with human judgment at the corpus level)Skip and remains widely used for quick system comparisons, though newer metrics increasingly complement or replace it in research settings.

## Applications and Limitations

BLEU's primary application is in evaluating machine translation systems, from early statistical phrase-based models to modern [neural-network](https://www.wikiprompt.org/wiki/neural-network) and [transformer](https://www.wikiprompt.org/wiki/transformer)-based approaches. It has also been adapted for tasks like text summarization and image captioning. Its low computational cost makes it ideal for iterative development, where thousands of translations must be scored quickly during [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) training loops.

However, BLEU has known limitations. It does not capture semantic equivalence, so paraphrases with different word choices can score poorly. It also struggles with morphologically rich languages and does not account for word order beyond n-gram adjacency. These shortcomings have driven the development of alternative metrics, though BLEU remains a baseline reference in most [large language model](https://www.wikiprompt.org/wiki/large-language-model) and [neural-network](https://www.wikiprompt.org/wiki/neural-network) translation research.

## Applications and Limitations

BLEU is widely used in academic research and industry to compare translation systems, including those built on [transformer](https://www.wikiprompt.org/wiki/transformer) architectures and [large language models](https://www.wikiprompt.org/wiki/large-language-model). It serves as a quick regression check during model development, complementing human evaluation. However, BLEU has known limitations: it correlates imperfectly with human judgments, especially for morphologically rich languages or when paraphrases are valid. It also rewards lexical overlap over semantic equivalence, a gap that later metrics like METEOR and ROUGE sought to address, though BLEU remains the most cited.

Despite these drawbacks, BLEU's simplicity and reproducibility have ensured its continued use in machine translation research and in related tasks. Its role in the field is foundational, influencing the design of evaluation protocols for [neural-network](https://www.wikiprompt.org/wiki/neural-network)-based translation systems and beyond.

## See Also

- [Machine-learning](https://www.wikiprompt.org/wiki/machine-learning)
- [Natural-language-processing](https://www.wikiprompt.org/wiki/natural-language-processing)

---
Source: https://www.wikiprompt.org/wiki/bleu
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-07T21:29:51.457906+00:00