# METEOR METEOR (Metric for Evaluation of Translation with Explicit ORdering) is an automatic metric for evaluating machine translation output, based on the harmonic mean of unigram precision and recall with recall weighted higher than precision. It was designed to address shortcomings in the BLEU metric by incorporating stemming and synonym matching. METEOR (Metric for Evaluation of Translation with Explicit ORdering) is a metric for evaluating the quality of machine translation output. It computes a score based on the harmonic mean of unigram precision and recall, with recall weighted higher than precision, and incorporates stemming and synonymy matching in addition to exact word matching. The metric was developed to address limitations in the BLEU metric and to produce a stronger correlation with human judgment at the sentence or segment level, whereas BLEU aims for correlation at the corpus level. The metric was introduced at the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and Summarization, held at the 43rd Annual Meeting of the Association for Computational Linguistics in Ann Arbor, Michigan, in June 2005. Reported results showed correlation with human judgment up to 0.964 at the corpus level, compared to BLEU's 0.817 on the same data sets. At the sentence level, the maximum reported correlation was 0.403. The metric's name is a recursive acronym standing for Metric for Evaluation of Translation with Explicit ORdering. ## Motivation and design METEOR was developed partly in response to limitations of the earlier BLEU metric, which had become a standard for [machine translation](https://www.wikiprompt.org/wiki/machine-learning) evaluation. BLEU operates primarily at the corpus level and relies on exact word matching with limited n-gram precision. It showed weak correlation with human judgment when applied to individual sentences. METEOR addressed these shortcomings by weighting recall more heavily than precision (a ratio of 9 to 1 in the harmonic mean), incorporating [linguistic](https://www.wikiprompt.org/wiki/natural-language-processing) resources for matching beyond exact surface forms, and producing scores that could be meaningfully compared at the segment level. The scoring procedure begins with an alignment step. The algorithm builds mappings between unigrams in the candidate translation and unigrams in a reference translation. Each unigram in the candidate may map to at most one unigram in the reference. When multiple alignments are possible with the same number of mappings, the algorithm prefers the alignment with fewer crossings between mapping lines, which favors monotonic, contiguous matches. ## Matching stages METEOR applies matching in several stages. The first stage uses exact word matching. Subsequent stages add matches based on stemming and synonymy, drawing on resources such as WordNet for English. Each stage only adds unigrams not already matched in earlier stages. This staged approach allows the metric to credit semantically equivalent but lexically different expressions, a capability that earlier metrics lacked. After alignment, precision and recall are computed from the number of matched unigrams. Recall is weighted more heavily than precision in the final F-mean score, using the formula 10PR/(R+9P). This weighting reflects findings that recall correlates more strongly with human judgment in translation quality. The final score is adjusted by a fragmentation penalty derived from the number of chunks of matched unigrams; longer contiguous matches produce fewer chunks and a lower penalty. ## Algorithm overview The algorithm operates on a sentence pair: a candidate translation and a reference translation. It constructs an alignment by mapping unigrams under constraints, favoring alignments with fewer crossings. Matching proceeds in stages, where each stage adds matches based on exact word forms, then stems, then synonyms. After alignment, unigram precision is the number of matched unigrams divided by the total unigrams in the candidate, while recall divides by the total unigrams in the reference. These are combined as a harmonic mean with recall weighted nine times more than precision. A penalty is then applied based on how fragmented the matched unigrams are into contiguous chunks; translations with many non-adjacent matches receive a higher penalty, reducing the final score by up to 50 percent. For a corpus, aggregate precision, recall, and penalty values are computed across all segments. When multiple reference translations are available, the candidate is scored against each and the highest score is retained. ## Comparison with BLEU BLEU, introduced earlier, measures n-gram overlap with a brevity penalty and is widely used in machine translation research. However, it relies on exact word matching and does not account for synonyms or morphological variants. METEOR was designed to overcome these issues by using stemming and synonym mapping, and by weighting recall more heavily. Reported experiments indicated that METEOR achieved a correlation with human judgment of up to 0.964 at the corpus level, while BLEU achieved 0.817 on the same data. At the segment level, METEOR showed a maximum correlation of 0.403, which proponents argued was better suited for evaluating individual sentences. ## Algorithm details The alignment process in METEOR maps unigrams from the candidate translation to unigrams in the reference, with each candidate unigram mapping to at most one reference unigram. The algorithm selects the alignment with the greatest number of mappings; when two alignments have the same number of mappings, it chooses the one with fewer crossing lines between matched unigrams. Matching proceeds in stages: first exact word matches, then stem matches (using a stemming tool), then synonym matches. Each stage only adds mappings for unigrams not already matched in prior stages. After the final alignment is computed, precision P and recall R are calculated. The harmonic mean Fmean is defined as 10PR/(R+9P), giving recall nine times the weight of precision. To account for fluency and word order, the algorithm groups matched unigrams into the fewest possible chunks, where a chunk is a set of unigrams that are adjacent in both the candidate and the reference. The penalty p is computed as 0.5 × (c/um)^3, where c is the number of chunks and um is the number of mapped unigrams. The final segment score is Fmean multiplied by (1 - p). ## Applications and extensions METEOR has been used in evaluation campaigns for machine translation, particularly in shared tasks and system comparisons where sentence-level reliability matters. Because it can be applied at the segment level, it is useful for tuning and analyzing translation outputs in ways that corpus-level metrics cannot easily support. Variants and extensions have appeared over time, including versions tailored for different language pairs and for summarization evaluation. The metric has also influenced subsequent evaluation approaches in related fields such as image captioning, where synonym matching and n-gram overlap have proven useful. ## Limitations The metric still relies on reference translations and does not capture aspects of translation quality such as fluency beyond what is implied by lexical overlap. Its reliance on predefined synonym resources can limit performance for domain-specific terminology or languages lacking such resources. While the correlation with human judgment at the corpus level was reported as high in initial experiments, sentence-level correlation remained moderate, reflecting the inherent difficulty of segment-level automated evaluation. Later variants and alternative metrics have continued to build on these ideas. ## See also - [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) - [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) - [natural language processing](https://www.wikiprompt.org/wiki/natural-language-processing) - [large language models](https://www.wikiprompt.org/wiki/large-language-model) --- Source: https://www.wikiprompt.org/wiki/meteor License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-08T06:10:03.291301+00:00