# Expectation–maximization algorithm

An iterative statistical method for finding maximum likelihood or MAP estimates of parameters in models with unobserved latent variables, alternating between expectation and maximization steps until convergence.

The expectation-maximization (EM) algorithm is an iterative method used in statistics to find maximum likelihood or maximum a posteriori estimates of parameters in statistical models where the model depends on unobserved latent variables. It is particularly useful when the equations for the parameters cannot be solved directly, such as in mixture models or when data contains missing values.

The EM iteration alternates between an expectation (E) step, which computes the expected log-likelihood of the complete data given the current parameter estimates, and a maximization (M) step, which updates the parameters by maximizing that expected log-likelihood. These updated parameter estimates are then used in the next E step, and the process repeats until convergence. The algorithm is guaranteed to converge to a local maximum or saddle point of the likelihood function, but not necessarily the global maximum.

## Historical Development

The EM algorithm was formally named and explained in a 1977 paper by Arthur Dempster, Nan Laird, and Donald Rubin, later known as the DLR paper. That work established the method as a core tool of statistical analysis. However, earlier authors had proposed the technique in specific cases.

One precursor was the gene-counting method developed by Cedric Smith for estimating allele frequencies. H.O. Hartley also proposed an early version in 1958, and Hartley and Hocking expanded on it in 1977. Rolf Sundberg provided a detailed treatment for exponential families in his thesis and subsequent papers, following collaboration with Per Martin-Löf and Anders Martin-Löf.

The 1977 DLR paper generalized these earlier methods and sketched a convergence analysis for a broad class of problems. However, that analysis had shortcomings, and a correct convergence proof was later published in 1983 by C. F. Jeff Wu, who established convergence outside the exponential family as well.

## Core Idea and Interlocking Equations

In statistical models with latent variables, maximum likelihood estimation typically requires solving equations that involve both chains. The solution to the parameters requires the values of the latent variables, and thereof require the parameters, leading to a mutually interdependent system that cannot be solved analytically.

The EM algorithm resolves this by initializing one set of values (often arbitrary guesses for the parameters) and alternating between estimation steps. For example, it can estimate latent variables based on current parameters, then use those latent variables to update the parameters, repeating the cycle until both sets converge to a fixed point. While intuitively simple, the method has a proven convergence property: the derivative of the likelihood approaches zero at the final point.

## Applications and Limitations

A common application is estimating the parameters of a mixture of Gaussians, where each observed data point belongs to an unobserved mixture component. EM can also be used for multiple linear regression with missing data, although it is often applied across domains like, [machine-learning](https://www.wikiprompt.org/wiki/machine-learning), [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence), and other fields with latent structures.

One limitation is that EM can converge to a local maximum rather than the global maximum, and some likelihoods can have singularities. In mixture models, for instance, a solution with non-sensical maxima may occur if a component is assigned zero variance, which is problematic but a known outcome of the iterative procedure.

## Extensions and Practical Notes

Extensions of EM, such as the expectation-conditional maximization (ECM) algorithm or the Monte Carlo EM, address potential convergence issues or computational complexity. In practice, EM is chosen when the complete data likelihood is simpler to optimize than the marginal likelihood, even if the observed data are incomplete. It remains a foundational method for estimating parameters with latent variables, with broad relevance in statistics.

## References

The DLR paper's name and the convergence analysis by Wu in 1983 define the modern formulation. Textbooks by authors like [christopher-bishop](https://www.wikiprompt.org/wiki/christopher-bishop) (Pattern Recognition and Machine Learning) and [chris-bishop](https://www.wikiprompt.org/wiki/chris-bishop) provide detailed treatments, linking EM to broader topics in probabilistic modeling and other learning algorithms.

## External links

- [Wikipedia: Expectation–maximization algorithm](https://en.wikipedia.org/wiki/Expectation–maximization_algorithm)

---
Source: https://www.wikiprompt.org/wiki/expectation-maximization-algorithm
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-14T06:27:37.858273+00:00