Maximum likelihood estimation (MLE) is a fundamental method in statistics for estimating the unknown parameters of a probability distribution. The core idea is to select the parameter values that make the observed data most probable under the assumed statistical model. This is achieved by constructing a likelihood function, which represents the probability of observing the given data as a function of the parameters, and then finding the parameter values that maximize this function. MLE is widely applied across fields such as Machine learning, econometrics, and Artificial intelligence due to its desirable properties, including consistency and efficiency under regular conditions.
The concept was first developed by Sir Ronald Fisher in the early 20th century, building on earlier work by Carl Friedrich Gauss and others. Fisher formally introduced the method in a 1912 paper and expanded on it in subsequent publications, establishing its theoretical foundations. The method gained prominence in the 1920s and 1930s as a cornerstone of statistical inference, and it remains a standard tool in both classical and modern data analysis.
Mathematical Formulation
Given a set of independent and identically distributed observations \(x_1, x_2, \ldots, x_n\) drawn from a probability distribution with probability density function (or mass function) \(f(x; \theta)\), where \(\theta\) is a vector of unknown parameters, the likelihood function is defined as:
\[ L(\theta) = \prod_{i=1}^n f(x_i; \theta) \]
In practice, it is often more convenient to work with the natural logarithm of the likelihood, called the log-likelihood, because it converts the product into a sum and simplifies differentiation:
\[ \ell(\theta) = \log L(\theta) = \sum_{i=1}^n \log f(x_i; \theta) \]
The maximum likelihood estimate \(\hat{\theta}\) is the value of \(\theta\) that maximizes \(\ell(\theta)\), typically found by setting the derivative of the log-likelihood with respect to \(\theta\) to zero and solving the resulting equations. For many common distributions, closed-form solutions exist, while others require numerical optimization methods such as gradient ascent or the Newton-Raphson algorithm.
Properties and Justification
MLE estimators possess several desirable asymptotic properties when certain regularity conditions are met. They are consistent, meaning that as the sample size increases, the estimates converge in probability to the true parameter values. They are also asymptotically normal, with the distribution of the estimator approaching a normal distribution centered at the true parameter, and asymptotically efficient, achieving the lowest possible variance as described by the Cramér-Rao lower bound. These properties make MLE a preferred choice in many statistical applications.
However, MLE can be biased in finite samples, and its performance depends on the correctness of the assumed model. If the model is misspecified, the estimates may be inconsistent. Additionally, for complex models with many parameters, the likelihood surface can have multiple local maxima, requiring careful optimization strategies.
Applications in Machine Learning
In Machine learning, MLE serves as the foundation for many supervised and unsupervised learning algorithms. For instance, logistic regression estimates its coefficients by maximizing the likelihood of the observed binary outcomes. Similarly, Neural network training often uses maximum likelihood principles, where the loss function corresponds to the negative log-likelihood of the data under the model's output distribution. For classification tasks with softmax outputs, this is equivalent to minimizing cross-entropy loss.
In Deep learning, MLE underpins the training of generative models, including Generative AI systems. Variational autoencoders and normalizing flows explicitly maximize likelihood or its lower bound. Large language models, such as those developed by OpenAI and Anthropic, are trained using maximum likelihood estimation on next-token prediction tasks, where the objective is to maximize the probability of the observed sequence of tokens.
Extensions and Variations
Several extensions of MLE address its limitations. Maximum a posteriori (MAP) estimation incorporates prior information by maximizing the posterior distribution, which is proportional to the likelihood times the prior. Penalized likelihood methods, such as ridge or lasso regression, add regularization terms to the log-likelihood to prevent overfitting. Expectation-maximization (EM) algorithm handles models with latent variables, iteratively computing expected values and maximizing the likelihood in a two-step process.
Bayesian approaches, while distinct from MLE, often use the likelihood function as a component. In contrast to MLE, which provides point estimates, Bayesian inference yields full posterior distributions over parameters. Despite this difference, MLE can be seen as a special case of Bayesian estimation with a uniform prior.
Historical Development
The method's origins trace back to the 18th century, with Daniel Bernoulli and later Gauss using similar ideas for specific problems. However, it was Ronald Fisher who formalized the general framework in his 1912 paper "On an Absolute Criterion for Fitting Frequency Curves." Fisher's subsequent work in the 1920s established the likelihood principle and the properties of maximum likelihood estimators. The method gained widespread adoption after the publication of his influential 1922 paper "On the Mathematical Foundations of Theoretical Statistics." Since then, MLE has become a standard topic in statistics curricula and a practical tool in scientific research.
Computational Considerations
For many real-world applications, the likelihood function is too complex for analytical maximization. Numerical optimization techniques, such as gradient-based methods, are employed. In Machine learning frameworks, automatic differentiation allows efficient computation of gradients of the log-likelihood with respect to parameters, enabling scalable training of models with millions of parameters. Specialized hardware, including GPU (in AI)s and AWS Trainium chips, accelerates these computations, making MLE-based training feasible for large-scale models.
Despite its age, MLE remains a vibrant area of research, with ongoing work on robust estimation, high-dimensional settings, and connections to information theory. Its simplicity and theoretical guarantees ensure its continued relevance in both statistics and modern Artificial intelligence.