Probabilistic graphical models (PGMs), also known as graphical models or structured probabilistic models, are probabilistic models in which a graph expresses the conditional dependence structure between random variables. They are widely used in probability theory, statistics - particularly Bayesian statistics - and machine learning. By encoding dependencies as edges and independences as graph separations, PGMs provide a compact representation of complex joint distributions, making inference and learning tractable in many applications.
The core idea is to represent a distribution over a multi-dimensional space using a graph, where nodes correspond to random variables and edges indicate probabilistic dependencies. This graph-based factorization captures a set of conditional independences that hold in the distribution, allowing efficient computation of marginal and conditional probabilities. Two main branches of graphical representations exist: Bayesian networks (directed) and Markov random fields (undirected). Both share the properties of factorization and independences but differ in the types of independences they can encode and the resulting factorization of the distribution.
Types of Probabilistic Graphical Models
Bayesian Networks
A Bayesian network, also called a directed graphical model or belief network, uses a directed acyclic graph (DAG) to represent the joint probability distribution. In such a model, the joint probability of random variables \(X_1, \ldots, X_n\) factorizes as a product of conditional distributions:
\[ P[X_1, \ldots, X_n] = \prod_{i=1}^n P[X_i \mid \text{pa}(X_i)] \]
where \(\text{pa}(X_i)\) denotes the set of parents of node \(X_i\) (nodes with edges directed towards \(X_i\)). This factorization implies that any node is conditionally independent of its non-descendants given its parents. The concept of d-separation provides a graph-theoretic criterion for determining conditional independences between sets of variables. In Bayesian networks, local independences (based on parent-child relationships) are equivalent to global independences (based on d-separation).
Classic machine learning models such as hidden Markov models, neural networks, and variable-order Markov models can be considered special cases of Bayesian networks. One of the simplest Bayesian networks is the Naive Bayes classifier, which assumes conditional independence of features given the class label.
Undirected Graphical Models
Undirected graphical models, also known as Markov random fields (MRFs), represent dependencies using undirected edges. In an undirected graph, the presence of an edge implies a dependence between the corresponding random variables, but the absence of an edge implies conditional independence given all other variables. For example, in a graph where nodes B, C, and D are all connected only to node A, the variables B, C, and D are conditionally independent given A. The joint probability distribution can then be factorized as a product of non-negative functions over cliques (fully connected subsets) of the graph. For instance, if the graph has edges AB, AC, and AD, the joint distribution can be written as:
\[ P[A,B,C,D] = f_{AB}[A,B] \cdot f_{AC}[A,C] \cdot f_{AD}[A,D] \]
for some non-negative functions \(f_{AB}, f_{AC}, f_{AD}\). This factorization reflects the local structure of dependencies.
Cyclic Directed Graphical Models
While Bayesian networks typically require acyclic graphs, some models allow cycles. In a cyclic directed graphical model, each variable may depend on its parents in a manner that can lead to feedback loops. For example, a graph with a cycle might suggest a joint density that factors as:
\[ P[A,B,C,D] = P[A] \cdot P[B] \cdot P[C,D \mid A,B] \]
but other interpretations are possible. Cyclic models are less common due to complexities in inference and interpretation.
Other Types
Several other types of probabilistic graphical models extend the basic frameworks:
- Dependency networks: These allow cycles and represent conditional distributions for each variable given all others.
- Tree-augmented classifiers (TAN): These are Bayesian networks that augment a naive Bayes structure with edges among features to capture dependencies.
- Targeted Bayesian network learning (TBNL): This focuses on learning a network that is optimized for a specific target variable.
- Factor graphs: These are undirected bipartite graphs connecting variables and factors, where each factor represents a function over the variables it connects. Factor graphs are useful for implementing belief propagation.
- Clique trees or junction trees: These are trees of cliques used in the junction tree algorithm for exact inference.
- Chain graphs: These graphs may have both directed and undirected edges but no directed cycles, allowing a mix of dependency types.
Applications and Significance
Probabilistic graphical models are fundamental in Machine learning, where they provide a principled framework for reasoning under uncertainty. They are used in Artificial intelligence for tasks such as Generative AI, where models like Large language models and Transformer (architecture)s, though often based on Neural networks, can be interpreted through probabilistic lenses. PGMs are also applied in fields like computer vision, natural language processing, bioinformatics, and robotics. Their ability to combine domain knowledge with data-driven learning makes them powerful tools for both inference and decision-making.
Related Concepts
Probabilistic graphical models are closely related to Deep learning and Neural networks, as many deep architectures can be seen as special cases of graphical models. For instance, hidden Markov models are a classic example. Researchers such as Michael I. Jordan and Daphne Koller have significantly advanced the theory and application of PGMs. Modern developments in Generative AI and Large language models often draw on probabilistic principles, even if they are not explicitly framed as graphical models.