Normalizing flows are a class of generative models in machine learning that explicitly model a probability distribution by applying a sequence of invertible transformations to a simple base distribution. The method leverages the change-of-variable law of probabilities to convert a simple distribution, such as a Gaussian, into a complex target distribution. This direct modeling of likelihood offers advantages: the negative log-likelihood can be computed and minimized as a loss function, and novel samples can be generated by sampling from the base distribution and applying the flow transformation. In contrast, other generative modeling methods, such as variational autoencoders (VAEs), generative adversarial networks (GANs), and diffusion models, do not explicitly represent the likelihood function.
The term 'normalizing flow' reflects two aspects: 'normalizing' refers to the fact that the transformations produce a normalized probability density (integrating to one), and 'flow' refers to the sequential composition of transformations that 'flow' the base distribution into the target. Flow-based models have been applied in fields such as image generation, density estimation, and anomaly detection, and they form the basis for more advanced architectures like RealNVP and Glow.
Method
Let \( z_0 \) be a (possibly multivariate) random variable with distribution \( p_0(z_0) \). For \( i = 1, \dots, K \), let \( z_i = f_i(z_{i-1}) \) be a sequence of random variables transformed from \( z_0 \). The functions \( f_1, \dots, f_K \) must be invertible, meaning the inverse function \( f_i^{-1} \) exists. The final output \( z_K \) models the target distribution.
The log likelihood of \( z_K \) is given by:
\[ \log p_K(z_K) = \log p_0(z_0) - \sum_{i=1}^K \log \left| \det \frac{d f_i(z_{i-1})}{d z_{i-1}} \right| \]
This formula arises from the change-of-variable rule, which adjusts the density by the absolute determinant of the Jacobian of each transformation. To efficiently compute the log likelihood, the functions \( f_1, \dots, f_K \) should be easily invertible, and the determinants of their Jacobians should be simple to compute. In practice, these functions are modeled using deep neural networks, trained to minimize the negative log-likelihood of data samples from the target distribution. Architectures are designed so that only the forward pass of the network is required for both the inverse and the Jacobian determinant calculations. Examples include NICE, RealNVP, and Glow.
Derivation of log likelihood
Consider \( z_1 \) and \( z_0 \). Note that \( z_0 = f_1^{-1}(z_1) \). By the change-of-variable formula, the distribution of \( z_1 \) is:
\[ p_1(z_1) = p_0(z_0) \left| \det \frac{d f_1^{-1}(z_1)}{d z_1} \right| \]
Using the inverse function theorem, the determinant of the Jacobian of the inverse is the reciprocal of the determinant of the Jacobian of the forward transformation, leading to the log-likelihood expression above. This derivation extends to multiple transformations by induction.
Architectural Innovations
Early flow-based models focused on designing transformations with tractable Jacobians. NICE (Non-linear Independent Components Estimation), introduced by Dinh et al. in 2014, used additive coupling layers that partition the input and apply simple affine transformations, making the Jacobian triangular with a determinant of one. RealNVP (Real-valued Non-Volume Preserving) extended this to affine coupling layers, allowing scale and shift operations, which improved expressiveness while maintaining invertibility. Glow, introduced by Kingma and Dhariwal in 2018, added invertible 1x1 convolutions and actnorm layers, enabling efficient training on high-resolution images.
These architectures are often composed of multiple coupling layers, each with a permutation or convolution to mix dimensions. The invertibility is guaranteed by construction, and the Jacobian determinant is computed as the product of the diagonal elements of the triangular Jacobian, which is efficient.
Applications and Comparisons
Normalizing flows are used for density estimation, where they learn the probability distribution of a dataset, and for generative sampling, where they produce new data points. They have been applied to image generation, audio synthesis, and molecular conformation generation. In contrast to GANs, which are known for sharp samples but lack likelihood estimation, flows provide exact log-likelihoods, which can be useful for model comparison and outlier detection. Compared to diffusion models, flows are often faster to sample from because they require only a single forward pass, whereas diffusion models require iterative denoising.
However, flows can be computationally expensive to train due to the need for invertible architectures and Jacobian computations. They also may require more parameters to capture complex distributions compared to other methods.
Training and Optimization
Training a normalizing flow involves minimizing the negative log-likelihood of the training data. The loss function is:
\[ \mathcal{L} = -\frac{1}{N} \sum_{n=1}^N \log p_K(x_n) \]
where \( x_n \) are data samples. This is typically done using stochastic gradient descent. The invertibility of the transformations ensures that the log-likelihood is well-defined, and the Jacobian determinant is computed as part of the forward pass. Regularization techniques, such as weight decay and dropout, can be applied to the network parameters.
Related Concepts
Normalizing flows are part of the broader field of artificial intelligence and neural networks. They are often compared with other generative models such as generative adversarial networks and variational autoencoders. Research in this area has been advanced by institutions like Stanford AI Lab and Berkeley AI Research, and by researchers such as Daphne Koller and Anima Anandkumar. The theoretical foundations connect to machine learning and deep learning.