The Fréchet inception distance (FID) is a metric used to assess the quality of images created by a generative model, such as a generative adversarial network (GAN) or a diffusion model. Introduced in 2017, FID has become the standard metric for evaluating synthetic image generators as of 2024. It compares the distribution of generated images with the distribution of a set of real images, known as the ground truth set, by measuring the distance between their feature statistics extracted from a deep neural network. A lower FID score indicates that the generated images are more similar in quality and diversity to the real reference images, with a score of 0 representing a perfect match.
The FID metric was inspired by the earlier inception score (IS), but it addresses a key limitation: IS evaluates only the distribution of generated images without comparing to real data. FID, by contrast, measures the similarity between the generated and real distributions, providing a more comprehensive assessment. It has been widely adopted to benchmark models such as StyleGAN1, StyleGAN2, and various diffusion models, and it remains a cornerstone in the evaluation of generative models.
Overview
The purpose of the FID score is to measure the diversity and fidelity of images created by a generative model relative to a reference dataset. The reference dataset could be ImageNet or COCO-2014, and it should represent the full diversity of images the model aims to generate. Because generative models produce novel images that differ from any training example, quality cannot be assessed by pixel-wise comparison, as done with the L2 norm.
Instead, FID models the two sets of images as if they were drawn from two multidimensional Gaussian distributions, denoted as \(\mathcal{N}(\mu, \Sigma)\) for real images and \(\mathcal{N}(\mu', \Sigma')\) for generated images. The distance between these distributions is calculated as the 2-Wasserstein distance, which accounts for both the mean and covariance differences. This approach captures not only whether individual images look realistic but also whether the generated set covers the same range of variation as the real set.
Rather than comparing images directly in pixel space, FID uses a convolutional neural network, specifically an Inception v3 architecture, to extract high-level features. The deepest layer, a 2048-dimensional activation vector from the last pooling layer, is used to represent each image. These features correspond to real-world objects, such as dog breeds or airplanes, making the comparison more semantically meaningful than raw pixels.
After processing all images through the Inception network, the means and covariances of the activation vectors are computed for both the real and generated sets. The FID distance is then given by the formula:
\[ d_F(\mathcal{N}(\mu, \Sigma), \mathcal{N}(\mu', \Sigma'))^2 = \lVert \mu - \mu' \rVert_2^2 + \operatorname{tr}\left(\Sigma + \Sigma' - 2(\Sigma \Sigma')^{1/2}\right) \]
Higher distances indicate a poorer generative model, while a score of 0 indicates a perfect model that exactly replicates the real distribution.
Formal Definition
For any two probability distributions \(\mu\) and \(\nu\) over \(\mathbb{R}^n\) with finite means and variances, the Fréchet distance is defined as:
\[ d_F(\mu, \nu) := \left( \inf_{\gamma \in \Gamma(\mu, \nu)} \int_{\mathbb{R}^n \times \mathbb{R}^n} \lVert x - y \rVert^2 \, d\gamma(x, y) \right)^{1/2} \]
where \(\Gamma(\mu, \nu)\) is the set of all couplings (joint distributions) with marginals \(\mu\) and \(\nu\). This is the 2-Wasserstein distance, which has a closed-form solution when both distributions are Gaussian. In the context of FID, the distributions are approximated as Gaussians using the empirical means and covariances of the feature vectors, making the computation tractable.
The use of the Fréchet distance, rather than simpler metrics like the Kullback-Leibler divergence, allows FID to capture both the mean shift and the covariance mismatch, providing a more nuanced measure of distributional similarity.
Comparison with Inception Score
The FID metric does not replace the inception score (IS); rather, it complements it. Classifiers that achieve the best (lowest) FID score tend to have greater sample variety, while classifiers achieving the best (highest) IS score tend to have better quality within individual images. This distinction arises because IS evaluates only the generated images, rewarding models that produce confident, diverse class predictions, but it does not penalize models that generate images outside the target distribution. FID, by comparing to a real reference set, penalizes both lack of fidelity and lack of diversity.
In practice, researchers often report both FID and IS to provide a fuller picture of model performance. However, FID has become the preferred metric in many recent studies due to its stronger correlation with human perception of image quality.
Use in Generative Model Evaluation
FID has been used to measure the quality of many recent models, including high-resolution StyleGAN1 and StyleGAN2 networks, and diffusion models. For example, StyleGAN2 achieved state-of-the-art FID scores on datasets like FFHQ and LSUN, demonstrating its ability to generate diverse and realistic images. Diffusion models, such as DDPM and score-based models, have also been evaluated using FID, often achieving competitive or superior scores compared to GANs.
The metric is particularly valuable for comparing models trained on the same dataset, as it provides a standardized way to quantify progress. However, FID is sensitive to the choice of reference dataset and the number of samples used; using too few samples can lead to unstable estimates. Researchers typically use thousands of images to compute reliable FID scores.
Limitations and Criticisms
Despite its widespread use, FID has limitations. It assumes that the feature distributions are Gaussian, which may not hold in practice, leading to potential inaccuracies. Additionally, FID is computed using a fixed Inception network, which may not capture all aspects of image quality, especially for domains outside natural images (e.g., medical or satellite imagery). The metric can also be biased by the number of samples, and it does not provide per-image quality assessments.
Recent works have proposed alternatives, such as using CLIP embeddings instead of Inception features, to better align with semantic similarity. These approaches aim to address some of FID's shortcomings, but FID remains the de facto standard in the field.
Implementation and Computation
Computing FID involves several steps. First, a reference dataset of real images is selected, and all images are passed through a pre-trained Inception v3 network to extract the 2048-dimensional feature vectors. The same is done for the generated images. The mean and covariance of the feature vectors are then calculated for both sets. Finally, the Fréchet distance is computed using the formula above.
In practice, the covariance matrices are often regularized to ensure numerical stability, especially when the number of samples is smaller than the feature dimension. Open-source implementations are available in libraries such as PyTorch and TensorFlow, making FID easy to compute for researchers and practitioners.
Future Directions
As generative models continue to evolve, the evaluation metrics must also adapt. While FID has been instrumental in advancing the field, there is ongoing research into more robust and interpretable metrics. The use of CLIP embeddings, as mentioned, is one such direction. Additionally, metrics that evaluate other aspects, such as fairness or robustness, are being explored. Nonetheless, FID remains a key tool in the Machine learning toolbox for assessing generative models.
For further reading, see related concepts such as Inception Score, Generative adversarial network, Diffusion model, and StyleGAN. The development of FID is also connected to advances in Deep learning and Computer vision.