The Inception Score (IS) is an algorithm used to assess the quality of images created by a generative image model such as a generative adversarial network (GAN). It was introduced in 2016 by Tim Salimans and colleagues in a paper on improved techniques for training GANs. The score is calculated based on the output of a separate, pretrained Inception v3 image classification model applied to a sample of (typically around 30,000) images generated by the generative model. The Inception Score is maximized when the following conditions are true: the entropy of the distribution of labels predicted by the Inceptionv3 model for the generated images is minimized (the classification model confidently predicts a single label for each image, corresponding to images being 'sharp' or 'distinct'), and the predictions of the classification model are evenly distributed across all possible labels (corresponding to the output being 'diverse'). It has been somewhat superseded by the related Fréchet inception distance (FID). While the Inception Score only evaluates the distribution of generated images, the FID compares the distribution of generated images with the distribution of a set of real images ('ground truth').
The Inception Score is a widely used metric in the field of Generative AI, particularly for evaluating the performance of GANs and other generative models. It provides a single scalar value that aims to capture both the quality and diversity of generated images, making it convenient for comparing different models or training runs. However, it has known limitations, including sensitivity to the choice of the pretrained classifier and a lack of comparison to real images, which the FID addresses.
Definition
Let there be two spaces, the space of images \(\Omega_X\) and the space of labels \(\Omega_Y\). The space of labels is finite. Let \(p_{gen}\) be a probability distribution over \(\Omega_X\) that we wish to judge. Let a discriminator be a function of type \(p_{dis}:\Omega_X \to M(\Omega_Y)\), where \(M(\Omega_Y)\) is the set of all probability distributions on \(\Omega_Y\). For any image \(x\) and any label \(y\), let \(p_{dis}(y|x)\) be the probability that image \(x\) has label \(y\), according to the discriminator. It is usually implemented as an Inception-v3 network trained on ImageNet.
The Inception Score of \(p_{gen}\) relative to \(p_{dis}\) is defined as:
\[ IS(p_{gen},p_{dis}) := \exp\left(\mathbb{E}_{x\sim p_{gen}}\left[D_{KL}\left(p_{dis}(\cdot|x)\|\int p_{dis}(\cdot|x)p_{gen}(x)dx\right)\right]\right) \]
Equivalent rewrites include:
\[ \ln IS(p_{gen},p_{dis}) := \mathbb{E}_{x\sim p_{gen}}\left[D_{KL}\left(p_{dis}(\cdot|x)\|\mathbb{E}_{x\sim p_{gen}}[p_{dis}(\cdot|x)]\right)\right] \]
In practice, the expectation over \(p_{gen}\) is approximated by drawing a finite sample of generated images, and the marginal distribution of labels is estimated from that sample.
Intuition and Interpretation
The Inception Score is designed to reward two properties of generated images: clarity and variety. Clarity is measured by the confidence of the classifier: if the classifier assigns a high probability to a single label for each image, the conditional label distribution \(p_{dis}(\cdot|x)\) has low entropy, which increases the KL divergence. Variety is measured by the uniformity of the marginal label distribution: if the generated images span many classes, the marginal distribution is close to uniform, which also increases the KL divergence. The exponential function is applied to make the score easier to interpret, with a higher value indicating better quality.
The score is named after the Inception v3 model, a deep convolutional neural network developed by Google for image classification. The model is used as a fixed feature extractor, and its predictions serve as a proxy for human perception of image content.
Usage in Generative Model Evaluation
The Inception Score became a standard metric in the evaluation of GANs and other generative models, particularly in the period from 2016 to 2018. It was used in many influential papers, including those on progressive GANs, StyleGAN, and other architectures. Researchers would report IS values on benchmark datasets such as CIFAR-10 and ImageNet to demonstrate improvements over prior work.
However, the score has been criticized for several reasons. First, it does not compare generated images to real images, so a model that produces a diverse set of unrealistic images could still achieve a high score. Second, the score is sensitive to the specific pretrained classifier used; different versions of Inception v3 or different training procedures can yield different scores. Third, the score can be gamed by models that produce images with certain artifacts that the classifier mistakes for real objects.
Relationship to Fréchet Inception Distance
The Fréchet inception distance (FID), introduced in 2017 by Martin Heusel and colleagues, was developed to address some of the limitations of the Inception Score. FID computes the Fréchet distance between the feature distributions of generated and real images, using the penultimate layer of the Inception v3 network as a feature extractor. Unlike IS, FID compares the generated distribution to a reference distribution of real images, providing a more comprehensive measure of quality and diversity.
In practice, FID has largely replaced IS as the preferred metric in many research settings, as it correlates better with human judgment and is less prone to gaming. Nevertheless, IS remains in use, especially in contexts where a reference dataset is not available or when a quick, simple metric is desired.
Limitations and Criticisms
Several studies have highlighted the limitations of the Inception Score. For instance, the score does not penalize mode collapse, a common failure mode of GANs where the model generates only a few distinct images. If the generated images are sharp but not diverse, the marginal label distribution may still be somewhat uniform if the collapsed modes happen to span many classes, but in practice mode collapse often reduces diversity and thus lowers the score. However, the score cannot distinguish between a model that generates a uniform distribution of all classes and one that generates a single image per class with perfect sharpness.
Another criticism is that the Inception Score is not invariant to the choice of the classifier. Different classifiers trained on different datasets may produce different scores for the same set of generated images. This makes it difficult to compare scores across different papers unless the exact same classifier is used.
Implementation Details
To compute the Inception Score, one typically uses a pretrained Inception v3 model, available in libraries such as TensorFlow or PyTorch. The model is applied to a sample of generated images, and the softmax outputs are collected. The conditional label distributions are then used to compute the KL divergence with the marginal distribution, and the exponential of the mean is taken. The sample size is often set to 30,000 images, as recommended in the original paper, to reduce variance.
In practice, the score is computed in batches to avoid memory issues, and the marginal distribution is estimated from the entire sample. The computation is relatively fast, making it suitable for monitoring training progress.
Applications and Extensions
The Inception Score has been applied beyond GANs to other generative models, such as variational autoencoders and diffusion models, although its use has declined with the rise of FID. It has also been extended in various ways, such as the modified Inception Score (mIS) used in some works, which adjusts for the number of classes or uses a different classifier.
In the broader context of Machine learning and Deep learning, the Inception Score represents an early attempt to quantify the quality of generative models, which remains an open challenge. Other metrics, such as the precision and recall for generative models, have been proposed to provide more granular information.
Historical Context
The Inception Score was introduced in the paper 'Improved Techniques for Training GANs' by Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen, published in 2016. The paper also introduced other techniques such as feature matching and minibatch discrimination. The score quickly gained popularity due to its simplicity and the lack of alternative metrics at the time.
Since then, the field of generative modeling has evolved significantly, with the development of more sophisticated models and evaluation methods. The Inception Score remains a historical milestone and is still referenced in many textbooks and tutorials on generative models.