In computer vision, a saliency map is an image that highlights either the region on which people's eyes focus first or the most relevant regions for machine learning models. The goal of a saliency map is to reflect the degree of importance of a pixel to the human visual system or an otherwise opaque ML model. For example, in an image of a landscape, a person might first look at a fort and light clouds, so those regions would be highlighted on the saliency map. Saliency maps are widely used in applications ranging from image compression to explainable artificial intelligence, where they provide visual explanations of model decisions.
The concept of saliency originates from studies of human visual attention. The primary visual cortex (V1) appears to be responsible for the saliency map, according to the V1 Saliency Hypothesis. In machine learning, saliency maps are typically generated for deep neural networks, especially convolutional neural networks and transformers, to indicate which parts of the input influence the output most.
Human Eye Applications
Saliency maps have applications in a variety of different problems. For human eye attention, they are used in:
- Image and video compression: The human eye focuses only on a small region of interest in the frame. Therefore, it is not necessary to compress the entire frame with uniform quality. Using a salience map reduces the final size of the video with the same visual perception.
- Image and video quality assessment: The main task for an image or video quality metric is a high correlation with user opinions. Differences in salient regions are given more importance and thus contribute more to the quality score.
- Image retargeting: This aims at resizing an image by expanding or shrinking the noninformative regions. Retargeting algorithms rely on the availability of saliency maps that accurately estimate all the salient image details.
- Object detection and recognition: Instead of applying a computationally complex algorithm to the whole image, one can apply it to the most salient regions of an image most likely to contain an object.
Explainable AI
Saliency maps are a prominent tool in explainable artificial intelligence, providing visual explanations of the decision-making process of machine learning models, particularly deep neural networks. These maps highlight the regions in input data that are most influential on the model's output, effectively indicating where the model is "looking" when making a prediction. In image classification tasks, for example, saliency maps can identify pixels or regions that contribute most to a specific class decision.
Developed for convolutional neural networks, saliency mapping techniques range from simply taking the gradient of the class score with respect to the input data to more complex algorithms, such as integrated gradients and class activation mapping. In transformer architecture, attention mechanisms led to analogous saliency maps, such as attention maps, attention rollouts, and class-discriminative attention maps.
Saliency as a Segmentation Problem
Saliency estimation may be viewed as an instance of image segmentation. In computer vision, image segmentation is the process of partitioning a digital image into multiple segments (sets of pixels, also known as superpixels). The goal of segmentation is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze. Image segmentation is typically used to locate objects and boundaries (lines, curves, etc.) in images. More precisely, image segmentation is the process of assigning a label to every pixel in an image such that pixels with the same label share certain characteristics.
Classic Algorithms
There are three forms of classic saliency estimation algorithms implemented in OpenCV:
- Static saliency: Relies on image features and statistics to localize the regions of interest of an image.
- Motion saliency: Relies on motion in a video, detected by optical flow. Objects that move are considered salient.
- Objectness: Objectness reflects how likely an image window covers an object. These algorithms generate a set of bounding boxes of where an object may lie in an image.
In addition to classic approaches, neural-network-based methods are also popular. There are examples of neural networks for motion saliency estimation:
- TASED-Net: Consists of two building blocks. First, the encoder network extracts low-resolution spatiotemporal features, and then the following prediction network decodes the spatially encoded features while aggregating all the temporal information.
- STRA-Net: Emphasizes two essential issues. First, spatiotemporal features integrated via appearance and optical flow coupling, and then multi-scale saliency learned via attention mechanism.
- STAViS: Combines spatiotemporal visual and auditory information. This approach employs a single network that learns to localize sound sources and to fuse the two saliencies to obtain a final saliency map.
There is a newer static saliency method called visual distortion sensitivity. It is based on the idea that the true edges, i.e., object contours, are more salient than other complex textured regions. It detects edges in a different way from classic edge detection algorithms. It uses a fairly small threshold for the gradient magnitudes to consider the mere presence of the gradients. It obtains four binary maps for vertical, horizontal, and two diagonal directions. Morphological closing and opening are applied to the binary images to close small gaps. To clear blob-like shapes, it utilizes the distance transform. After all, the connected pixel groups are individual edges (or contours). A threshold of size of connected pixel set is used to determine whether an image block contains a perceivable edge (salient region) or not.
Example Implementation
A simple saliency map can be computed by calculating the distance of each pixel to the rest of pixels in the same frame. The saliency value for a pixel \(I_k\) is given by:
\(\mathrm{SALS}(I_k) = \sum_{i=1}^{N} |I_k - I_i|\)
where \(I_i\) is the value of pixel \(i\), in the range of [0,255]. The expanded form is:
\(\mathrm{SALS}(I_k) = |I_k - I_1| + |I_k - I_2| + ... + |I_k - I_N|\)
where N is the total number of pixels in the current frame. The formula can be restructured by grouping pixels with the same intensity value:
\(\mathrm{SALS}(I_k) = \sum_{n=0}^{255} F_n \times |I_k - I_n|\)
where \(F_n\) is the frequency of intensity value \(I_n\). The frequencies are expressed in the form of a histogram, and the computational time of histogram is \(O(N)\). This approach is efficient and forms the basis for many classic saliency detection methods.
Neural Network Approaches
Modern saliency map generation often relies on deep learning. For convolutional neural networks, gradient-based methods compute the derivative of the class score with respect to the input image, producing a saliency map that highlights pixels with the strongest influence. Integrated gradients and class activation mapping (CAM) are more advanced techniques that provide smoother and more discriminative maps. In transformers, attention mechanisms produce attention maps that can be aggregated across layers, known as attention rollouts, to create saliency maps for visual and textual inputs. These methods are widely used in Machine learning and Deep learning for model interpretability.
Applications in Other Domains
While saliency maps are most common in computer vision, they have been adapted to other domains. In natural language processing, saliency maps can highlight words or tokens that most influence a model's prediction, especially in Transformer (architecture)-based models like Large language models. In audio processing, saliency maps can indicate time-frequency regions relevant for classification. The underlying principle remains the same: identify the parts of the input that matter most for the output.
Challenges and Limitations
Saliency maps are not without criticism. They can be sensitive to small perturbations in the input, leading to unstable explanations. Some methods produce maps that are not class-discriminative, meaning they highlight regions that are generally salient rather than specific to a particular prediction. Additionally, evaluating the quality of saliency maps is difficult, as there is no ground truth for model attention. Researchers continue to develop new techniques to improve reliability and interpretability, often drawing on insights from Neural network research and Artificial intelligence ethics.
Future Directions
As Artificial intelligence models become more complex, the need for transparent explanations grows. Saliency maps are likely to remain a key tool in explainable AI, especially for high-stakes applications like medical imaging and autonomous driving. Advances in Generative AI and multimodal models may lead to new forms of saliency that combine visual, textual, and auditory cues. Research at institutions like MIT CSAIL, Stanford AI Lab, and BAIR (Berkeley AI Research) continues to push the boundaries of interpretability, with saliency maps serving as a foundational technique.