A Feature Pyramid Network (FPN) is a neural network architecture designed to extract multi-scale feature maps for object detection and other computer vision tasks. Introduced by Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie in their 2017 paper "Feature Pyramid Networks for Object Detection," FPN addresses the challenge of detecting objects at varying scales by building a pyramid of feature maps with rich semantic information at every level. It has become a foundational component in many modern detectors, such as Faster R-CNN and Mask R-CNN, and is widely used in Machine learning and Deep learning applications.
The core idea of FPN is to leverage the inherent multi-scale, hierarchical feature representations of a convolutional Neural network (typically a backbone like ResNet) and enhance them with a top-down pathway that propagates high-level semantic features to lower resolutions. This is achieved through lateral connections that merge corresponding feature maps from the bottom-up pathway, enabling the network to produce strong features at each scale. As a result, FPN improves detection performance, especially for small objects, without significantly increasing computational cost.
Architecture
FPN consists of two main pathways: a bottom-up pathway and a top-down pathway. The bottom-up pathway is the standard feedforward convolutional network (e.g., ResNet) that computes feature maps at multiple scales, typically with strides of 4, 8, 16, and 32 pixels relative to the input image. These feature maps are denoted as C2, C3, C4, and C5, corresponding to the last residual block outputs at each stage.
The top-down pathway generates higher-resolution feature maps by upsampling spatially coarser but semantically stronger features from higher layers. Each upsampled feature map is then merged with the corresponding bottom-up feature map via a 1x1 convolutional layer (to reduce channel dimensions) followed by element-wise addition. The merged result is passed through a 3x3 convolution to produce the final feature map for that scale, denoted as P2, P3, P4, and P5. These feature maps are used for object detection, with each level responsible for detecting objects of a certain size range.
Applications
FPN has been widely adopted in object detection frameworks. In Faster R-CNN, FPN is used as the feature extractor for the Region Proposal Network (RPN) and the subsequent ROI-based classifier, allowing the model to handle objects of varying sizes effectively. Mask R-CNN extends FPN for instance segmentation, using the multi-scale features to generate segmentation masks. FPN also benefits other tasks such as semantic segmentation and keypoint detection, where multi-scale context is crucial.
Beyond detection, FPN has inspired similar architectures in other domains, such as U-Net for biomedical image segmentation, which also employs a symmetric encoder-decoder with skip connections. However, FPN's specific design of lateral connections and top-down fusion has proven particularly effective for detection tasks.
Variants and Improvements
Several variants of FPN have been proposed to enhance its efficiency and accuracy. For instance, BiFPN (Bidirectional Feature Pyramid Network) used in EfficientDet introduces weighted bidirectional feature fusion, allowing the network to learn the importance of different input features. Another variant, PANet (Path Aggregation Network), adds an additional bottom-up path augmentation to shorten the information path between low-level and high-level features. These improvements have pushed the state of the art in object detection on benchmarks like COCO.
Impact and Legacy
FPN has had a lasting impact on the field of computer vision. Its introduction marked a shift from using single-scale features to multi-scale feature pyramids, which is now standard practice in detection architectures. The paper has been cited thousands of times and has influenced subsequent research in Residual Network (ResNet) design and feature fusion. FPN's principles are also applied in other areas, such as Data Augmentation strategies that simulate scale variations during training.
In summary, Feature Pyramid Network is a key architectural innovation that enables robust multi-scale object detection, and its influence extends across many modern vision systems.