A voting-based novel spatio-temporal fusion framework for video saliency using transfer learning mechanism

In computer vision and image processing, salient object detection in an image and video is one of the complex research problems. The most attention-grabbing object of the image or video is generally considered the most prominent object. Due to the motion effect and change in the object’s shape and structure, saliency detection in the video is much more complex than detecting a salient object in an image. In the last two decades, the researcher(s) community has proposed several methods to mimic the human visual capability to find the attention-grabbing object. The techniques presented in the literature are based on statistics or deep learning approaches. Deep learning-based methods have recently gained more attention due to robust detection results. However, it is challenging for one model to function effectively in every situation, so the overall detection accuracy gets degraded. Consequently, an ensemble technique rather than a single model may be preferable to segment salient objects accurately. In this research article, the author(s) have presented a voting-based spatio-temporal fusion framework for saliency detection in video. In the proposed framework, the author(s) used three static networks to estimate spatial saliency using the transfer learning concept and one dynamic network based on 3D convolution to learn the temporal effect. To construct the final saliency output, each spatial saliency map is fused with a temporal saliency map, and finally, a refined saliency map having spatial and temporal information is generated after pixel-wise voting. The proposed framework’s findings on four publicly available and widely used video saliency datasets are competitive in terms of S-Measure, F-Measure and mean absolute error as compared to the state-the-art methods.