Transformer-based multi-level attention integration network for video saliency prediction

Most existing models for video saliency prediction heavily rely on 3D convolutional operations to extract spatio-temporal features. However, it is worth noting that 3D convolution produces a local receptive field, which may struggle to capture long-range spatio-temporal dependencies effectively. To compensate for such shortage, this paper introduces a novel approach called the Transformer-based Multi-level Attention Integration Network (TMAI-Net) for video saliency prediction. TMAI-Net is designed as a two-stream encoder-decoder model, carefully integrating multi-level features of semantic information. Our model incorporates a Multi-level Interactive Attention(MLIA) module and a Transformer, both implemented based on self-attention mechanism, which are placed at different levels of the model to capture long-range spatio-temporal feature dependencies. Additionally, our model operates on input video frames and attentional patches, allowing the Transformer module to capture structural similarities between related objects in global features and attention features. This, in turn, enables the model to allocate increased attention to salient areas. The efficacy of our proposed approach is validated through extensive experiments conducted on three widely recognized benchmark datasets.