Cross Attention Guided Multimodal Network for Video Action Recognition
摘要
Multimodal vision-language contrastive models play a crucial role in video action recognition, thanks to their dual-encoder architecture, which effectively processes both visual and language information. However, these models are often limited to processing a single modality, which restricts the development of multimodal frameworks. Although visual-language guidance frameworks have been proposed, most fail to implement deep cross-modal feature interaction within the backbone network, making it difficult to effectively align information from different modalities in the latent feature space. To address this limitation, we propose a novel Multimodal Backbone Network Guidance Framework (MBGF) that introduces interaction between visual and language modalities within the backbone network, which not only enhances feature representation but also significantly improves the depth of interaction between visual and language modalities. MBGF modifies the traditional dual-encoder structure by connecting the parallel encoding layers of the video and language encoders, forming a unified encoding framework, which is further divided into guided and separation layers. Furthermore, the GE block is introduced in the guided layer to facilitate bidirectional guidance between the video and language modalities through the cross-attention mechanism, effectively enhancing the representation ability of visual and textual features. Through extensive experiments on multiple benchmark datasets, including Kinetics-400, mini Kinetics-200, HMDB51, and UCF101, in both fully supervised and few-shot settings, we demonstrate the effectiveness and practicality of the MBGF framework.