Stme-net: spatio-temporal motion excitation network for action recognition
摘要
Video action recognition, as one of the fundamental tasks in video understanding, relies crucially on accurate temporal modeling. However, accurately modeling the temporal information of videos remains a challenging task. To address this problem, we design two new modules: the Spatial Motion Extraction (SME) module and the Spatio-temporal Motion Excitation (STME) module. The SME module features two branches for extracting motion and spatial features. The motion branch refines pixel differences between neighboring frames through a channel attention module, enhancing detailed motion features. These features are fused with spatial information to yield fine-grained local spatio-temporal features. The STME module, comprising the multi-motion excitation (MME), temporal excitation (TE), and spatio-temporal excitation (STE) sub-modules, efficiently captures long-range motion, temporal, and global spatio-temporal features. The MME introduces a bi-directional, multi-scale structure for effective long-range motion extraction, while the TE module employs a hierarchical pyramid with residual connectivity for fine-grained long-range temporal extraction. The STE module utilizes 3D convolutional layers for global spatio-temporal feature extraction. The seamless integration of these sub-modules within a standard ResNet network forms the Spatio-temporal Motion Excitation Network. Extensive evaluations on Something V1 and V2 and HMDB51 datasets against state-of-the-art methods demonstrate the effectiveness of our approach in achieving accurate recognition of both simple and complex video actions.