<p>Video action recognition, as one of the fundamental tasks in video understanding, relies crucially on accurate temporal modeling. However, accurately modeling the temporal information of videos remains a challenging task. To address this problem, we design two new modules: the Spatial Motion Extraction (SME) module and the Spatio-temporal Motion Excitation (STME) module. The SME module features two branches for extracting motion and spatial features. The motion branch refines pixel differences between neighboring frames through a channel attention module, enhancing detailed motion features. These features are fused with spatial information to yield fine-grained local spatio-temporal features. The STME module, comprising the multi-motion excitation (MME), temporal excitation (TE), and spatio-temporal excitation (STE) sub-modules, efficiently captures long-range motion, temporal, and global spatio-temporal features. The MME introduces a bi-directional, multi-scale structure for effective long-range motion extraction, while the TE module employs a hierarchical pyramid with residual connectivity for fine-grained long-range temporal extraction. The STE module utilizes 3D convolutional layers for global spatio-temporal feature extraction. The seamless integration of these sub-modules within a standard ResNet network forms the Spatio-temporal Motion Excitation Network. Extensive evaluations on Something V1 and V2 and HMDB51 datasets against state-of-the-art methods demonstrate the effectiveness of our approach in achieving accurate recognition of both simple and complex video actions.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Stme-net: spatio-temporal motion excitation network for action recognition

  • Qian Zhao,
  • Yanxiong Su,
  • Hui Zhang

摘要

Video action recognition, as one of the fundamental tasks in video understanding, relies crucially on accurate temporal modeling. However, accurately modeling the temporal information of videos remains a challenging task. To address this problem, we design two new modules: the Spatial Motion Extraction (SME) module and the Spatio-temporal Motion Excitation (STME) module. The SME module features two branches for extracting motion and spatial features. The motion branch refines pixel differences between neighboring frames through a channel attention module, enhancing detailed motion features. These features are fused with spatial information to yield fine-grained local spatio-temporal features. The STME module, comprising the multi-motion excitation (MME), temporal excitation (TE), and spatio-temporal excitation (STE) sub-modules, efficiently captures long-range motion, temporal, and global spatio-temporal features. The MME introduces a bi-directional, multi-scale structure for effective long-range motion extraction, while the TE module employs a hierarchical pyramid with residual connectivity for fine-grained long-range temporal extraction. The STE module utilizes 3D convolutional layers for global spatio-temporal feature extraction. The seamless integration of these sub-modules within a standard ResNet network forms the Spatio-temporal Motion Excitation Network. Extensive evaluations on Something V1 and V2 and HMDB51 datasets against state-of-the-art methods demonstrate the effectiveness of our approach in achieving accurate recognition of both simple and complex video actions.