<p>In the task of human behavior recognition, modeling human skeletons as spatio-temporal graphs using Graph Convolutional Networks (GCNs) has achieved outstanding performance. Existing GCN-based methods typically focus on the two-dimensional or three-dimensional features of the skeleton. However, the same action may represent different behaviors in different environments, making it suboptimal to consider only skeleton features for behavior recognition tasks with diverse scenarios. To simultaneously account for both skeleton features and environmental factors that influence human behaviors, this study proposed a novel two-stream Graph Convolutional Network, 2S-EGCN, which incorporates environmental factors for human behavior recognition. In this network, we designed an innovative environmental factor sampling strategy that samples fixed-scale environmental factors from variable-scale feature maps. To better integrate environmental factors with skeleton features, we further developed a Skeleton-Environment Interaction Module. This module uses a specific feature fusion method to combine environmental factors with skeleton features, allowing for the modeling of both pure skeleton information and skeleton information fused with environmental factors, thus improving behavior recognition accuracy. Extensive experiments conducted on the large Kinetics dataset demonstrate that our model outperforms the state-of-the-art, improving top-1 accuracy by 1.71–55.61% and achieving top-5 accuracy of 93.41%.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Environmental factors-aware two-stream GCN for skeleton-based behavior recognition

  • Zhuoran Li,
  • Lianshan Yan,
  • Hua Li,
  • Yu Wang

摘要

In the task of human behavior recognition, modeling human skeletons as spatio-temporal graphs using Graph Convolutional Networks (GCNs) has achieved outstanding performance. Existing GCN-based methods typically focus on the two-dimensional or three-dimensional features of the skeleton. However, the same action may represent different behaviors in different environments, making it suboptimal to consider only skeleton features for behavior recognition tasks with diverse scenarios. To simultaneously account for both skeleton features and environmental factors that influence human behaviors, this study proposed a novel two-stream Graph Convolutional Network, 2S-EGCN, which incorporates environmental factors for human behavior recognition. In this network, we designed an innovative environmental factor sampling strategy that samples fixed-scale environmental factors from variable-scale feature maps. To better integrate environmental factors with skeleton features, we further developed a Skeleton-Environment Interaction Module. This module uses a specific feature fusion method to combine environmental factors with skeleton features, allowing for the modeling of both pure skeleton information and skeleton information fused with environmental factors, thus improving behavior recognition accuracy. Extensive experiments conducted on the large Kinetics dataset demonstrate that our model outperforms the state-of-the-art, improving top-1 accuracy by 1.71–55.61% and achieving top-5 accuracy of 93.41%.