Environmental factors-aware two-stream GCN for skeleton-based behavior recognition
摘要
In the task of human behavior recognition, modeling human skeletons as spatio-temporal graphs using Graph Convolutional Networks (GCNs) has achieved outstanding performance. Existing GCN-based methods typically focus on the two-dimensional or three-dimensional features of the skeleton. However, the same action may represent different behaviors in different environments, making it suboptimal to consider only skeleton features for behavior recognition tasks with diverse scenarios. To simultaneously account for both skeleton features and environmental factors that influence human behaviors, this study proposed a novel two-stream Graph Convolutional Network, 2S-EGCN, which incorporates environmental factors for human behavior recognition. In this network, we designed an innovative environmental factor sampling strategy that samples fixed-scale environmental factors from variable-scale feature maps. To better integrate environmental factors with skeleton features, we further developed a Skeleton-Environment Interaction Module. This module uses a specific feature fusion method to combine environmental factors with skeleton features, allowing for the modeling of both pure skeleton information and skeleton information fused with environmental factors, thus improving behavior recognition accuracy. Extensive experiments conducted on the large Kinetics dataset demonstrate that our model outperforms the state-of-the-art, improving top-1 accuracy by 1.71–55.61% and achieving top-5 accuracy of 93.41%.