Supernode Fusion Model Based on Bimodal Action Recognition

Skeleton action recognition based on graph convolutional neural networks (GCNs) has become a hot research topic in recent years. Existing graph convolutional networks have limitations in interacting with high-order node information due to the constraint of convolution kernels in graph convolutions. Additionally, sparse representation of skeleton data makes it difficult to capture subtle motion information. Multi-modal approaches have been used to complement skeleton data, but communication between different modalities is challenging due to differences in data formats. This paper proposes a skeleton-optical flow supernode-based action recognition model called 2 S-SNGCN. On one hand, it utilizes the supernode features to learn spatial relationships between high-order nodes and aggregate long-term temporal context. On the other hand, it shares the supernode features across different modalities to enhance the information correlation and improve action recognition accuracy. The proposed model achieves a Top-1 accuracy of 94.3% on the NTU RGB+D dataset and 90.5% on the NTU RGB+D120 dataset under the X-sub protocol. This is a 0.5% and 2.9% improvement, respectively, compared to the JOLO-GCN model.