SDE-Net: Skeleton Action Recognition Based on Spatio-Temporal Dependence Enhanced Networks

Graph Convolutional Networks (GCNs) have succeeded remarkably in skeleton-based action recognition tasks. However, the existing GCN-based methods, where the interframe edges of the graph connect only the same joints and ignore the correlations between different joints, cannot effectively capture the spatiotemporal dependencies between joints. So this paper, we design a spatiotemporal dependence enhancement network (named SDE-Net), which utilizes Convolutional Neural Network (CNN) to make up for the deficiency of GCN in spatiotemporal dependence modeling. SDE-Net makes good use of the structural information between joints and effectively enhances the spatiotemporal dependence of different joints between neighboring frames in the region. SDE-Net consists of three parts: (1) Action Feature Extraction Module (FEM), which obtains local and global features through graph convolutional layers with different temporal kernel sizes, respectively, and sets up short connections to effectively retain local information for obtaining accurate action descriptions; (2) Optimization Module (OPM), which preserves the joint topology information constructed by Graph Convolutional Networks by predicting the adjacency matrix, avoids the problem that crucial joint topology information is prone to be lost when it is transferred between different networks; (3) Convolutional Neural Network, this paper combines the Convolutional Neural Network to model different joints between adjacent frames to achieve the enhancement of spatiotemporal dependence between joints. We validated the performance of the model on two mainstream datasets, NTU RGB+D, and NTU-120 RGB+D, compared to other models, our model obtains higher accuracy.