Multi-scale spatiotemporal topology unveiled: enhancing skeleton-based action recognition

In recent years, skeleton-based action recognition has received considerable attention due to the robustness of human skeletons in complex environments. However, many existing methods face challenges in effectively learning global temporal information due to inadequate extraction of spatiotemporal features and the neglect of long-term dependencies. Furthermore, subtle joint movements play a critical role in skeleton-based behavior recognition, as such movements are essential for distinguishing between similar actions. To address the aforementioned challenges, this paper proposes a Multi-Scale Spatiotemporal Topology-Aware Network (MSTC3D), which integrates data from various sampled frames into a dual-channel network and employs lateral connections to merge features from different temporal scales. This facilitates the dynamic learning of global temporal channel variations, enhancing the modeling of long-term temporal dependencies. The proposed Multi-Scale 3D Convolutional Block (M3D) incorporates a pyramid-like structure to expand the receptive field effectively, thereby enabling the accurate capture of multi-layered detailed features of subtle joint movements. Moreover, to further enhance the model’s fine-grained recognition capability for features associated with various joints and regions, a Spatial Topological Focus Module is embedded within the M3D. By comprehensively considering both short-term and long-term temporal dependencies, and leveraging the efficient feature representation provided by multi-scale convolutional blocks, MSTC3D demonstrates superior performance in action recognition tasks. Experiments on the NTU RGB+D and FineGym datasets validate the effectiveness of MSTC3D, showing state-of-the-art performance compared to CNN-based methods and achieving comparable superior performance to leading GCN-based methods.