Serial Spatial and Temporal Transformer for Point Cloud Sequences Recognition

Point cloud sequences are unordered and irregular, which means extracting spatial and temporal features from them is challenging. This paper presents a novel network named Serial Spatial and Temporal Transformer (SerialSTTR) for point cloud sequences recognition. Specifically, point-based self-attention is used to gather global information on each point at the spatial level, and frame-based self-attention is used to reconstruct the sequences with motion features at the temporal level. In addition, an orderly local module is proposed to supplement the local feature learning ability that spatial transformer lacks. And relative position encoding is adopted to complete the order information for temporal transformer. Extensive experiments demonstrate that the SerialSTTR achieves the state-of-the-art performance on 3D human action recognition with the challenging dataset MSR-Action3D. And to show its generalizability, experiments on gesture recognition with SHREC’17 dataset are performed, which also present competitive results.