MCANet: a lightweight action recognition network with multidimensional convolution and attention

Most lightweight networks used in action recognition tasks are based on convolutional neural networks (CNNs), which have spatial inductive biases that enable action recognition with fewer parameters and faster inference speed. While CNNs are highly effective at extracting local spatiotemporal features, they inherently lack the capability for global spatiotemporal modeling, which is the strength of vision transformers (ViTs). However, ViTs are characterized by a large number of parameters and relatively slower inference times, making them less suitable for lightweight applications. To meet the dual requirements of lightweight design and high recognition accuracy, this paper proposes MCANet, a novel action recognition network optimized for deployment on lightweight devices. MCANet integrates the strengths of both CNNs and ViTs, maintaining low model complexity and rapid inference speed while providing local and global spatiotemporal modeling capabilities. The efficacy of the proposed approach is validated through comprehensive evaluations of several benchmark datasets, including Kinetics400, UCF101, and HMDB51. In terms of accuracy, MCANet matches the performance of the best current action recognition model and shows a 3% improvement over other lightweight networks on Kinetics400. Regarding lightweight performance, MCANet improves inference speed by more than 17% compared to the current best lightweight model.