<p>Human behavior recognition is one of the core challenges in computer vision, and its key lies in how to model the complex spatio-temporal dependence relationship in video data. Although significant progress has been made in the existing methods, it is still a thorny problem to capture multi-scale spatio-temporal features and accurately focus on key regions and frames in videos. This paper proposes a behavior recognition algorithm (MS-CANetw) based on multi-scale spatio-temporal feature extraction and a collaborative attention mechanism. By fusing the feature maps of different depths of the trunk network to capture both details and semantic space information, and using multi-branch time series convolution to perceive short-term and long-term motion changes. We designed a collaborative attention mechanism which coordinated by a spatial attention mechanism (attention to “where”) and a temporal attention mechanism (attention to “when”) to guide each other: spatial salience is used to recalibrate temporal weights, while temporal context information is used to enhance spatial feature representation, so as to realize the collaborative optimization of both and accurately focus on the spatio-temporal segments most relevant to behavior. Extensive experiments conducted on three authoritative datasets (NTU RGB + D60, NTU RGB + D120, and Kinetics-400) demonstrate that our algorithm significantly outperforms many mainstream methods. Particularly on the NTU RGB + 60 dataset, it achieved an accuracy rate of 93.1%, which fully validates its superiority. The ablation experiment further proves the effectiveness of multi-scale design and collaborative attention mechanism, respectively, and the necessity of their cooperation.</p>