Scholarlink

<p>Skeleton-based human action recognition is a significant research direction in the field of computer vision, aiming to understand and identify human actions in videos through the analysis of skeleton sequences. Although graph convolutional networks (GCNs) have advanced this area by modeling the topological structure of human joints, current methods still suffer from insufficient spatiotemporal modeling and lack of interaction modeling with manipulated objects. To address these challenges, we proposes a multimodal human action recognition algorithm that integrates semantic similarity with spatiotemporal feature fusion, leveraging both the spatiotemporal characteristics of skeleton sequences and semantic correlations between humans and objects. Specifically, we propose a novel algorithm that combines a self-attention-enhanced graph convolutional module (SAE-GCN), a hierarchical depthwise separable temporal convolution module, and a multi-stream dynamic fusion strategy for robust spatiotemporal feature modeling of skeleton sequences, thereby generating initial action predictions. To capture contextual object information, a spatial overlap-based algorithm in conjunction with an advanced object detection model is introduced to identify objects in the vicinity of the human body. Subsequently, a pre-trained language model is utilized to generate semantic embeddings of action and object predictions, and the semantic similarity between modalities is computed to estimate the relevance between actions and objects. We further propose a new scoring mechanism that integrates semantic similarity, action probabilities, and spatial overlap scores to refine the final prediction by exploiting human-object interactions. The experimental results demonstrate that the proposed multi-modal human action recognition algorithm achieves Top-1 accuracies of 94.8% and 98.0% on the NTU RGB + D dataset under the X-Sub and X-View evaluation protocols, respectively; on the NTU RGB+D120 dataset, the Top-1 accuracies reach 92.0% (X-Sub) and 93.2% (X-Set). The average Top-1 accuracies from five-fold cross-validation are 95.30% and 92.18% on the NTU RGB + D and NTU RGB+D120 datasets, respectively. These results fully confirm that the proposed multi-modal human action recognition algorithm provides a reliable technical solution for practical applications.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A multimodal human action recognition algorithm integrating semantic similarity and spatiotemporal feature fusion

  • 期刊论文

  • 发表日期：

    2026年6月23日

查看全文
  • Cheng Liu,
  • Xuanpeng Zhao,
  • Kaile Ni,
  • Xiaoxuan Chen,
  • Lijun Zhao

摘要

Skeleton-based human action recognition is a significant research direction in the field of computer vision, aiming to understand and identify human actions in videos through the analysis of skeleton sequences. Although graph convolutional networks (GCNs) have advanced this area by modeling the topological structure of human joints, current methods still suffer from insufficient spatiotemporal modeling and lack of interaction modeling with manipulated objects. To address these challenges, we proposes a multimodal human action recognition algorithm that integrates semantic similarity with spatiotemporal feature fusion, leveraging both the spatiotemporal characteristics of skeleton sequences and semantic correlations between humans and objects. Specifically, we propose a novel algorithm that combines a self-attention-enhanced graph convolutional module (SAE-GCN), a hierarchical depthwise separable temporal convolution module, and a multi-stream dynamic fusion strategy for robust spatiotemporal feature modeling of skeleton sequences, thereby generating initial action predictions. To capture contextual object information, a spatial overlap-based algorithm in conjunction with an advanced object detection model is introduced to identify objects in the vicinity of the human body. Subsequently, a pre-trained language model is utilized to generate semantic embeddings of action and object predictions, and the semantic similarity between modalities is computed to estimate the relevance between actions and objects. We further propose a new scoring mechanism that integrates semantic similarity, action probabilities, and spatial overlap scores to refine the final prediction by exploiting human-object interactions. The experimental results demonstrate that the proposed multi-modal human action recognition algorithm achieves Top-1 accuracies of 94.8% and 98.0% on the NTU RGB + D dataset under the X-Sub and X-View evaluation protocols, respectively; on the NTU RGB+D120 dataset, the Top-1 accuracies reach 92.0% (X-Sub) and 93.2% (X-Set). The average Top-1 accuracies from five-fold cross-validation are 95.30% and 92.18% on the NTU RGB + D and NTU RGB+D120 datasets, respectively. These results fully confirm that the proposed multi-modal human action recognition algorithm provides a reliable technical solution for practical applications.