<p>Most existing deep learning-based sign language recognition methods primarily rely on RGB videos as input. However, RGB videos often contain considerable visual redundancy, which hinders the model’s ability to focus on critical information and capture the spatiotemporal dependencies essential for sign language recognition. To address these challenges, we propose a Multimodal and Bidirectional Visual-Pose Alternating Collaborative Attention method, termed MB-SLR, to enhance sign language modeling through effective multimodal information fusion. First, we introduce a hybrid input framework that combines visual and pose modalities to improve spatiotemporal feature representation. Specifically, the visual modality consists of RGB videos and keypoint sequences, while the pose modality is encoded using four types of embeddings, further enriched by pose semantic embeddings. Second, we propose a Bidirectional Visual-Pose Alternating Collaborative Attention (BiVPA Co-attention) module to facilitate cross-modal information fusion, allowing different modalities to complement and enhance one another. Moreover, the BiVPA Co-attention module jointly integrates modality features across both temporal and spatial dimensions, effectively alleviating information mismatch and dependency gaps among modalities. Finally, we design a head network incorporating BERT to resolve semantic ambiguities and perform sign language recognition. In addition, the head network combines the complementary features generated by the BiVPA module with symbolic gloss features, further enhancing the effectiveness of multimodal feature fusion. Experimental results on three benchmark datasets, namely NMFs-CSL, MSASL, and WLASL, demonstrate that the proposed MB-SLR method achieves substantial performance improvements.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal Sign Language Recognition with Bidirectional Visual-Pose Alternating Attention and Semantic Embedding

  • Shanshan Wan,
  • Yuhan Zhu,
  • Lan Yang,
  • Houchen Lv

摘要

Most existing deep learning-based sign language recognition methods primarily rely on RGB videos as input. However, RGB videos often contain considerable visual redundancy, which hinders the model’s ability to focus on critical information and capture the spatiotemporal dependencies essential for sign language recognition. To address these challenges, we propose a Multimodal and Bidirectional Visual-Pose Alternating Collaborative Attention method, termed MB-SLR, to enhance sign language modeling through effective multimodal information fusion. First, we introduce a hybrid input framework that combines visual and pose modalities to improve spatiotemporal feature representation. Specifically, the visual modality consists of RGB videos and keypoint sequences, while the pose modality is encoded using four types of embeddings, further enriched by pose semantic embeddings. Second, we propose a Bidirectional Visual-Pose Alternating Collaborative Attention (BiVPA Co-attention) module to facilitate cross-modal information fusion, allowing different modalities to complement and enhance one another. Moreover, the BiVPA Co-attention module jointly integrates modality features across both temporal and spatial dimensions, effectively alleviating information mismatch and dependency gaps among modalities. Finally, we design a head network incorporating BERT to resolve semantic ambiguities and perform sign language recognition. In addition, the head network combines the complementary features generated by the BiVPA module with symbolic gloss features, further enhancing the effectiveness of multimodal feature fusion. Experimental results on three benchmark datasets, namely NMFs-CSL, MSASL, and WLASL, demonstrate that the proposed MB-SLR method achieves substantial performance improvements.