In the present paper, we propose MoDeG-Prompt, a depth-enhanced multimodal method for dynamic gesture recognition that integrates adaptive cross-modal prompting and temporal consistency optimization to improve recognition accuracy in few-shot learning scenarios. By using both RGB and depth modalities, MoDeG-Prompt effectively resolves spatial ambiguities such as occluded hand configurations, and increases feature granularity to better capture complex and subtle gesture dynamics. The core of the method is a modified multimodal cross-attention module that dynamically aligns RGB and depth streams, focusing on the extraction of discriminative spatio-temporal patterns such as finger articulations, hand motion trajectories, and facial expressions. To ensure temporal coherence across successive frames, a temporal consistency module penalizes abrupt feature discrepancies, preserving smooth transitions in dynamic gesture sequences. In addition, MoDeG-Prompt incorporates Vision Mamba and Mamba2, both state-space models that efficiently capture short- and long-range spatio-temporal dependencies, improving scalability and significantly reducing computational overhead. Extensive experiments on the Ankara University Turkish Sign Language (AUTSL) corpus show that MoDeG-Prompt performs with 98.87% accuracy, outperforming previous state-of-the-art models in dynamic gesture recognition. This method provides a robust solution for practical implementation, including sign language translation and gesture-based human-computer interaction.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MoDeG-Prompt: Depth-Enhanced Multimodal Gesture Recognition with Dynamic Cross-Modal Prompting for Few-Shot Learning

  • Dmitry Ryumin,
  • Angelina Egorova

摘要

In the present paper, we propose MoDeG-Prompt, a depth-enhanced multimodal method for dynamic gesture recognition that integrates adaptive cross-modal prompting and temporal consistency optimization to improve recognition accuracy in few-shot learning scenarios. By using both RGB and depth modalities, MoDeG-Prompt effectively resolves spatial ambiguities such as occluded hand configurations, and increases feature granularity to better capture complex and subtle gesture dynamics. The core of the method is a modified multimodal cross-attention module that dynamically aligns RGB and depth streams, focusing on the extraction of discriminative spatio-temporal patterns such as finger articulations, hand motion trajectories, and facial expressions. To ensure temporal coherence across successive frames, a temporal consistency module penalizes abrupt feature discrepancies, preserving smooth transitions in dynamic gesture sequences. In addition, MoDeG-Prompt incorporates Vision Mamba and Mamba2, both state-space models that efficiently capture short- and long-range spatio-temporal dependencies, improving scalability and significantly reducing computational overhead. Extensive experiments on the Ankara University Turkish Sign Language (AUTSL) corpus show that MoDeG-Prompt performs with 98.87% accuracy, outperforming previous state-of-the-art models in dynamic gesture recognition. This method provides a robust solution for practical implementation, including sign language translation and gesture-based human-computer interaction.