Scholarlink

<p>Continuous sign language recognition (CSLR) aims to transcribe sign language videos into target gloss sequences. This task presents certain challenges as it requires learning the spatial and temporal dimensions of sign language frame sequences. The architecture of the backbone can be categorized into three main components: the visual module, the contextual module, and the alignment module. However, the spike phenomenon of the connectionist temporal classification (CTC) algorithm, which is commonly used in the alignment module, causes the features extracted by the network to be not discriminative enough, leading to inaccurate recognition results. To address this, we propose a dual spatial–temporal enhancement network (DSTEN) that aims at enhancing the network training from both spatial and temporal perspectives. The first enhancement focuses on improving the visual module, which often suffers from insufficient training. Since sign language mainly conveys information through the signer’s face and hands, this paper proposes to insert a spatial-channel attention (SCA) module assisted by heatmaps of keypoints into the visual module, i.e., dense keypoints-assisted spatial enhancement (DKSE), which aims to capture more fine-grained discriminative features. The second enhancement is designed to optimize the sequence learning capability of the contextual module, as existing contextual modeling networks commonly used for most sequence tasks are not entirely suitable for CSLR. Considering the characteristic of local semantic consistency in sign language, we propose a multi-scale temporal enhancement (MSTE) network based on 1D convolutional neural network (1D-CNN) to capture long- and short-term dependencies. It demonstrates excellent contextual modeling capability and effectively captures information at different time scales through an adaptive multi-scale mechanism while providing feedback to the visual module. Extensive experiments demonstrate the effectiveness of the proposed network by competitive performance on the PHOENIX-2014, PHOENIX-2014T, CSL and CSL-Daily datasets.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

DSTEN-CSLR: dual spatial–temporal enhancement network for continuous sign language recognition

  • 期刊论文

  • 发表日期：

    2025年5月6日

查看全文
  • Ming Yu,
  • Aobo Gan,
  • Cuihong Xue,
  • Gang Yan

摘要

Continuous sign language recognition (CSLR) aims to transcribe sign language videos into target gloss sequences. This task presents certain challenges as it requires learning the spatial and temporal dimensions of sign language frame sequences. The architecture of the backbone can be categorized into three main components: the visual module, the contextual module, and the alignment module. However, the spike phenomenon of the connectionist temporal classification (CTC) algorithm, which is commonly used in the alignment module, causes the features extracted by the network to be not discriminative enough, leading to inaccurate recognition results. To address this, we propose a dual spatial–temporal enhancement network (DSTEN) that aims at enhancing the network training from both spatial and temporal perspectives. The first enhancement focuses on improving the visual module, which often suffers from insufficient training. Since sign language mainly conveys information through the signer’s face and hands, this paper proposes to insert a spatial-channel attention (SCA) module assisted by heatmaps of keypoints into the visual module, i.e., dense keypoints-assisted spatial enhancement (DKSE), which aims to capture more fine-grained discriminative features. The second enhancement is designed to optimize the sequence learning capability of the contextual module, as existing contextual modeling networks commonly used for most sequence tasks are not entirely suitable for CSLR. Considering the characteristic of local semantic consistency in sign language, we propose a multi-scale temporal enhancement (MSTE) network based on 1D convolutional neural network (1D-CNN) to capture long- and short-term dependencies. It demonstrates excellent contextual modeling capability and effectively captures information at different time scales through an adaptive multi-scale mechanism while providing feedback to the visual module. Extensive experiments demonstrate the effectiveness of the proposed network by competitive performance on the PHOENIX-2014, PHOENIX-2014T, CSL and CSL-Daily datasets.