This research proposes a deep learning framework for Continuous Indian Sign Language Translation (CISLT), converting sign gestures into natural language text. It fuses spatial and visual features by combining 3D hand landmarks and video frames to capture temporal and semantic patterns in ISL. A custom dataset of eight sentence classes with frame-level word alignments is used for supervised training. Preprocessing involves frame extraction, normalisation, and landmark detection via MediaPipe. The architecture includes ResNet-18 for visual features, an MLP for landmarks, and a Seq2Seq model with spatiotemporal attention for word-level prediction. Data augmentation and precise sequence alignment improve reliability. The model performs well on unseen ISL data, demonstrating its practical and adaptable value for Deaf and hard-of-hearing communication.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Continuous Indian Sign Language Translator

  • Arpita Dutta,
  • Chandrima Kar,
  • Pratik Biswas,
  • Ayaan Ahmed,
  • Md Zaib Reyaz

摘要

This research proposes a deep learning framework for Continuous Indian Sign Language Translation (CISLT), converting sign gestures into natural language text. It fuses spatial and visual features by combining 3D hand landmarks and video frames to capture temporal and semantic patterns in ISL. A custom dataset of eight sentence classes with frame-level word alignments is used for supervised training. Preprocessing involves frame extraction, normalisation, and landmark detection via MediaPipe. The architecture includes ResNet-18 for visual features, an MLP for landmarks, and a Seq2Seq model with spatiotemporal attention for word-level prediction. Data augmentation and precise sequence alignment improve reliability. The model performs well on unseen ISL data, demonstrating its practical and adaptable value for Deaf and hard-of-hearing communication.