Vision-based sign language recognition significantly advances communication within the deaf community, enhancing accessibility and inclusion for those who are deaf or hard of hearing. This paper presents a system developed for real-time recognition of South African Sign Language (SASL) using Google’s MediaPipe framework for spatial feature extraction and a long short-term memory (LSTM) network for temporal modelling. We utilise a subset of the ASL Citizen dataset, focusing on five classes: “SCHOOL,” “TIMEOUT,” “MORNING,” “THANK YOU,” and “I LOVE YOU,” which serve as proxies for SASL vocabulary. Keypoint sequences from both hands and body pose are extracted via MediaPipe and fed into a two-layer LSTM for classification. Trained with a TensorFlow TFRecord pipeline, our model achieves a test accuracy of 35.5% and highlights the challenges posed by limited data and variability among signers. This work demonstrates the potential of combining MediaPipe and LSTM for real-time sign recognition and emphasises the need for larger, language-specific datasets to improve accuracy.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Real-Time South African Sign Language Interpretation Using Computer Vision Methods

  • Precious Magodi,
  • Tevin Moodley

摘要

Vision-based sign language recognition significantly advances communication within the deaf community, enhancing accessibility and inclusion for those who are deaf or hard of hearing. This paper presents a system developed for real-time recognition of South African Sign Language (SASL) using Google’s MediaPipe framework for spatial feature extraction and a long short-term memory (LSTM) network for temporal modelling. We utilise a subset of the ASL Citizen dataset, focusing on five classes: “SCHOOL,” “TIMEOUT,” “MORNING,” “THANK YOU,” and “I LOVE YOU,” which serve as proxies for SASL vocabulary. Keypoint sequences from both hands and body pose are extracted via MediaPipe and fed into a two-layer LSTM for classification. Trained with a TensorFlow TFRecord pipeline, our model achieves a test accuracy of 35.5% and highlights the challenges posed by limited data and variability among signers. This work demonstrates the potential of combining MediaPipe and LSTM for real-time sign recognition and emphasises the need for larger, language-specific datasets to improve accuracy.