Real-time attention-based embedded LSTM for dynamic sign language recognition on edge devices

Sign language recognition attempts to recognize meaningful hand gesture movements and is a significant solution for intelligent communication across societies with speech and hearing impairments. Nevertheless, understanding dynamic sign language from video-based data remains a challenging task in hand gesture recognition. However, real-time gesture recognition on low-power edge devices with limited resources has become a topic of research interest. Therefore, this work presents a memory-efficient deep-learning pipeline for identifying dynamic sign language on embedded devices. Specifically, we recover hand posture information to obtain a more discriminative 3D key point representation. Further, these properties are employed as inputs for the proposed attention-based embedded long short-term memory networks. In addition, the Indian Sign Language dataset for calendar months is also proposed. The post-training quantization is performed to reduce the model’s size to improve resource consumption at the edge. The experimental results demonstrate that the developed system has a recognition rate of 99.7% and an inference time of 500 ms on a Raspberry Pi-4 in a real-time environment. Lastly, memory profiling is performed to evaluate the performance of the model on the hardware.