Real-Time Sign Language Recognition of Words and Sentence Generation using MediaPipe and LSTM

Sign language is a way of communication of the speech and hearing impaired people. Lot of research has been carried out on recognition of signs of alphabets in American Sign Language (ASL) but not much on recognition of signs of words. In this paper, we propose a system for the recognition of signs of words in ASL in real time. The detected words are displayed as a sequence one after other like a sentence generated on the output window. A dataset of signs of nine words is created using OpenCV and MediaPipe Holistic which is used to train the Long Short-Term Memory (LSTM) network. The accuracy, precision and recall after training and testing the network is 100%. This work can be used to predict meaningful sentences.