Hybrid Information-Based Sign Language Recognition System

Sign language is generally used by the deaf and mute community of the world for communication. An efficient sign language recognition system can prove to be a breakthrough for the deaf–mute population of the world by assisting them to better communicate with people who don’t understand sign language. The current solutions available for sign language recognition require a lot of computation power or are dependent on some additional hardware to work. Thus, reducing their application in the real world and we aim to bridge this gap. In this paper, we are presenting our approach for developing machine learning models for sign language recognition with low computation requirements while maintaining high accuracy. We are using a combination of hand gesture images and hand skeleton information as model input to classify gestures. For this task, we are training CNN models and linear models to handle gesture image and hand skeleton data and later the results of these models serve as input for final classification layers. The hand skeleton information can also be used to detect and track the hand position in the frame, thus reducing need for additional computation overhead to detect hand in the frame.