Hand Sign Detection and Voice Conversion for the Hearing and Speech Impaired Using Convolutional Neural Networks

Sign Language Recognition (SLR) tries to convert sign language into written or spoken form to enable communication between a deaf-mute person and a normal person. The task of communicating with disabled and impaired persons is extremely challenging, owing to its complexity and variety of hand motions that have a major societal impact. In order to depict sign language motions, SLR approaches are increasingly being used. These classification models are based on hand-crafted characteristics. Building reliable features that can adapt to the diverse range of hand signs is nonetheless challenging. To address this issue, the authors proposed a special 3D convolutional neural network (CNN) to ease the communication between normal and mute communities. Our proposed model eliminates the requirement to create features by automatically extracting discriminative spatial–temporal features from the actual livestream without any prior information. To improve performance, the 3D CNN is fed with multiple channels of video feeds that include colour, depth, trajectory information, joint body locations, and depth clues. We illustrate the proposed model's superiority to the conventional methods based on hand-crafted characteristics by testing them against a real dataset collected using a depth camera.