Real Time Sign Language Recognition Using Custom Convolutional Neural Network and YOLOv5

Our aim is to close the gap in communication between impaired people and make their life much easier with other people. Different solutions were available in the market but no single solution solves people with visually impaired, deaf, or speech impairment. This proposed work gives a single solution to the above three impaired people for better communication. This work uses two augmented American sign language datasets for implementation. Various trained and finetuned models are used in this work, but our proposed Custom Convolutional Neural Network (CCNN) model attained 82.92%, and 88% accuracy which outperformed other pre-trained models on the augmented dataset1 and dataset2 respectively. The best-performing model, the Custom CNN model, was chosen to create a complete pipeline from sign language to speech. The Custom CNN model’s output is input to the silero’s Text-to-Speech pre-trained model, which gives the speech version as output.