Bridging Signs to Sentences: Enhancing Sign Language Interpretation
摘要
This paper mainly focuses on bridging the gap between sign language recognition (SLR) and sentence formation by integrating the recognition of different signs or machine learning models with large language models (LLMs), which results in contributing and enhancing the communication of the deaf community. There are around 466 million deaf individuals worldwide, where they primarily rely on Sign language for communication. Current SLR technologies have certain limitations that deal with difficulty in sentence formation and high processing requirements. This paper’s dataset consists of 36 classes, where 26 of them are alphabets, and the rest 9 are numbers, each consisting of 500 images. Therefore, a total of 18,000 images are present in the dataset for accurate prediction of sign languages. MediaPipe, developed by Google, is used as a tool for feature extraction by identifying 21 hand landmarks. The features extracted are then passed onto an ML model (like a Random Forest), and then the result of this model is passed on to an LLM (here, Groq’s Gemma-7b-it) that forms a sentence based on the predictions. The ML model achieved a high accuracy of 99.54% and the LLMs achieved an accuracy of 93.75%. With the combination of sign language recognition and advanced modeling, this work helps to bridge signs to sentences, providing to the deaf community.