A Cloud-Based Sign Language Translation System via CNN with Smart Glasses

In situations where ordinary people and the hearing-impaired person need to communicate, it is possible that the average person does not know sign language and thus communication may be impaired, which means that a technology or device to assist communication is needed. Therefore, this study develops a new cloud sign language translation system on smart device based on the Browser/Server architecture, so that when the hearing-impaired person makes a sign language movement in front of the user who using the system with a smart device (e.g., smart glasses), the screen of the smart device will display the subtitle of the sign languages. We use MediaPipe to recognize and collect sign language action data from WLASL dataset and provide it to TensorFlow’s 1D-CNN deep learning model for training, so as to realize the sign language translation function. In the test phase, we invited five experimenters to test the sign language translation system ten times for each person, and the final average accuracy rate was 72%. Through such an interpreting system, it brings a convenient and efficient communication experience to the hearing-impaired person and people who need to communicate with the hearing-impaired person, and at the same time, it can also provide support for broader sign language research and application.