In daily life, communication is an important basis for social interaction. However, people who are deaf or hard of hearing encounter many obstacles in communicating due to their hearing impairment, which seriously affects their social participation and quality of life. To solve this problem, this paper designs and implements a bidirectional intelligent sign language recognition glove based on multi-sensor fusion and artificial intelligence. The system uses Flex2.2 bending sensors and MPU6050 six-axis posture sensors to collect hand dynamic information, and then transmits this data via STM32 Bluetooth communication to the mobile phone side, which in turn realizes the voice output and recording of sign language translation. On the mobile phone side, this project utilizes the Long Short-Term Memory (LSTM) network to construct a sign language recognition model and introduces the Transformer architecture to realize the speech output function. The system supports the bidirectional conversion of sign language to speech and speech to text, thus facilitating barrier-free communication between hearing-impaired people and hearing people. The experimental results show that the system performs well in the accuracy of sign language recognition and voice interaction experience, enabling potential applications in the fields of barrier-free communication and assisted medical care.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Design and Implementation of Bidirectional Intelligent Sign Language Recognition Gloves Based on Multi-sensor Fusion and Artificial Intelligence

  • Guochen Zhang,
  • Rui Wen,
  • Qi Zhou,
  • Gang Cen,
  • Junyan Luo,
  • Zhiqi Jin

摘要

In daily life, communication is an important basis for social interaction. However, people who are deaf or hard of hearing encounter many obstacles in communicating due to their hearing impairment, which seriously affects their social participation and quality of life. To solve this problem, this paper designs and implements a bidirectional intelligent sign language recognition glove based on multi-sensor fusion and artificial intelligence. The system uses Flex2.2 bending sensors and MPU6050 six-axis posture sensors to collect hand dynamic information, and then transmits this data via STM32 Bluetooth communication to the mobile phone side, which in turn realizes the voice output and recording of sign language translation. On the mobile phone side, this project utilizes the Long Short-Term Memory (LSTM) network to construct a sign language recognition model and introduces the Transformer architecture to realize the speech output function. The system supports the bidirectional conversion of sign language to speech and speech to text, thus facilitating barrier-free communication between hearing-impaired people and hearing people. The experimental results show that the system performs well in the accuracy of sign language recognition and voice interaction experience, enabling potential applications in the fields of barrier-free communication and assisted medical care.