This research proposes an innovative solution for detecting and translating Sign Language fingerspelling into text, addressing accessibility challenges for the Deaf and hard-of-hearing community. The model combines an enhanced Squeeze-former for feature extraction and a Transformer Decoder for sequence decoding, outperforming traditional ASR and machine translation technologies. Augmentations such as Cut Mix, Finger Dropout, and Time Stretch enhance generalization, while a diverse dataset of three million fingerspelled characters in real-world scenarios ensures superior performance. The 3 channel image interpretation, Llama attention, and optimized training allow for efficient deployment of deeper models. Transformer-based decoding is emphasized over CTC-based approaches, with a confidence score aiding in identifying corrupted examples during postprocessing. Training insights, including a cosine learning rate schedule, mixed precision, and tf-lite inference, demonstrate robustness and scalability. Post-processing techniques, incorporating confidence scores, improve accuracy, and the paper concludes with an ensemble approach, highlighting the minimal impact of supplemental data. Overall, the research significantly advances sign language recognition, promoting inclusivity in line with the mission of universal AI accessibility.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Finger Text: Deep Learning for Fingerspelling Translation

  • A. Arulmurugan,
  • Anushka Avinash Paliwal,
  • Akshat Agarwal

摘要

This research proposes an innovative solution for detecting and translating Sign Language fingerspelling into text, addressing accessibility challenges for the Deaf and hard-of-hearing community. The model combines an enhanced Squeeze-former for feature extraction and a Transformer Decoder for sequence decoding, outperforming traditional ASR and machine translation technologies. Augmentations such as Cut Mix, Finger Dropout, and Time Stretch enhance generalization, while a diverse dataset of three million fingerspelled characters in real-world scenarios ensures superior performance. The 3 channel image interpretation, Llama attention, and optimized training allow for efficient deployment of deeper models. Transformer-based decoding is emphasized over CTC-based approaches, with a confidence score aiding in identifying corrupted examples during postprocessing. Training insights, including a cosine learning rate schedule, mixed precision, and tf-lite inference, demonstrate robustness and scalability. Post-processing techniques, incorporating confidence scores, improve accuracy, and the paper concludes with an ensemble approach, highlighting the minimal impact of supplemental data. Overall, the research significantly advances sign language recognition, promoting inclusivity in line with the mission of universal AI accessibility.