Enhancing Smart Home Accessibility for the Deaf Through Transformer-Based Sign Language Production and Avatars

The deaf community primarily relies on sign language for communication, but most hearing individuals lack proficiency in this language, creating a significant communication barrier. To close this gap, advanced deep learning methods have been developed to enable bidirectional communication. These methods facilitate automatic sign language recognition using cameras or wearable sensors and generate sign language from voice and text inputs. State-of-the-art techniques, particularly transformers, have achieved impressive results in realistic text-to-image translation. However, these advancements have yet to be integrated into real-world smart environments for the deaf. In our research, we present a lightweight transformer specifically designed for sign language production. Trained on the well-known PHOENIX-Weather-2014T dataset, this model generates realistic sign language sequences from text inputs, which are then animated by a lifelike avatar running on a game engine platform. In the Smart Home applications at the University of Almería, user commands and interactions are translated in real-time into sign language, displayed by the avatar on a screen. This innovation allows deaf individuals to receive messages and interact with their environment more inclusively, significantly enhancing their living experience in smart homes.