Enhancing Indian sign language recognition through data augmentation and visual transformer

This paper introduces a novel approach to Indian Sign Language Recognition (ISLR) by integrating Keras, Visual Transformers (ViT), and sophisticated data augmentation techniques. Our methodology emphasizes the development of a Vision Transformer model trained on a comprehensive dataset, enhanced with both image data and keypoint information, leveraging the capabilities of the Mediapipe library. To improve the model’s ability to generalize, we applied advanced augmentation strategies, including ImageDataGenerator, among others. Through extensive experimentation, involving rigorous hyperparameter optimization across numerous epochs, we sought to determine the most effective model configuration. The results of our validation process revealed a promising evaluation loss of 0.2941 and an impressive accuracy rate of 97.52%. This integration of data augmentation techniques with ViT transformers establishes a groundbreaking framework in the field of ISLR, underscoring its significant potential for practical implementation and marking a substantial progression in the technology for sign language recognition.