Advancing Accessibility: ASL Visual Recognition Technology Through EfficientNet

Abstract American Sign Language (ASL), a dynamic visual-gestural language, is utilized by the Deaf community in the US, Canada, and globally. Unlike spoken languages, ASL employs handshapes, expressions, and postures for communication. Recent strides in machine learning and computer vision drive the creation of automated ASL recognition systems, bridging the divide between Deaf and hearing individuals. Our approach proposes a CNN architecture, leveraging EfficientNetV2S through transfer learning and combining recent novel techniques like data augmentation, dropout layers, and the Nadam optimizer with L2 regularization and exponential learning rate decay. The final layer utilizes softmax probabilities to map 160px × 160px RGB images to 28 classes, enhancing communication for hearing-impaired individuals. This multi-stream CNN design achieves approximately 95% accuracy, optimized for various factors including frame rate, sign execution speed, and lighting conditions. Diverse data sets and augmentation strategies counter overfitting, yielding a model with 98.64% training accuracy and 95.01% unseen validation accuracy.