Sign language is a crucial means of communication for people with disabilities, enabling them to express themselves through gestures and facial expressions. This paper presents a study that uses a specific deep learning paradigm to recognize gestures representing the American Sign Language (ASL) and Arabic Sign Language (ArSL) alphabet. The main objective of this study is to use modern technology to bridge the communication gap for the deaf and hard of hearing. The study trained and tested the VGG-16 CNN model using a Transformer learning model with a comprehensive dataset of over 24,300 (ASL) hand gestures and 7,856 (ArSL) hand gestures. The model’s construction parameters were adjusted to achieve maximum recognition accuracy. The test results showed that the VGG-16 CNN model achieved high accuracy, reaching 96.98% for the Arabic Sign Language dataset and 94.4% for the American Sign Language dataset. These findings have significant practical implications, as they demonstrate the potential of modern technology in improving communication for individuals with hearing impairments.