The Deaf and Hard of Hearing community relies heavily on sign language as their main method of communication. This research delves deeply into the use of artificial intelligence (AI) for the real-time identification and translation of sign language gestures into written and spoken language. Our innovative approach combines computer vision and Natural Language Processing (NLP) techniques to break down barriers between signers and non-signers. By utilizing convolutional neural networks (CNNs) and recurrent neural networks (RNNs), we are able to analyze live video feeds of individuals signing. These networks are trained on a diverse dataset that covers various sign languages and dialects, resulting in an exceptional accuracy rate of over 95 in real-time sign language recognition, surpassing all previous methods (Adeyanju et al. in Intell Syst Appl 12:200056, 2021, [1]). In the conversion stage, we employ advanced sequence-to-sequence models, such as Long Short-Term Memory (LSTM) networks, to convert the identified signs into coherent written text and synthesized speech. Our system is adaptable to multiple sign languages, making it versatile and all-inclusive. Additionally, we incorporate language models that have been pre-trained on large datasets, ensuring grammatical accuracy and fluidity in the generated text. The results we have obtained demonstrate the effectiveness of our cutting-edge AI technology in promoting communication between sign language users and non-signers (Papastratis et al. in Sensors (Basel) 21:5843, 2021, [2]).