The time and efforts needed to learn sign language have created communication barriers with hearing-impaired communities, highlighting the need to develop an effective translation medium. While professional human sign experts exist, relying on their physical presence for every translation is impractical and text-based deep learning models often fail to provide dynamic translations. In this research, a ConvLSTM2D-based deep learning approach is proposed to develop a real-time speech-to-sign language translation model that overcomes the limitations of relying on human experts and enhances dynamic translation capabilities. The WLASL-2021 dataset is utilized to train the proposed model with the sign language gestures performed by expert signers. Data preprocessing, including Frame Extraction, Selection, Normalization, and Augmentation, is performed before it is fed to the convolutional layer of CNN for extracting key features of the data. ConvLSTM2D layers, MaxPooling3D, and Dense layers with softmax activation enhance the model’s dynamic translation abilities and resolves overfitting issues. This work proposes a system that captures real-time speech, converts into text, pre-processes the text through tokenization, stop words removal, lemmatization and Named Entities Recognition (NER) decomposition, then identifies and visualizes corresponding sign language gestures with high accuracy. The trained model achieves a notable accuracy of 98.18%, successfully identifying the sign gestures corresponding to the spoken words. The proposed system performs effective translations of real-time speech into sign language gestures, thus, contributing to bridging the communication barriers for hearing impaired communities.