Achieving a balance between accuracy and computational efficiency remains a key challenge in real-time sign language recognition. While deep learning approaches, particularly those based on convolutional neural networks (CNNs), have shown promise, they often suffer from high prediction variance, overfitting, and misclassification. To address these issues, we propose an ensemble framework comprising three diverse deep learning models. Notably, the third model integrates an attention mechanism that enhances the network’s ability to capture subtle yet discriminative spatio-temporal features. The proposed ensemble approach was evaluated on two publicly available datasets—Massey and ISL—achieving accuracies of 98.02% and 99.86%, respectively, thereby outperforming existing state-of-the-art methods. Furthermore, we developed a real-time application prototype equipped with a user-friendly graphical user interface (GUI), demonstrating the practical viability of our system in real-world settings.