Sign Language is a global mode of communication used by the deaf and hard of hearing, employing body key points to form meaningful glosses. It varies regionally, with Uganda using a specific Uganda Sign Language (USL). The Real-Time Translation from Ugandan Sign Language to Speech project aims to translate USL into English speech using pose estimation, hand tracking, and sequence-to-sequence modeling. The methodology involved creating a dataset of USL gestures with English annotations, focusing on non-manual features like hand position and orientation, extracted using the MediaPipe library. An encoder neural network captured spatio-temporal information, and models using ResNet50 and VGG19 were compared. The architecture included convolutional layers for feature extraction and transformer blocks for capturing long-range dependencies. The model achieved 100% accuracy across all classes, with perfect precision, recall, and \(F_1\) -scores. Training over 60 epochs showed consistent improvements, with training accuracy rising from 87.16 to 88.12%, and validation accuracy reaching 100%. The validation loss dropped significantly to 0.000042. Regularization techniques like dropout prevented overfitting, confirmed by a detailed confusion matrix showing no misclassifications.