This paper presents a hybrid approach to sign language recognition that combines transformer-based modeling with keypoint-guided frame selection. In video-based sign language datasets, noise and redundancy are prevalent due to uninformative or repetitive frames, which can degrade model performance and increase computational cost. To address this issue, we propose DotRand, a keypoint-driven frame selection algorithm that judiciously selects the most informative frames based on motion dynamics, reducing irrelevant visual content while preserving essential gesture sequences. Our hybrid model integrates DotRand with the Video Swin Transformer, a hierarchical vision transformer designed for spatiotemporal feature learning. We evaluate this approach on two datasets: the Thai Sign Language (TSL) dataset and the widely used Word-Level American Sign Language 100 (WLASL100) benchmark. The TSL dataset, being low-resource and limited in size, presents specific challenges for deep learning. To overcome these limitations, we introduce data augmentation techniques, including controlled rotation and Gaussian noise injection to improve the model’s robustness in real-world environments. For comparison, we implement a CNN-BiLSTM baseline, a commonly used architecture for video-based modeling. Experimental results show that our proposed model achieves 92.12% top-1 accuracy and 97.27% top-5 accuracy on the TSL dataset. On the WLASL100 dataset, our model yields 70.54% top-1 accuracy and 88.76% top-5 accuracy, significantly outperforming the CNN-LSTM, which achieves only 46.90% top-1 and 77.91% top-5 accuracy. The inclusion of both a high-resource (WLASL100) and a low-resource (TSL) dataset enables a comprehensive evaluation of the model’s benchmark generalization capability. These findings highlight the effectiveness of combining keypoint-informed frame selection, data augmentation, and transformer-based architectures efficient sign language recognition in noisy and resource-constrained settings.