Distracted driving remains a primary factor in road accidents worldwide, highlighting the need for the creation of effective and reliable detection technologies. This study examines the efficacy of three pre-trained models, VGG-16, ResNet-50, and EfficientNet-B0 in detecting distracted drivers, utilizing two benchmark datasets: State-Farm Distracted Driver Detection (SFD3) and RoboFlow Distracted Driver (RFDD). Every model was optimized and evaluated using classification accuracy and computational effectiveness. Both datasets contain 13 different categories. Expensively, EfficientNet-B0 demonstrated outstanding results, attaining a testing accuracy of 99.71% on the SFD3 dataset and 95.4% on the RFDD dataset. The findings surpassed those of ResNet-50 and VGG-16, demonstrating EfficientNet-B0’s capability to attain higher accuracy while preserving a compact model size and minimal computational costs. In addition, we develop a real-time driver alarm system, based on the proposed model, that quickly sends distracted behaviors to an Android application. Moreover, the results indicate that EfficientNet-B0 is the optimal architecture evaluated for real-time distracted driver detection, providing a viable solution for use in intelligent transportation systems.