Road safety is a very important issue in today’s society. With the growth of artificial intelligence and smart systems, opportunities arise to create systems that can reduce road causalities and prevent serious accidents. As human error is still the major cause of all collisions, driver distraction prevention is a very important subject. In this paper, we propose a convolutional neural network (CNN) model enhanced with dual self-attention modules for the classification of distracted drivers. The network architecture is based on the possibility of increasing the classifier’s attention to features by introducing a double attention module for a more accurate extraction of important features. The proposed model was tested using 5-fold cross-validation and achieved an average accuracy of 99.47%. The obtained results indicate, that the model can converge very quickly and reaches state-of-the-art performance. The proposed approach can be used as a warning module in cars with potential driver distraction.