Driver Distraction Detection (DDD) enhances driver safety by identifying and alerting the distraction of drivers while driving a car. However, the existing works did not concentrate on multi-factor authentication for identifying authorized, drunken, distracted, and drowsy drivers. Thus, this paper presents an effective framework for DDD. At first, the Real-Time (RT) videos of the car drivers are pre-processed. Then, Karladall Correlation You Only Look Once Version 3 (KC-YOLOV3) is used for driver detection. In phase I, the face and eye detected results using Linear Gini Scaling Viola Jones (LGSVJ) are given to Hotfix-FTS Mish-Residual Attention Network (HT-FTSM-RAN) technique, which classifies the authorized and unauthorized drivers along with the drunken and non-drunken drivers. In phase II, the authorized and non-drunken drivers are checked for drowsiness and distraction using the HT-FTSM-RAN classifier. This gives an alert signal to the drivers to prevent the risk of accidents. The proposed work gives maximum accuracy of 98.97 and 99.21% to detect authenticated and drunken drivers, respectively, and gives 99.78% accuracy to detect drowsy and distracted drivers. Thus, this framework outperformed the existing methodologies.