Deep learning algorithms are used in a system that detects driver distraction in real time. The system is designed to identify many forms of distracted driving, including texting, eating, and using a phone. To record the structural information of the driver’s behaviour, the proposed system combines the representation of a body pose or body-object connection. High-level semantics and convolutional neural network (CNN) features are combined using a multi-stream deep fusion network (MDFN). On difficult datasets, experimental findings show that the suggested strategy considerably increases the driver’s action recognition accuracy. One of the main factors contributing to accidents on the road is driver distraction. The number of accidents caused by distracted driving can be reduced with real-time driver distraction in real time. The purpose of this study is to develop a deep learning-based system for the detection of driver distraction detection.