Multimodal Fatigue Detection in Drivers via Physiological and Visual Signals

Driving conditions such as stress, sleepiness and fatigue can easily lead to traffic accidents, and the detection of these unsafe conditions is an important means of ensuring driving safety. The fatigue detection system currently on the market suffers from a single detection feature and low detection accuracy. In view of the powerful feature extraction and fusion capabilities of neural networks, a multimodal driver fatigue detection method based on physiological and visual signals is proposed for processing physiological signals from wearable devices and visual signals from cameras during the driving process. Firstly, the model processes the physiological signal sequences and locates significant changes in physiological state, and extracts visual features based on visual signals, mainly eye features and complementary mouth and head features. Secondly, the fatigue features are obtained by fusing the physiological and visual features in the temporal dimension, based on which an effective fatigue detection model can be trained. We use publicly available datasets to segment physiological signal sequences such as heart rate and quantitatively capture change points. After testing in different visual environments such as day and night and with or without face occlusion, this model can meet the requirements of basic real-time fatigue detection with a high detection accuracy.