DriveSense: A Multi-modal Emotion Recognition and Regulation System for a Car Driver

Negative emotions significantly impact cognitive behavior and are a critical factor in driver safety and road traffic security. As intelligent driving systems evolve, the recognition and management of driver emotions has emerged as a crucial focus in automotive Human-Machine Interaction (HMI). We introduce DriveSense, an innovative emotion recognition and regulation system. DriveSense utilizes multi-modal data from onboard sensors, processes this data via deep learning in the cloud, and communicates with the HMI to implement regulation strategies. Our multi-modal emotion recognition model combines facial expression analysis and speech processing through Mel-frequency cepstral coefficients (MFCC), achieving a 60.37% accuracy on the RAVDESS dataset. We further validate DriveSense’s utility through an experiment with 40 participants using a simulated driving scenario to test an adaptive music-based emotion regulation strategy. The results indicate that adaptive music can mitigate negative emotions effectively, underscoring DriveSense’s potential to improve driver safety and secure driving practices.