Multimodal Depression Detection Network Based on Emotional and Behavioral Features in Conversations

Early detection of depression has always been a challenge. Currently, research on automatic depression detection mainly focuses on using low-level features such as audio, text, or video from interview dialogue as input data, ignoring some high-level features contained in the dialogue. We proposes a multimodal depression detection method for extracting emotional and behavioral features from dialogue and detecting early depression. Specifically, we design an emotional feature extraction module and a behavioral feature extraction module, which input the extracted emotional and behavioral features as high-level features into the depression detection network. In this process, a weighted attention fusion module is used to guide the learning of text and audio modalities and predict the final result. Experimental results on the public dataset DAIC-WOZ show that the extracted emotional and behavioral features effectively complement the high-level semantics missing in the network. Our proposed method improves the F1-score by 6% compared to traditional approaches. The experimental data also indicate the importance of the model’s detection results in early depression detection. This technology has certain application value in professional fields such as caregiving, emotional interaction, psychological diagnosis, and treatment.