FL-STGCN: Mental Health Recognition Using RGB and IR Landmark Feature Fusion
摘要
Mental health recognition is a critical aspect of healthcare, often relying on subjective evaluations or time-consuming assessments by psychologists or specialists. With the growing availability of facial data through various sensors, there is an emerging opportunity to develop automated systems for mental health recognition. This paper introduces a novel FaceLandmark-STGCN approach for mental health recognition, leveraging RGB and infrared (IR) facial landmark fusion. The proposed method combines the strengths of RGB and IR modalities, harnessing the rich information present in facial landmarks and temporal dynamics. Specifically, we utilize spatial-temporal graph convolutional networks (STGCN) to capture intricate facial movements and expressions over time. Additionally, we introduce a fusion strategy that effectively integrates RGB and IR face-landmarks features at both spatial and temporal levels. The experimental evaluations conducted using our in-house dataset named CARD: Contactless Affect Recognition Database encompassing individuals with varying mental health conditions demonstrate the effectiveness of the proposed FaceLandmark-STGCN framework. The results showcase superior performance compared to state-of-the-art deep neural and graph neural network methods, highlighting the potential of multimodal face-landmark fusion for accurate and efficient mental health recognition using both in-house and NIVE multimodal datasets. This research contributes to advancing automated mental health assessment systems, offering promising avenues for real-world applications in healthcare and well-being monitoring.