Semantic-wise guidance for efficient multimodal emotion recognition with missing modalities

Emotions play an important role in human–computer interaction. Multimodal emotion recognition combines feature information from different modalities to recognize emotional states. However, in real application scenarios, data from all modalities may not always be available. Thus, in multimodal emotion recognition a big challenge is how to utilize the semantic information from available modalities to predict missing modality data. To address this issue, this study proposes a Semantic-Wise Guidance for Missing Modality Imagination Network (SWG-MMIN) consisting of three main modules, that is, the Comprehensive Modality Feature Enrichment (CMFE) module, the Semantic-Wise Fusion (SWF) module, and the Semantic-Wise Feature Guided Imagination (SWGI) module. The CMFE module addresses the issue of semantic loss in the process of integrating multimodal features by enhancing the semantic information. The SWF module performs an adaptive fusion of invariant and specific features of multimodal data. The SWGI module facilitates the missing modality data generation and enhances the robustness of joint multimodal representation. Extensive experiments are conducted on two benchmark datasets, IEMOCAP and MSP-IMPROV. The experimental results demonstrate that the SWG-MMIN model surpasses all baseline models under full modalities and uncertain missing modalities, significantly improving emotion recognition performance.