Semantic Enhancement Network Integrating Label Knowledge for Multi-modal Emotion Recognition

Human emotion recognition is a meaningful and complex task, which plays a critical role in human-computer interaction. A number of multi-modal methods have been developed to make good use of dependencies between mul- tiple physiological signals, which achieve superior performance than uni-modal methods, but still suffer from information loss and information redundancy problems. Some import information in a single modality is ignored, while some irrelevant information is overemphasized. To solve these challenges, this study proposes a novel multi-modal emotion recognition framework called SEIL. The framework employs the concept of channel independence, assigning two channels to each modality to learn emotional expressions. One channel is used to fuse with other modalities, another is left as a separate input, thereby minimizing the loss of emotional information in the final fusion stage. To address the information redundancy problem, we adopt sliding window and aggregate methods to filter out noise information, and select useful local features at the same time. We assume that domain knowledge plays an important role in multi-modal fusion, so we further introduce label knowledge into SEIL, and utilize a label-aware attention mechanism to enhance the emotional key frames of the fused vector. The proposed framework is tested on IEMOCAP dataset, results show that the SEIL framework outperforms all the baseline algorithms in terms of WA and UA metrics, achieving maximum improvements of 4.5% and 7.5% respectively.