MHAMF: mamba-based emotional hyper-modal assisted multi-granularity fusion for emotion recognition in conversations
摘要
Emotion recognition in conversations is a critical component of emotional dialogue systems, playing a significant role in areas such as human-computer interaction and social media analysis. However, existing research primarily focuses on modeling contextual semantic information and speaker-specific details, while overlooking the subtle temporal information within long-range context dependencies and the heterogeneity between modalities. This results in insufficient global semantic understanding of conversations and prevents the optimal fusion of different modality information. To address these challenges, this paper proposes the Mamba-Based Emotional Hyper-Modal Assisted Multi-Granularity Fusion Method (MHAMF) for emotion recognition in conversations. This method constructs two feature subspaces and utilizes the mamba structure to extract subtle temporal information from dynamic emotions in the dialogue, thereby building an auxiliary hyper-modal. Additionally, through a hierarchical cross-modal fusion structure, it iteratively optimizes the fusion of fine-grained multi-modal features at each layer, adaptively weighting the two subspace features to maximize the complementary fusion effect of multi-modal features. Extensive experiments on two public datasets, IEMOCAP and MELD, demonstrate that MHAMF achieves significant accuracy and robustness in emotion recognition in conversations.