Hierarchical sentiment inconsistency enhancement for multimodal sarcasm detection
摘要
Multimodal sarcasm detection leverages multimodal information such as images and texts to identify special instances where the expressed sentiment contradicts the genuine sentiment. Most existing sentiment inconsistency-based approaches overlooked the transitions of sentiment polarity within the textual modality and the guiding role of sarcasm in inter-modal sentiment inconsistency modeling. To address the aforementioned limitations, we propose a novel Hierarchical Sentiment Inconsistency-Enhanced framework (HSIE) for multimodal sarcasm detection. For intra-modal sentiment inconsistency, we propose a dual-level sentiment analysis module that integrates SenticNet-guided word-level sentiment inconsistency and LLM-guided sentence-level sentiment alignment to derive a global textual sentiment representation capable of capturing dynamic sentiment transitions. For inter-modal sentiment inconsistency, we propose a margin-based KL divergence method, which dynamically adjusts the image sentiment distribution by leveraging the relationship between sentiment and sarcasm, and leverages textual sentiment to guide the model to learn more discriminative cross-modal sentiment representations aligned with sarcastic semantics. For interplay between sarcasm and multimodal sentiment inconsistency, we introduce a sarcasm label-based contrastive learning method, which further enhances the multimodal sentiment inconsistency and the model’s ability to identify sarcasm by aligning and distancing multimodal sentiment feature representations within a same semantic space. Extensive experiments on a publicly available multimodal sarcasm detection dataset demonstrate the superiority of our proposed method. Compared to state-of-the-art approaches, our model achieves F1-score improvements of 4.84% and 4.25% on the MMSD and MMSD2.0 datasets, respectively.