MetaCRN: Language-Augmented Multimodal Metaphor Detection Using Cross-Modal Dynamic Replacement
摘要
Metaphors are important rhetorical devices that appear frequently in both language and visual content and are commonly used in daily life. As social media and the internet have developed, internet memes have become a significant part of cultural communication, with metaphors playing a prominent role. However, current multimodal metaphor detection faces challenges, including low-quality meme texts and insufficient interaction between modalities. To tackle these issues, this paper proposes the MetaCRN framework for multimodal metaphor detection, which is based on a large language model. The framework designs a Linguistic Insight and Knowledge Augmentation module and uses the Deepseek large language model to deeply analyze meme texts, extract the source and target domains in the texts, and generate concise explanations to help the model understand metaphors in the texts. To further optimize text-visual information fusion, this paper introduces a dynamic feature replacement fusion strategy that facilitates information exchange and weighting via a dynamic feature replacement mechanism and a spatial gated feedforward network, enhancing modality interaction. Experimental results demonstrate that MetaCRN outperforms several baseline models on the public Met-meme dataset and the multimodal sarcasm dataset, confirming its superiority in multimodal metaphor detection.