Creative images are images rich in implicit semantic information formed through artistic processing to express specific intentions such as persuasion, exhortation, and warning. Understanding and interpreting the implied intentions of creative images which usually accompany with metaphors is very valuable but challenging. In this paper, we introduce a Multi-modal Metaphor Explanation (MME) task: Given a creative image with a related condensed slogan, MME aims to generate the explanation to reveal the delicate metaphor. Different from image captioning which understands and describes the literal semantics of images, explaining multi-modal metaphor requires correctly recognizing semantic associations between the source and target domains, understanding aesthetic conception from concrete visual operations or artistic designs, and further interpreting the implicit intentions. To support the research on MME, we develop CIME, a new dataset with 3753 high-quality creative images and related condensed slogans. We build an extensive benchmark and set up rigorous evaluation metrics for this new research problem. Experimental results on the automatic evaluation metrics demonstrate the superiority of our dataset quality. Our dataset is available at https://github.com/VILANLab/Metaphor-CIME

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multi-modal Metaphor Explanation: A Dataset and Benchmark

  • Zikai Wang,
  • Wenye Zhao,
  • Cheng Yang,
  • Pijian Li,
  • Qingbao Huang

摘要

Creative images are images rich in implicit semantic information formed through artistic processing to express specific intentions such as persuasion, exhortation, and warning. Understanding and interpreting the implied intentions of creative images which usually accompany with metaphors is very valuable but challenging. In this paper, we introduce a Multi-modal Metaphor Explanation (MME) task: Given a creative image with a related condensed slogan, MME aims to generate the explanation to reveal the delicate metaphor. Different from image captioning which understands and describes the literal semantics of images, explaining multi-modal metaphor requires correctly recognizing semantic associations between the source and target domains, understanding aesthetic conception from concrete visual operations or artistic designs, and further interpreting the implicit intentions. To support the research on MME, we develop CIME, a new dataset with 3753 high-quality creative images and related condensed slogans. We build an extensive benchmark and set up rigorous evaluation metrics for this new research problem. Experimental results on the automatic evaluation metrics demonstrate the superiority of our dataset quality. Our dataset is available at https://github.com/VILANLab/Metaphor-CIME