The rapid spread of social media in recent years has given rise to a new multimodal entity: internet memes. The advanced models reconstruct the memes category detection task into an ITM (Image-Text Matching) task based on contrastive learning, showing great improvement over traditional classifiers. However, current methods do not effectively consider memes’ characteristics when generating hard negative samples. To address this issue, we propose a memes category detection model via a Simulation of memes Recreation (abbreviated as SimRe). Considering the memes distinguished characteristics, i.e., the ease of creation and modification of memes, we design image-text linear interpolation to simulate the recreated process of memes. Then, our be/not prompt templates add the category words of memes into text modality. Based on interpolation and prompt, harder negative samples with a secondary creation style can be generated, bringing perturbation to category label distribution of the original dataset. Finally, experimental results on two datasets show that our model outperforms state-of-the-art methods in terms of accuracy based measures and meanwhile its robustness is enhanced via weight adjustment.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

SimRe: A Simulation of Memes Recreation for Memes Category Detection

  • Lin Li,
  • Leqi Zhong,
  • Jian Cui,
  • Shaopeng Tang,
  • Xiaohui Tao

摘要

The rapid spread of social media in recent years has given rise to a new multimodal entity: internet memes. The advanced models reconstruct the memes category detection task into an ITM (Image-Text Matching) task based on contrastive learning, showing great improvement over traditional classifiers. However, current methods do not effectively consider memes’ characteristics when generating hard negative samples. To address this issue, we propose a memes category detection model via a Simulation of memes Recreation (abbreviated as SimRe). Considering the memes distinguished characteristics, i.e., the ease of creation and modification of memes, we design image-text linear interpolation to simulate the recreated process of memes. Then, our be/not prompt templates add the category words of memes into text modality. Based on interpolation and prompt, harder negative samples with a secondary creation style can be generated, bringing perturbation to category label distribution of the original dataset. Finally, experimental results on two datasets show that our model outperforms state-of-the-art methods in terms of accuracy based measures and meanwhile its robustness is enhanced via weight adjustment.