Image-text sentiment classification represents a significant component of multimodal sentiment analysis, which seeks to evaluate user attitudes derived from the images and texts shared on social media platforms. Nonetheless, current approaches that rely on multimodal fusion continue to encounter obstacles as a result of the semantic inconsistency between the modalities and the varying complexity of each image-text pair. To address these issues, we propose a model-agnostic multi-curriculum enhancement framework (MCEF) aimed at sentiment classification of image-text pairs. In particular, we design a hybrid approach to curriculum learning based on semantic similarity and the information entropy of image-text pairs to measure the semantic consistency of the images and texts and the information quantity of each pair. By gradually training with an adjusted order of training samples, our approach effectively handles the substantial noise caused by the two aforementioned issues. We validated the proposed method using models with different multimodal fusion approaches, such as Se-MLNN and CLIP-CA-CG. Extensive experimental findings demonstrate that the MCEF framework enhances the performance of the two foundational models, Se-MLNN and CLIP-CA-CG. Taking CLIP-CA-CG as an example, the accuracy on the MVSA-Multiple dataset increased from 73.95% to 75.18%, and the F1-score improved from 73.83% to 74.97%.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MCEF: Multi-curriculum Enhancement Framework for Social Image-Text Sentiment Classification

  • Honghao Huang,
  • Ya Pan,
  • Yong Fan

摘要

Image-text sentiment classification represents a significant component of multimodal sentiment analysis, which seeks to evaluate user attitudes derived from the images and texts shared on social media platforms. Nonetheless, current approaches that rely on multimodal fusion continue to encounter obstacles as a result of the semantic inconsistency between the modalities and the varying complexity of each image-text pair. To address these issues, we propose a model-agnostic multi-curriculum enhancement framework (MCEF) aimed at sentiment classification of image-text pairs. In particular, we design a hybrid approach to curriculum learning based on semantic similarity and the information entropy of image-text pairs to measure the semantic consistency of the images and texts and the information quantity of each pair. By gradually training with an adjusted order of training samples, our approach effectively handles the substantial noise caused by the two aforementioned issues. We validated the proposed method using models with different multimodal fusion approaches, such as Se-MLNN and CLIP-CA-CG. Extensive experimental findings demonstrate that the MCEF framework enhances the performance of the two foundational models, Se-MLNN and CLIP-CA-CG. Taking CLIP-CA-CG as an example, the accuracy on the MVSA-Multiple dataset increased from 73.95% to 75.18%, and the F1-score improved from 73.83% to 74.97%.