Multimodal Topic and Sentiment Recognition for Chinese Data Based on Pre-trained Encoders

With the rapid development of mobile internet technology, massive amounts of multimodal data have emerged from major online platforms. However, defining multimodal topic categories is a more subjective task, and the lack of a public Chinese dataset hinders the progress of the task. And fine-grained multimodal sentiment classification is extremely challenging, especially for aspect-level classification. In this paper, we build a Chinese dataset based on Weibo for multimodal topic classification and propose a pre-trained encoder-based model for topic classification and fine-grained sentiment classification. The experimental results on Multi-ZOL and our proposed dataset show that the proposed model outperforms the benchmark model on the multi-modal topic classification task, achieving an accuracy of 0.99. On the fine-grained multimodal sentiment classification task, the proposed model achieves an accuracy of 0.5860 and a macro-F1 of 0.5791, which is competitive with SOTA.