A conditioned joint-modality attention fusion approach for multimodal aspect-level sentiment analysis

Multimodal aspect-level sentiment analysis (MALSA) aims to predict the sentiment polarity of each given aspect in multimodal contexts. The previous studies usually developed deep neural networks to capture the impacts that a given aspect brings to text and images. However, the dynamic interaction between the intra-modality and inter-modality relations is seldom investigated before fusing the textual and visual representations. This paper presents a conditioned joint-modality attention fusion approach for the MALSA task, which can iteratively deliver useful information flow between and across textual and visual modalities under the guidance of aspect information for sentiment polarity prediction. The point is the dual conditioned-attention mechanism, which calculates intra-modality attention flows dynamically modulated by the other modality. Experiments are conducted on three public datasets including Twitter-2015, Twitter-2017 and Multi-ZOL. Results show that the proposed model outperforms the state-of-the-art models, and demonstrate the effectiveness of the proposed approach.