Exploring Multimodal Features for Sentiment Classification of Social Media Data

Effectively capturing and interpreting sentiments from image and text data is a challenge for the sentiment analysis task. While the expressive objects in an image that evokes human emotion are commonly explored in sentiment analysis, the object’s attributes often remain unexplored. In this paper, we describe the extraction of objects, features, and attributes from images and text individually and in different combinations. We conducted extensive experiments, using two different datasets, to evaluate the performance of sentiment analysis models across these features. We demonstrate the best multimodal features across image attributes and text features that can be used to classify the sentiment. We also identify the efficacy of the CNN and Fusion models among the four considered models. This study contributes to sentiment analysis by utilizing untapped attributes of objects in images and demonstrating the advantages of combining features from multimodal data using deep learning models.