A Robust Ensemble Approach to Face Expression Recognition and Image Sentiment Analysis

The field of image-based sentiment analysis is rapidly expanding, yet it grapples with complexities arising from the diverse factors influencing emotional states depicted in images. In response to these challenges, this work introduces a Facial Expression Recognition (FER) model, which integrates three distinct classifiers: (i) Convolutional Neural Network (CNN), (ii) ResNet50, and (iii) InceptionV3. The ensemble approach, specifically the Model Averaging technique, was employed to consolidate the predictions from these three models. Subsequently, the FER model underwent training and evaluation on the FER2013 dataset, representative of an unconstrained setting. Experimental outcomes underscored the pre-eminence of the ensemble strategy in discerning positive and neutral facial expressions, attaining accuracy levels of 91.7%, 81.7%, and 76.5% for happy, surprise, and neutral expressions, respectively. Conversely, when identifying expressions of disgust, anger, and sadness, ResNet50 emerged as the top-performing model. Notably, the Custom CNN demonstrated proficiency in detecting fear, reporting an accuracy of 55.7%. Impressively, the holistic FER model also showed commendable performance in this category, achieving an accuracy close to 52.8%. This study, thus, accentuates the merits of the ensemble approach in enhancing FER accuracy, with the integrated FER model reaching an overall accuracy of 72.3%.