Hierarchical Supervised Contrastive Learning for Multimodal Sentiment Analysis

Multimodal sentiment analysis (MSA) is dedicated to deciphering human emotions in videos. It is a challenging task due to the semantic disparities among various modalities (e.g., linguistic, visual, and acoustic) present in video content. To bridge these gaps, we leverage contrastive learning and introduce a novel hierarchical multimodal approach termed Hierarchical Supervised Contrastive Learning (HSCL). Initially, we utilize an unimodal fusion combined with a supervised contrastive learning strategy to distill pertinent content from each modality. Subsequently, we combine the bimodal data in pairs and further align them through supervised contrastive learning. This paired data aids in understanding the intricate nuances of multidimensional human emotions. In our method, supervised contrastive learning is tailored to accentuate the significance of label information, facilitating the extraction of sentiment cues from diverse sources. Experimental studies on two benchmark datasets demonstrate the effectiveness of our method. Specifically, compared to the baseline on the CMU-MOSI benchmark, our method achieves 2.59% accuracy improvement for emotion recognition. Codes are released at https://github.com/Turdidae810/HSCL .