Comparison and Performance Evaluation of Fusion Mechanism for Audio–Video Based Multimodal Emotion Recognition

Artificial emotional intelligence is an emerging research area in artificial intelligence. In artificial intelligence, machine learning and deep learning techniques have provided more efficient and precise results in unimodal emotion recognition, but still, it is lacking with the limited information for feature extraction and does not consider the contextual behaviors and factors during the emotion classification. To overcome these shortcomings, in our proposed work, we focus on two crucial fusion mechanisms: Tensor fusion and low rank fusion that influence the accuracy and computational complexity of multimodal emotion recognition. In this multimodal approach, we are using audio and video modality to evaluate the performance of tensor fusion and low rank fusion mechanism. These fusion mechanisms are an emerging challenge in various applications such as multimodal emotion recognition, sentiment analysis, and computer vision. In our work, we are comparing and evaluating the performance of two fusion mechanisms, the tensor fusion network and low rank fusion on audio–video on IEMOCAP and CMU-MOSI datasets. We evaluate the performance on the following matrix: Mean Absolute Error (MAE), Pearson correlation (Corr), F1 score, and accuracy.