Multimodal Sentiment Analysis with Graph Neural Network Using Audio and Text Features in Customer Service Applications
摘要
Nowadays, Multimodal Sentiment Analysis (MSA) has growing important in customer service application where understanding the customer emotions and opinions are crucial for providing efficient support. However, the existing Deep Multimodal Attention Fusion (DMAF) has failed to capture the nuances of human emotion expressed through audio signals. Hence, this research proposes an efficient Multimodal Fusion-based Graph Neural Networks (MF-GNNs) to handle the inherent noise and variability in customer service interactions. Initially, the data is collected from summa lingua technologies which comprises of audio and text as input data. Then, the text data is preprocessed with stemming and lemmatization to group related words by removing the affixed elements. Simultaneously, the audio signals are preprocessed with Nonnegative Matrix Factorization (NMF) to decompose the signals to remove the noise. After that, the data is further fed into feature extraction models to extract contextual and acoustic features. Finally, these features are combined by proposed MF-GNN to classify the sentiment of customer as positive, negative, or neutral. From the results, the proposed MF-GNN provided best results in accuracy (0.94), precision (0.91), recall (0.92), and F1-score (0.89), respectively, when compared with the existing Cross Modal with Bidirectional Encoder Representations from Transformers (CM-BERT).