A deep multimodal fusion method for personality traits prediction

Personality traits influence an individual’s behavior, preferences and decision-making processes, making automated personality recognition an important area of research. In this paper, we propose a novel deep multimodal fusion for predicting personality traits from diverse data modalities, including text, audio, and visual inputs. Our proposed method extracts complex patterns and features from these multimodal data sources using advanced deep learning methods including Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks, and Vision Transformer (ViT). Specifically, we use pre-trained models ViT-B16 and VGG16 for visual feature extraction, VGGish for audio feature extraction, and GloVe for text analysis. Additionally, we investigate the potential benefits of using self-attention and cross-attention mechanisms to provide accurate predictions regarding an individual’s personality traits. The method we propose combines information from several modalities using various fusion techniques, improving the predictive capability of the model. Experimental results using the publicly available ChaLearn First Impressions-V2 dataset demonstrate that our technique is effective, achieving higher accuracy than the current literature. This work contributes to the advancement of multimodal deep learning techniques and provides valuable results in the field of personality recognition.