Multi-modal Personality Trait Detection: Integrating Facial, Vocal, and Demographic Features for Big Five Personality Prediction
摘要
To estimate the Big Five personality traits (Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism), this work combines face expressions, voice analysis, and demographic data in a novel multi-modal technique. Our system analyses facial features from video frames using a three-pronged approach: (1) a vision-based model using VGG16 as a feature extractor; (2) speech and text analysis using BERT-based models; and (3) demographic analysis including age and gender data using Random Forest Regression. We evaluate our approach using the VPTD dataset and show that over single-modality approaches, integration of numerous modalities significantly raises prediction accuracy. With a test loss of 0.00011, our visual model exhibits good prediction ability. We also use real-time stress detection with eyebrow movement tracking and emotion identification to provide a whole personality evaluation framework. Experimental evidence shows that our multi-modal approach outperforms single-modality baselines, so the decision-level fusion of several modalities produces more accurate and strong personality predictions than any single component. This study develops the growing topic of computational personality evaluation by offering a full, real-time personality detection system with applications in psychology, human-computer interaction, and personalised services.