Multimodal Personality Trait Recognition via Spatiotemporal Modeling and Dual-Stage Fusion
摘要
Personality trait recognition aims to infer an individual’s characteristics across the Big Five dimensions from multimodal data and has become a key research topic in artificial intelligence and social behavior modeling. Although progress has been made, most existing methods still separate spatial and temporal modeling, which limits their ability to capture the dynamic evolution of local image regions over time. Moreover, current studies mainly focus on modality-level fusion but lack collaborative mechanisms to address representational differences across visual perspectives and modalities. To overcome these challenges, we propose the Spatial Temporal Attentive Guided Network (STAG-Net), a hybrid framework that jointly models personality expression from visual and audio modalities. In the visual stream, VisionLSTM unifies intra-frame spatial detail modeling and inter-frame temporal dynamics, allowing the network to capture region-wise temporal evolution. To further strengthen spatial representation, we introduce the Multi-Scale Spatial Context Module (MSCM), which enhances multi-scale spatial context aggregation. For the audio stream, high-level embeddings are extracted using a pre-trained VGGish model. In the fusion stage, we design the Dual-View Feature Integration Module (DFIM) to align facial and scene features in the early stage, ensuring consistency across visual perspectives. To address the modality gap between vision and audio, we further propose the Cross-Modal Guided Fusion Module (CGFM), which employs a multi-head attention mechanism for cross-modal alignment. Together, DFIM and CGFM form a two-stage fusion strategy that enables both intra-modal and inter-modal integration. These results suggest that STAG-Net is a viable approach for personality trait modeling, offering a direction for advancing multimodal recognition methods.