Personalized emotion analysis based on fuzzy multi-modal transformer model

Analyzing and detecting human intensions and emotions are important means to improve the communication between users and machines in the areas of human-computer interaction (HCI) and human-robot interaction (HRI). Despite significant progress in utilizing state-of-the-art (SOTA) Transformer-based models, various obstacles persist in managing complicated input interdependencies and extracting intricate contextual semantics. Moreover, it lacks practical applicability and struggles to accurately capture and effectively manage the inherent complexity and unpredictability of human emotions. In recognition of the identified research gaps, we introduce a robust and innovative fuzzy multi-modal Transformer (FMMT) model. Our novel fuzzy Transformer model uniquely heightens the comprehension of emotional contexts by concurrently analyzing audio, visual, and text data through three distinct branches. By incorporating fuzzy mathematic theory and introducing a unique temporal embedding technique to trace the evolution of emotional states, it effectively handles the inherent uncertainty in human emotions, thereby filling a significant void in emotional AI. Building upon the FMMT model, we further explored the emotion expression approach. Furthermore, performance comparison analysis with SOTA baseline methods and detailed ablation study were performed. The results show that the proposed FMMT performs better than the baseline methods. Finally, we conducted detailed experimental verification and empirical analyses of the practicality of the designed method by verifying uncertainty emotion and analyzing emotional state transitions combined with personalized factor. Overall, our research makes a significant contribution to emotion analysis through the implementation of a novel fuzzy Transformer model. This model enhances emotion perception and advances the methods for analyzing emotional expression, thus setting an edge over prior studies.