Utilizing Quantum Particle Swarm Optimization for Multimodal Fusion of Gestures and Facial Expressions in Ensemble Conducting for Emotional Recognition

The conductor-orchestra interaction is a multimodal process, integrating channels like music, visual cues, postures, and gestures to convey artistic intent accurately. For robots, discerning human emotions from these channels enhances human-machine interaction. Current gesture recognition systems in ensembles prioritize rhythm, tempo, and dynamics; research on the emotional factors of ensembles conducting in music needs to be more extensive. We introduce the Facial Expression and Ensemble Conducting Gesture (FEGE) dataset, comprising eight distinct emotions for recognition. This article presents a Quantum Particle Swarm Optimization Algorithm (QPSO)-based parameter optimization for a Multimodal Fusion Network (QMFN) operating in a multi-feature space, aiming for emotional recognition in dual visual tasks. The network maps conduct facial expressions and gestures into a multi-feature space via dual-modality processing. It learns distinct and shared representations and decodes them using classifiers optimized through QPSO parameters. Experiments on the FEGE dataset validate our network’s efficacy. The proposed bimodal fusion network achieves an 83.17% accuracy in dual visual emotion recognition, marking about a 15% enhancement over single-modal recognition results. The proposed method can also be better applied to human-computer interaction systems for ensemble conducting training, aiming to enhance the deeper artistic intent conveyed by the most crucial emotional factors during the conducting process.