Modal-Aware Prompting with Missing Modalities for Biosignal-Based Emotion Recognition
摘要
Biosignal-based multimodal emotion recognition with missing modalities has attracted considerable interest in real-world applications. Recently, prompt learning has emerged as a promising approach introducing learnable prompts that make the model adapted to missing modalities through fine-tuning. However, existing prompt-based methods generate prompts solely for different missing cases. This leads to an exponential increase in the number of prompts and a reduction in accuracy. In this paper, we propose a modal-aware prompting method for biosignal-based emotion recognition with missing modalities. Specifically, rather than randomly generating one-hot vectors as prompts for specific missing cases, we extract fewer prompts through modality-specific subtasks. These prompts from subtasks enhance the precision of emotion recognition by obtaining information of the missing cases and emotion classes simultaneously. Additionally, we leverage inter-modality similarity to align geometric distribution of features, improving robustness of the fused features against missing modalities. Supported by extensive experiments, our method enhances the performance of multimodal emotion recognition with missing modalities.