Biosignal-based multimodal emotion recognition with missing modalities has attracted considerable interest in real-world applications. Recently, prompt learning has emerged as a promising approach introducing learnable prompts that make the model adapted to missing modalities through fine-tuning. However, existing prompt-based methods generate prompts solely for different missing cases. This leads to an exponential increase in the number of prompts and a reduction in accuracy. In this paper, we propose a modal-aware prompting method for biosignal-based emotion recognition with missing modalities. Specifically, rather than randomly generating one-hot vectors as prompts for specific missing cases, we extract fewer prompts through modality-specific subtasks. These prompts from subtasks enhance the precision of emotion recognition by obtaining information of the missing cases and emotion classes simultaneously. Additionally, we leverage inter-modality similarity to align geometric distribution of features, improving robustness of the fused features against missing modalities. Supported by extensive experiments, our method enhances the performance of multimodal emotion recognition with missing modalities.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Modal-Aware Prompting with Missing Modalities for Biosignal-Based Emotion Recognition

  • Hongyu Jiang,
  • Wenqing Ji,
  • Xi Chen,
  • Yalan Ye,
  • Hengtan Zhang

摘要

Biosignal-based multimodal emotion recognition with missing modalities has attracted considerable interest in real-world applications. Recently, prompt learning has emerged as a promising approach introducing learnable prompts that make the model adapted to missing modalities through fine-tuning. However, existing prompt-based methods generate prompts solely for different missing cases. This leads to an exponential increase in the number of prompts and a reduction in accuracy. In this paper, we propose a modal-aware prompting method for biosignal-based emotion recognition with missing modalities. Specifically, rather than randomly generating one-hot vectors as prompts for specific missing cases, we extract fewer prompts through modality-specific subtasks. These prompts from subtasks enhance the precision of emotion recognition by obtaining information of the missing cases and emotion classes simultaneously. Additionally, we leverage inter-modality similarity to align geometric distribution of features, improving robustness of the fused features against missing modalities. Supported by extensive experiments, our method enhances the performance of multimodal emotion recognition with missing modalities.