In the field of Automatic Speaker Identification (ASI), relying exclusively on the speech modality introduces significant challenges, especially when faced with adverse recording conditions such as noise, reverberation, and emotional fluctuations. Audio-visual datasets enable the development of multimodal approaches, allowing the design of robust solutions for identification. This paper introduces Cross-Modal Interaction Speaker Identification (CoMISI), a multimodal approach that fuses visual and audio modalities through a novel Cross-Modal Interaction Fusion (CMIF) module. The proposed approach enhances embeddings from both modalities and leverages deep cross-modal interactions to improve feature representation. To evaluate the performance of the proposed approach in diverse acoustic and visual conditions, we used the GRID audio-visual dataset for neutral speech speaker identification and the RAVDESS audio-visual dataset for emotional speech speaker identification. For a more comprehensive assessment under challenging conditions, both datasets were augmented with noise, reverberation, and visual distortions. Extensive experiments conducted on both the original and augmented versions of the GRID and RAVDESS datasets demonstrate that the proposed approach significantly outperforms state-of-the-art methods. The code for the proposed approach and the dataset generation scripts are publicly available.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

CoMISI: Multimodal Speaker Identification in Diverse Audio-Visual Conditions Through Cross-Modal Interaction

  • Yassin Terraf,
  • Youssef Iraqi

摘要

In the field of Automatic Speaker Identification (ASI), relying exclusively on the speech modality introduces significant challenges, especially when faced with adverse recording conditions such as noise, reverberation, and emotional fluctuations. Audio-visual datasets enable the development of multimodal approaches, allowing the design of robust solutions for identification. This paper introduces Cross-Modal Interaction Speaker Identification (CoMISI), a multimodal approach that fuses visual and audio modalities through a novel Cross-Modal Interaction Fusion (CMIF) module. The proposed approach enhances embeddings from both modalities and leverages deep cross-modal interactions to improve feature representation. To evaluate the performance of the proposed approach in diverse acoustic and visual conditions, we used the GRID audio-visual dataset for neutral speech speaker identification and the RAVDESS audio-visual dataset for emotional speech speaker identification. For a more comprehensive assessment under challenging conditions, both datasets were augmented with noise, reverberation, and visual distortions. Extensive experiments conducted on both the original and augmented versions of the GRID and RAVDESS datasets demonstrate that the proposed approach significantly outperforms state-of-the-art methods. The code for the proposed approach and the dataset generation scripts are publicly available.