Real-time magnetic resonance imaging (rtMRI) enables detailed visualization of articulatory structures during speech production, making it invaluable for analysing articulatory-phonological features and advancing clinical speech technologies. While MRI captures the anatomical dynamics of articulation, concurrent audio signals provide complementary acoustic information that enhances temporal resolution in speech processing. Nevertheless, deriving meaningful phonological representations from rtMRI data remains difficult when audio signals are unavailable – situations that commonly arise during MRI scanning due to acoustic noise interference or in cases involving speech disorders such as those seen in glossectomy patients. To address this limitation, we propose a contrastive learning framework for automatically classifying three fundamental articulatory dimensions from MRI data: manner of articulation, place of articulation, and voicing. During training, paired MRI frames and speech segments are encoded separately using vision transformer (ViT) and Wav2Vec2 architectures, respectively, with contrastive learning employed to maximize cross-modal alignment between visual and acoustic representations. Critically, only MRI data is required during inference, enabling phonological classification without audio input. We evaluated four experimental configurations on the USC-TIMIT dataset: unimodal rtMRI, unimodal audio, multimodal middle fusion, and our contrastive learning-based approach. Results show that contrastive learning achieves state-of-the-art performance with an average F1-score of 0.81 across 15 phonological classes, representing absolute improvements of 0.23 over the unimodal baseline and 0.09 over multimodal fusion, thereby confirming the efficacy of cross-modal contrastive representation learning for MRI-based articulatory analysis when audio signals are unavailable [1].

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Abstract: Audio-vision Contrastive Learning for Phonological Class Recognition

  • Daiqi Liu,
  • Tomás Arias-Vergara,
  • Jana Hutter,
  • Andreas Maier,
  • Paula A. Pérez-Toro

摘要

Real-time magnetic resonance imaging (rtMRI) enables detailed visualization of articulatory structures during speech production, making it invaluable for analysing articulatory-phonological features and advancing clinical speech technologies. While MRI captures the anatomical dynamics of articulation, concurrent audio signals provide complementary acoustic information that enhances temporal resolution in speech processing. Nevertheless, deriving meaningful phonological representations from rtMRI data remains difficult when audio signals are unavailable – situations that commonly arise during MRI scanning due to acoustic noise interference or in cases involving speech disorders such as those seen in glossectomy patients. To address this limitation, we propose a contrastive learning framework for automatically classifying three fundamental articulatory dimensions from MRI data: manner of articulation, place of articulation, and voicing. During training, paired MRI frames and speech segments are encoded separately using vision transformer (ViT) and Wav2Vec2 architectures, respectively, with contrastive learning employed to maximize cross-modal alignment between visual and acoustic representations. Critically, only MRI data is required during inference, enabling phonological classification without audio input. We evaluated four experimental configurations on the USC-TIMIT dataset: unimodal rtMRI, unimodal audio, multimodal middle fusion, and our contrastive learning-based approach. Results show that contrastive learning achieves state-of-the-art performance with an average F1-score of 0.81 across 15 phonological classes, representing absolute improvements of 0.23 over the unimodal baseline and 0.09 over multimodal fusion, thereby confirming the efficacy of cross-modal contrastive representation learning for MRI-based articulatory analysis when audio signals are unavailable [1].