Scholarlink

<p>Due to the increasing security requirements of portable devices, the audio-face cross-modal speaker verification task has gained significant attention. To enhance the robustness of audio-face fusion methods under challenging conditions, this paper proposes a novel audio-face fusion speaker verification method. This method comprises four components: an audio-net, a face-net, a cross-modal fusion strategy, and a metric learning mechanism. The audio-net employs convolution neural network–transformer (CNN-Transformer) architecture, enabling the interactive utilization of global and local feature information. The cross-modal fusion strategy incorporates an adaptive feature fusion mechanism that dynamically adjusts the information flow distribution across modalities. The metric learning mechanism adopts an enhanced adaptive angular margin loss function, which dynamically modulates the gradient of the loss. The proposed method was trained on the Zhvoice, Voxceleb1, and Cnceleb_v2 speech datasets and the CASIA-WebFace face dataset, and evaluated on their corresponding test sets. Relative to the audio-facce direct fusion model ECAPA-TDNN+ResNet50, the proposed model achieves EER improvements of 87.8%, 53.6%, and 28.8%, respectively, and MinDCF improvements of 72.7%, 19.7%, and 37.8%, respectively. This approach demonstrates considerable practical utility for advancing audio-face cross-modal application technology in the domain of personal identification.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Audio-face fusion of multi-methods for speaker verification in strong interference

  • 期刊论文

  • 发表日期：

    2026年3月25日

查看全文
  • Xuan Zhang,
  • Jun Tang,
  • Xiaochen Liu,
  • Huijun Zhao,
  • Chenguang Wang,
  • Chong Shen,
  • Jun Liu

摘要

Due to the increasing security requirements of portable devices, the audio-face cross-modal speaker verification task has gained significant attention. To enhance the robustness of audio-face fusion methods under challenging conditions, this paper proposes a novel audio-face fusion speaker verification method. This method comprises four components: an audio-net, a face-net, a cross-modal fusion strategy, and a metric learning mechanism. The audio-net employs convolution neural network–transformer (CNN-Transformer) architecture, enabling the interactive utilization of global and local feature information. The cross-modal fusion strategy incorporates an adaptive feature fusion mechanism that dynamically adjusts the information flow distribution across modalities. The metric learning mechanism adopts an enhanced adaptive angular margin loss function, which dynamically modulates the gradient of the loss. The proposed method was trained on the Zhvoice, Voxceleb1, and Cnceleb_v2 speech datasets and the CASIA-WebFace face dataset, and evaluated on their corresponding test sets. Relative to the audio-facce direct fusion model ECAPA-TDNN+ResNet50, the proposed model achieves EER improvements of 87.8%, 53.6%, and 28.8%, respectively, and MinDCF improvements of 72.7%, 19.7%, and 37.8%, respectively. This approach demonstrates considerable practical utility for advancing audio-face cross-modal application technology in the domain of personal identification.