Traditional authentication methods such as passwords, PINs, and fingerprint recognition face increasing challenges regarding security and usability. In response to these issues, this paper presents a deep learning-based biometric recognition system that integrates both facial and vocal modalities within a unified framework. We utilize CoAtNet, a hybrid convolution-attention architecture, as a shared backbone in a Siamese setup for face and voice recognition tasks. Experiments conducted on the Kaggle Face dataset, LFW, and LibriSpeech dev-clean demonstrate that the system performs exceptionally well independently: it achieved an accuracy of 97.78% for face recognition and 97.50% for voice recognition, with an Equal Error Rate (EER) of 2.44%. A version that combines both face and voice inputs exhibits lower accuracy due to the premature merging of the two data types, highlighting the importance of careful integration of different information types. Our findings validate the effectiveness of CoAtNet for unimodal biometric verification and indicate potential directions for enhancing cross-modal alignment in future research.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Face and Voice Recognition Using a Shared CoAtNet Framework: From Unimodal to Multimodal

  • Tuan-Duc Nguyen,
  • Dinh-Dung Nguyen,
  • Anh-Tuan Nguyen,
  • Cong-Hoang Diem,
  • Duc-Tho Mai

摘要

Traditional authentication methods such as passwords, PINs, and fingerprint recognition face increasing challenges regarding security and usability. In response to these issues, this paper presents a deep learning-based biometric recognition system that integrates both facial and vocal modalities within a unified framework. We utilize CoAtNet, a hybrid convolution-attention architecture, as a shared backbone in a Siamese setup for face and voice recognition tasks. Experiments conducted on the Kaggle Face dataset, LFW, and LibriSpeech dev-clean demonstrate that the system performs exceptionally well independently: it achieved an accuracy of 97.78% for face recognition and 97.50% for voice recognition, with an Equal Error Rate (EER) of 2.44%. A version that combines both face and voice inputs exhibits lower accuracy due to the premature merging of the two data types, highlighting the importance of careful integration of different information types. Our findings validate the effectiveness of CoAtNet for unimodal biometric verification and indicate potential directions for enhancing cross-modal alignment in future research.