Face and Voice Recognition Using a Shared CoAtNet Framework: From Unimodal to Multimodal
摘要
Traditional authentication methods such as passwords, PINs, and fingerprint recognition face increasing challenges regarding security and usability. In response to these issues, this paper presents a deep learning-based biometric recognition system that integrates both facial and vocal modalities within a unified framework. We utilize CoAtNet, a hybrid convolution-attention architecture, as a shared backbone in a Siamese setup for face and voice recognition tasks. Experiments conducted on the Kaggle Face dataset, LFW, and LibriSpeech dev-clean demonstrate that the system performs exceptionally well independently: it achieved an accuracy of 97.78% for face recognition and 97.50% for voice recognition, with an Equal Error Rate (EER) of 2.44%. A version that combines both face and voice inputs exhibits lower accuracy due to the premature merging of the two data types, highlighting the importance of careful integration of different information types. Our findings validate the effectiveness of CoAtNet for unimodal biometric verification and indicate potential directions for enhancing cross-modal alignment in future research.