UMR-Net: Unified Multimodal Representation Network for Multimodal Biometric Recognition with Missing Modality
摘要
Most existing multimodal biometric recognition algorithms require test samples with complete multimodal data. However, it often encounters the problem of missing modality data and thus suffers severe performance degradation in practical scenarios. Also, most existing multimodal biometric recognition methods mainly employ multiple separate feature extraction networks for multimodal biometrics, which increased inference time and hindered the widespread employment of multimodal biometric recognition in embedded devices for autonomous systems. To this end, we proposed a Unified Multimodal Representation Network (UMR-Net) for multimodal biometric recognition based on palmvein and palmprint modalities. Firstly, an Image-Level Adaptive Fusion (ILAF) module is proposed to achieve the dynamic fusion of palmprint and palmvein images in pixel space. Secondly, a single-stream feature encoder is designed to jointly learn the unified representations of unimodal or multimodal inputs. Specifically, the palmVein-aware Feature Alignment (VFA) module and palmPrint-aware Feature Alignment (PFA) module are presented to guarantee that the multimodal biometric features contain the semantic information of each modality. Besides, a Cross-modal Semantic Consistency (CSC) loss is designed to sure that the unimodal biometric features generated by a single-stream feature encoder do not exhibit modality bias. Extensive experiments conducted on three public benchmark datasets demonstrate the effectiveness of the proposed UMR-Net.