Partial multimodal hashing method with contrastive semantic enhancement and label-guided completion
摘要
Most multimodal hashing methods assume that all modalities are available for every sample, which limits their use in practice where missing modalities are common. Existing completion-based approaches typically reconstruct the missing modality from the observed one, but their performance often drops sharply at high missing rates due to completion bias and noise. Moreover, inter-modal heterogeneity and the limited semantic expressiveness of raw text features are frequently underexplored, which further hurts retrieval accuracy. To address these issues, we propose a partial multimodal hashing method with contrastive semantic enhancement and label-guided completion. We first build a multimodal contrastive semantic-enhanced representation learning framework: BLIP-2 generates image captions to enrich textual semantics, and contrastive learning improves cross-modal alignment and caption reliability. A self-attention semantic fusion module then integrates the original text features with the image-derived semantics to strengthen textual representations. To mitigate completion bias under high missing rates, we introduce a neighbor–label joint completion mechanism. Category labels are injected as semantic priors into a label-guided adversarial generative network: a conditional generator is encouraged to produce features consistent with class semantics, while a classification discriminator improves realism and discriminability, resulting in more robust completion. Finally, we adopt a multimodal feature interaction and fusion strategy with a dual-gating mechanism to enhance cross-modal collaboration and reduce heterogeneity through contrastive constraints. Experiments on MIR Flickr and MS COCO show that the proposed method consistently outperforms 11 state-of-the-art methods, including NCH and PMH-F3, under both complete and partial-modality settings.