<p>Most multimodal hashing methods assume that all modalities are available for every sample, which limits their use in practice where missing modalities are common. Existing completion-based approaches typically reconstruct the missing modality from the observed one, but their performance often drops sharply at high missing rates due to completion bias and noise. Moreover, inter-modal heterogeneity and the limited semantic expressiveness of raw text features are frequently underexplored, which further hurts retrieval accuracy. To address these issues, we propose a partial multimodal hashing method with contrastive semantic enhancement and label-guided completion. We first build a multimodal contrastive semantic-enhanced representation learning framework: BLIP-2 generates image captions to enrich textual semantics, and contrastive learning improves cross-modal alignment and caption reliability. A self-attention semantic fusion module then integrates the original text features with the image-derived semantics to strengthen textual representations. To mitigate completion bias under high missing rates, we introduce a neighbor–label joint completion mechanism. Category labels are injected as semantic priors into a label-guided adversarial generative network: a conditional generator is encouraged to produce features consistent with class semantics, while a classification discriminator improves realism and discriminability, resulting in more robust completion. Finally, we adopt a multimodal feature interaction and fusion strategy with a dual-gating mechanism to enhance cross-modal collaboration and reduce heterogeneity through contrastive constraints. Experiments on MIR Flickr and MS COCO show that the proposed method consistently outperforms 11 state-of-the-art methods, including NCH and PMH-F<sup>3</sup>, under both complete and partial-modality settings.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Partial multimodal hashing method with contrastive semantic enhancement and label-guided completion

  • Hairong Wang,
  • Zhenye Yang,
  • Zhongxuan Li,
  • Jing Wang,
  • Hongying Li

摘要

Most multimodal hashing methods assume that all modalities are available for every sample, which limits their use in practice where missing modalities are common. Existing completion-based approaches typically reconstruct the missing modality from the observed one, but their performance often drops sharply at high missing rates due to completion bias and noise. Moreover, inter-modal heterogeneity and the limited semantic expressiveness of raw text features are frequently underexplored, which further hurts retrieval accuracy. To address these issues, we propose a partial multimodal hashing method with contrastive semantic enhancement and label-guided completion. We first build a multimodal contrastive semantic-enhanced representation learning framework: BLIP-2 generates image captions to enrich textual semantics, and contrastive learning improves cross-modal alignment and caption reliability. A self-attention semantic fusion module then integrates the original text features with the image-derived semantics to strengthen textual representations. To mitigate completion bias under high missing rates, we introduce a neighbor–label joint completion mechanism. Category labels are injected as semantic priors into a label-guided adversarial generative network: a conditional generator is encouraged to produce features consistent with class semantics, while a classification discriminator improves realism and discriminability, resulting in more robust completion. Finally, we adopt a multimodal feature interaction and fusion strategy with a dual-gating mechanism to enhance cross-modal collaboration and reduce heterogeneity through contrastive constraints. Experiments on MIR Flickr and MS COCO show that the proposed method consistently outperforms 11 state-of-the-art methods, including NCH and PMH-F3, under both complete and partial-modality settings.