<p>Current cross-modal retrieval methods predominantly depend on feature extraction techniques such as convolutional neural networks (CNNs), vision transformers (ViT), and contrastive language-image pre-training (CLIP). While ViT excels at capturing long-range dependencies, it often overlooks fine-grained local features. Conversely, CNNs are effective in extracting local patterns but may fall short in constructing comprehensive global representations. Thus, we propose a hybrid network integrating ViT and channel focused network (CFN) for enhanced multimodal representation. Moreover, hashing methods are widely used in cross-modal retrieval due to the high efficiency and low storage overhead, yet suffer from cross-modal hash code heterogeneity and semantic loss during dimensionality reduction. To address these issues, we propose a feature fusion-based deep adversarial hashing (FFDAH) method, which incorporates an efficient multimodal feature fusion module that leverages a Transformer encoder for deep encoding of multimodal data to mine semantic information. Experiments on three benchmark datasets demonstrate that FFDAH outperforms state-of-the-art methods.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Research on feature fusion-based deep adversarial hashing for cross-modal retrieval

  • Xiao fei Wang,
  • Meng Wang,
  • Li ping Yang,
  • Jun Zhu

摘要

Current cross-modal retrieval methods predominantly depend on feature extraction techniques such as convolutional neural networks (CNNs), vision transformers (ViT), and contrastive language-image pre-training (CLIP). While ViT excels at capturing long-range dependencies, it often overlooks fine-grained local features. Conversely, CNNs are effective in extracting local patterns but may fall short in constructing comprehensive global representations. Thus, we propose a hybrid network integrating ViT and channel focused network (CFN) for enhanced multimodal representation. Moreover, hashing methods are widely used in cross-modal retrieval due to the high efficiency and low storage overhead, yet suffer from cross-modal hash code heterogeneity and semantic loss during dimensionality reduction. To address these issues, we propose a feature fusion-based deep adversarial hashing (FFDAH) method, which incorporates an efficient multimodal feature fusion module that leverages a Transformer encoder for deep encoding of multimodal data to mine semantic information. Experiments on three benchmark datasets demonstrate that FFDAH outperforms state-of-the-art methods.