In this paper we propose a multimodal image retrieval framework that takes advantage of contrastive learning through the combination of CLIP (ViT-B/32) and BLIP. The proposed approach enriches the original input-text query with BLIP to provide more contextually rich and semantically interpretable representations. It then maps these back into a shared feature space with image embeddings from CLIP to enable good cross-modal alignment. Cosine similarity between the query consisting of the concatenated text information and the image information and the image embeddings of the database is calculated for the best matching results. We evaluate the performance of the system in the Fashion IQ dataset with Recall@K measures, obtaining large improvements over the previous baselines. Experimental results show that the boosted text generated through BLIP leads to better accuracy of retrieving relevant content as well as better contextual coherence.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Enhancing Image Retrieval Through Contrastive Learning With Dual Queries

  • Pradeep Shet,
  • Sujatha C,
  • Vishwanath P. Baligar

摘要

In this paper we propose a multimodal image retrieval framework that takes advantage of contrastive learning through the combination of CLIP (ViT-B/32) and BLIP. The proposed approach enriches the original input-text query with BLIP to provide more contextually rich and semantically interpretable representations. It then maps these back into a shared feature space with image embeddings from CLIP to enable good cross-modal alignment. Cosine similarity between the query consisting of the concatenated text information and the image information and the image embeddings of the database is calculated for the best matching results. We evaluate the performance of the system in the Fashion IQ dataset with Recall@K measures, obtaining large improvements over the previous baselines. Experimental results show that the boosted text generated through BLIP leads to better accuracy of retrieving relevant content as well as better contextual coherence.