Enhancing Image Retrieval Through Contrastive Learning With Dual Queries
摘要
In this paper we propose a multimodal image retrieval framework that takes advantage of contrastive learning through the combination of CLIP (ViT-B/32) and BLIP. The proposed approach enriches the original input-text query with BLIP to provide more contextually rich and semantically interpretable representations. It then maps these back into a shared feature space with image embeddings from CLIP to enable good cross-modal alignment. Cosine similarity between the query consisting of the concatenated text information and the image information and the image embeddings of the database is calculated for the best matching results. We evaluate the performance of the system in the Fashion IQ dataset with Recall@K measures, obtaining large improvements over the previous baselines. Experimental results show that the boosted text generated through BLIP leads to better accuracy of retrieving relevant content as well as better contextual coherence.