CAMIR: fine-tuning CLIP and multi-head cross-attention mechanism for multimodal image retrieval with sketch and text features

Sketches and texts are two input modes of queries that are widely used in image retrieval tasks of different granularities. Text-based image retrieval (TBIR) is mainly used for coarse-grained retrieval, while sketch-based image retrieval (SBIR) aims to retrieve images based on hand-drawn sketches, which pose unique challenges due to the abstract nature of sketches. Existing methods mainly focus on retrieval based on a single modality but fail to explore the connections between multiple modalities comprehensively. In addition, the emerging contrastive language image pre-training (CLIP) model and powerful contrastive learning methods are underexplored in this field. We propose a novel multimodal image retrieval framework (CAMIR) to address these challenges. It obtains sketch and text features through a fine-tuned CLIP model, fuses the extracted features using multi-head cross-attention, and combines contrastive learning for retrieval tasks. In the indexing stage, we introduce Faiss, an open-source similarity search library developed by Meta AI Research, to enhance retrieval efficiency. Comprehensive experiments on the benchmark dataset Sketchy demonstrate the effectiveness of our proposed framework, achieving superior performance compared to existing methods while highlighting the potential of integrating sketch and text features for retrieval tasks.