Pretrained models for cross-modal retrieval: experiments and improvements

Cross-modal retrieval, the process of retrieving relevant data from one modality in response to a query in another, has become increasingly important with the growing amount of multimodal data. This paper proposes using a pretrained model CLIP as the backbone of a cross-modal retrieval system and explores various methods to enhance its performance. The proposed approach reduces the output feature dimension to 384, reducing model parameters, storage capacity, and retrieval time by 62.5%. By conducting cross-training on the training dataset, the model not only enhances its intermodal invariance but also achieves multimodal retrieval. The residual connections and an increased dropout ratio of 30% increase average retrieval performance. Additionally, we propose the utilization of class proxies as missing data to accomplish training in an incomplete (imbalanced) dataset. The proposed approach is evaluated on four benchmark datasets: Wikipedia, NUS-WIDE, Pascal-Sentence, and XmediaNet, achieving 3.4%, 1.9%, 2.3%, and 5.8% retrieval performance improvement, respectively. The results demonstrate the effectiveness of the proposed approach in significantly improving the performance of cross-modal retrieval systems, outperforming state-of-the-art methods on benchmark datasets while reducing the number of model parameters, retrieval time, and database storage space.