Cross-modal quaternion relation mining and gap bridging for image-text retrieval
摘要
Image-text retrieval, which aims to retrieve one modality (i.e., image or text) given a query from a different modality (i.e., text or image) remains a key and challenging task in many information retrieval applications. For this challenge, more and more researchers are focusing on applying visual language pre-training models to obtain robust features for better retrieval accuracy. However, these methods have two limitations: 1) They only focus on mining the remaining truly hard negative instances, while ignore the violations on the performance of the image-text retrieval caused by false negative instances; 2) They can’t effectively solve the problem of modality gap in image-text data only by simply calculating the semantic similarity of pairs of instances through the loss function. To address these limitations, we propose a cross-modal quaternion relation mining and gap bridging method, called QRMGB. QRMGB incorporates a quaternion relation mining (QRM) component that reduces the impact of false negative instances on image-text retrieval by mining those that have strong semantic associations with positive instances. Specifically, the QRM component constructs quaternions and proposes a novel quaternion loss function called QuadrFN loss to effectively mine relations for image-text data. Additionally, we propose a Gaussian noise bridging (GNB) component to mitigate the modality gap in image-text data by adding Gaussian noise to the text features. Furthermore, GNB leverages a gap cosine loss function to maximize the similarity between image features and noisy text features, which distribute text features to overlap with images. Finally, we evaluate QRMGB on four widely used image-text datasets and show that it significantly outperforms existing state-of-the-art methods for image-text retrieval applications.