Enhancing CLIP-Based Text-Person Retrieval by Leveraging Negative Samples

Text-person retrieval (TPR) is a fine-grained cross-modal retrieval task that aims to find matching person images through detailed text descriptions. Various recent cross-modal pre-trained vision-language based models (e.g., CLIP [12]) have greatly improved the performance of text-image retrieval (TIR), but direct migration of these general models to the TPR has limited effectiveness: CLIP encodes images and text into global features separately, which makes it difficult to focus on fine-grained attributes of person, and due to the limited volume of TPR, fine-tuning CLIP may cause unstable parameter updates. In this paper, we offer a cost-effective approach to improve CLIP-based models by constructing negative samples and learning knowledge from them. Specifically, 1) replacing specific types of attribute words in the original text to generate hard negative text that do not match the semantics of the image, improving the model’s understanding of fine-grained semantic knowledge; 2) a momentum contrastive learning framework for image intra-modal is introduced to scale up the data within a batch, thus providing enough negative samples and additional positive samples to learn person identity features, ensuring the gradient stability of the CLIP-based model. The proposed method achieves good results on three existing publicly available datasets.