Complementary expert balanced learning for long-tail cross-modal retrieval

Cross-modal retrieval aims to project the high-dimensional cross-model data to a common low-dimensional space. Previous work relies on balanced dataset for training. But with the growth of massive real datasets, the long-tail phenomenon has be found in more and more datasets and how to train with those imbalanced datasets is becoming an emerging challenge. In this paper, we propose the complementary expert balanced learning for long-tail cross-modal retrieval to alleviate the impact of long-tail data. In the solution, we design a multiple experts complementary to balance the difference between image and text modalities. Separately for each expert, to find the common feature space of images and texts, we design an individual pairs loss. Moreover, a balancing process is proposed to mitigate the impact of the long tail on the retrieval accuracy of each expert network. In addition, we propose complementary online distillation to enable collaborative operation between individual experts and improve image and text matching. Each expert allows mutual learning between individual modalities, and multiple experts can complement each other to learn the feature embedding between two modalities. Finally, to address the reduction in the number of data after long-tail processing, we propose high-score retraining which also helps the network capture global and robust features with meticulous discrimination. Experimental results on widely used benchmark datasets show that the proposed method is effective in long-tail cross-modal learning.