Dual-branch networks for privacy-preserving cross-modal retrieval in cloud computing

Cross-modal retrieval can break through the limitations of modalities and carry out information retrieval across data of different modalities to meet the needs of users in obtaining multi-modal correlation retrieval. Cloud computing has the advantages of high efficiency and low cost, but data security hinders its development. While cloud computing offers high efficiency and cost-effectiveness, concerns surrounding data security impede its full potential. Privacy-preserving cross-modal retrieval emerges as a viable solution, catering to users’ demands for efficient retrieval while safeguarding data confidentiality. However, a major challenge still exists in this field: how to bridge the inherent semantic gap within heterogeneous and chaotic information. To address this challenge, this paper proposes dual-branch networks for privacy-preserving cross-modal retrieval in cloud computing. Firstly, a dual-branch feature extraction network of encrypted image-text is constructed, enhancing the extraction of meaningful features from encrypted data. Secondly, a cross-modal alignment method is designed to eliminate the heterogeneous gap between different modalities through the alignment within and between modalities. Finally, to fully exploit the storage and computing advantages of cloud computing, both encrypted data and the cross-modal feature extractor are deployed to the cloud. Leveraging the dynamic update capabilities of cloud-stored encrypted data enables continuous model refinement, enhancing retrieval accuracy while reducing the storage and computational burdens on data owners. Extensive experiments conducted on the publicly available benchmark image-text dataset Wikipedia indicate that, compared to existing methods, our approach achieves improvements of 5.4%, 1%, 1.6%, and 20.1% in the four metrics of image-to-text (i2t), text-to-image (t2i), image-to-all (i2all), and text-to-all (t2all), respectively.