The growing demand for multimodal image-text retrieval in e-commerce presents significant challenges in both retrieval accuracy and user privacy. To address these issues, we propose MF-ETR, a privacy-enhanced retrieval model tailored for Chinese e-commerce platforms. MF-ETR incorporates Local Differential Privacy (LDP) to protect sensitive image and text features during transmission and processing, ensuring strong privacy guarantees without degrading performance. The model employs CN-CLIP-based visual encoders and RoBERTa-wwm-ext for textual representation, and introduces two novel attention mechanisms: combined attention, which captures intra-modal semantics via self-attention, and joint attention, which enhances cross-modal interactions through compositional learning. These enhancements substantially improve retrieval effectiveness. Experimental results on benchmark datasets show that MF-ETR with joint attention outperforms its merged-attention variant by 1.9% in text-to-image and 1.0% in image-to-text retrieval (MeanR), while maintaining robust privacy protection. These findings demonstrate that MF-ETR achieves a strong balance between privacy preservation and retrieval performance, making it a promising solution for secure multimodal applications in e-commerce.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Privacy-Enhanced Multi-modal Feature Fusion for E-Commerce: A Novel Approach to Image and Text Retrieval

  • Yanrong Zhang,
  • Bin Chen,
  • Chengxiang Zhu,
  • Chi Xu,
  • Jiale Liang

摘要

The growing demand for multimodal image-text retrieval in e-commerce presents significant challenges in both retrieval accuracy and user privacy. To address these issues, we propose MF-ETR, a privacy-enhanced retrieval model tailored for Chinese e-commerce platforms. MF-ETR incorporates Local Differential Privacy (LDP) to protect sensitive image and text features during transmission and processing, ensuring strong privacy guarantees without degrading performance. The model employs CN-CLIP-based visual encoders and RoBERTa-wwm-ext for textual representation, and introduces two novel attention mechanisms: combined attention, which captures intra-modal semantics via self-attention, and joint attention, which enhances cross-modal interactions through compositional learning. These enhancements substantially improve retrieval effectiveness. Experimental results on benchmark datasets show that MF-ETR with joint attention outperforms its merged-attention variant by 1.9% in text-to-image and 1.0% in image-to-text retrieval (MeanR), while maintaining robust privacy protection. These findings demonstrate that MF-ETR achieves a strong balance between privacy preservation and retrieval performance, making it a promising solution for secure multimodal applications in e-commerce.