Existing cross-modal retrieval methods often exhibit deficiencies in handling feature correlation and fine-grained relationship descriptions, leading to inaccurate modality similarity measurements. Moreover, there is a lack of in-depth exploration of vision-language knowledge distillation schemes, failing to effectively utilize multimodal knowledge to guide the learning of student networks. In order to address these issues, this paper proposes a graph-enhanced cross-modal retrieval method based on vision-language knowledge distillation (GECM-KD). This method first extracts features through CLIP and utilizes Graph Attention Networks to extract more fine-grained information from similar data nodes. By combining fine-grained features of images and text, and utilizing knowledge distillation techniques, rich semantic information is transferred from the teacher model to the student model. Additionally, a framework for vision-language knowledge distillation is designed to effectively extract detailed multimodal semantic information from the model, which is then leveraged to optimize the student network. Results from multiple datasets confirm that our approach greatly enhances both retrieval accuracy and efficiency in diverse cross-modal retrieval scenarios.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Graph Enhanced Cross-Modal Retrieval Based on Visual-Language Knowledge Distillation

  • Yibin Leng,
  • Huaxiang Zhang,
  • Li Liu,
  • Dongmei Liu

摘要

Existing cross-modal retrieval methods often exhibit deficiencies in handling feature correlation and fine-grained relationship descriptions, leading to inaccurate modality similarity measurements. Moreover, there is a lack of in-depth exploration of vision-language knowledge distillation schemes, failing to effectively utilize multimodal knowledge to guide the learning of student networks. In order to address these issues, this paper proposes a graph-enhanced cross-modal retrieval method based on vision-language knowledge distillation (GECM-KD). This method first extracts features through CLIP and utilizes Graph Attention Networks to extract more fine-grained information from similar data nodes. By combining fine-grained features of images and text, and utilizing knowledge distillation techniques, rich semantic information is transferred from the teacher model to the student model. Additionally, a framework for vision-language knowledge distillation is designed to effectively extract detailed multimodal semantic information from the model, which is then leveraged to optimize the student network. Results from multiple datasets confirm that our approach greatly enhances both retrieval accuracy and efficiency in diverse cross-modal retrieval scenarios.