The intersection of language and visual perception plays a crucial role in advancing artificial intelligence, particularly in the field of Compositional Image Retrieval (CIR). CIR involves retrieving a target image based on a query composed of a reference image and a textual modification that specifies desired changes. This task is especially valuable in domains such as fashion, where users may search for items by specifying modifications like color, pattern, or style. In this paper, we propose a deep learning-based CIR framework that integrates ResNet-50 as an image encoder and BERT as a text encoder to extract semantically rich features from both modalities. These features are fused using element-wise multiplication and further transformed using linear and convolutional mappings. The combined representation is optimized using a triplet loss function to enhance retrieval accuracy. The proposed model is evaluated on the Fashion200k dataset and demonstrates a +2.1 improvement in Recall@50 over the TIRG baseline [9]. We also enhance baseline models for a fair comparison. Beyond the fashion domain, our approach generalizes well to other CIR tasks, offering a robust and adaptable solution for multimodal information retrieval.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Bridging Vision and Language: A Deep Learning-Based Approach for Compositional Image Retrieval

  • Padmashree Desai,
  • G. R. Sampreeti

摘要

The intersection of language and visual perception plays a crucial role in advancing artificial intelligence, particularly in the field of Compositional Image Retrieval (CIR). CIR involves retrieving a target image based on a query composed of a reference image and a textual modification that specifies desired changes. This task is especially valuable in domains such as fashion, where users may search for items by specifying modifications like color, pattern, or style. In this paper, we propose a deep learning-based CIR framework that integrates ResNet-50 as an image encoder and BERT as a text encoder to extract semantically rich features from both modalities. These features are fused using element-wise multiplication and further transformed using linear and convolutional mappings. The combined representation is optimized using a triplet loss function to enhance retrieval accuracy. The proposed model is evaluated on the Fashion200k dataset and demonstrates a +2.1 improvement in Recall@50 over the TIRG baseline [9]. We also enhance baseline models for a fair comparison. Beyond the fashion domain, our approach generalizes well to other CIR tasks, offering a robust and adaptable solution for multimodal information retrieval.