SHAF: Semantic-Guided Hierarchical Alignment and Fusion for Composed Image Retrieval

Composed image retrieval, aimed at enhancing user searches by accurately capturing intent, involves semantically aligning and fusing image and text features. We propose the Semantic-guided Hierarchical Alignment and Fusion network (SHAF), which is specifically designed to combine information from both visual and textual modalities across various network layers. SHAF employs attention mechanisms to progressively align text and image features from low to high levels, effectively bridging the semantic gap between these modalities. The network integrates complementary information from images and text fragments in the query through dynamic weight allocation and feature enhancement mechanisms. This process generates a composite feature within a unified embedding space. Extensive experiments on the FashionIQ and Shoes (+7.15 and +7.58 in R@10) datasets show that SHAF performs better than the state-of-the-art models in composed image retrieval tasks. The code is publicly available on GitHub: https://github.com/Maserhe/SHAF .