Semantic-alignment transformer and adversary hashing for cross-modal retrieval

Deep Cross-Modal Hashing (DCMH) has garnered significant attention in the field of cross-modal retrieval due to its advantages such as high computational efficiency and small storage space. However, existing DCMH methods still face certain limitations: (1) they neglect the correlation between labels, while label features exhibit high sparsity; (2) they lack fine-grained semantic alignment; (3) they fail to effectively address data imbalance. In order to tackle these issues, this paper introduces a framework named Semantic-Alignment Transformer and Adversary Hashing for Cross-modal Retrieval (SATAH). To the best of our knowledge, this is the first attempt at the Semantic-Alignment Transformer algorithm. Specifically, this paper first designs a label learning network that utilizes a crafted transformer module to extract label information, guiding adversarial learning and hash function learning accordingly. Subsequently, a Balanced Conditional Generative Adversarial Network (BCGAN) is constructed, marking the first instance of adversarial training guided by label information. Furthermore, a Weighted Semi-Hard Cosine Triplet Constraint is proposed to better ensure high-ranking similarity relationships among all items. Lastly, considering the correlation between labels, a semantic-alignment constraint is introduced to handle label correlation from a fine-grained perspective, capturing similarity on a global scale more effectively. Extensive experiments are conducted on multiple representative cross-modal datasets. In experiments with 64-bit hash code length, SATAH achieves average mAP values of 84.75%, 68.87%, and 68.73% on MIR Flickr, NUS-WIDE, and MS COCO datasets, respectively, outperforming state-of-the-art methods. The code is available at https://github.com/Daydaylight/SATAH.