Current cross-modal retrieval models predominantly rely on one-to-one image-text pairs for training, with most approaches projecting each sample into a single embedding vector. However, recent studies indicate that this single-vector representation fails to capture the inherent complexity of images and texts, resulting in the loss of critical semantic information during the embedding process. Consequently, these models often struggle to effectively learn the nuanced features of both modalities. In this paper, we propose a novel training strategy that transitions from traditional one-to-one image-text pairing to a one-to-many framework and introduce an innovative Set Prediction Module with Weight to better capture the diverse semantics of the input data. By incorporating a more diverse set of textual representations during training, our method significantly enhances the performance of cross-modal retrieval models. Extensive evaluations across various backbone networks on the COCO and Flickr30K datasets demonstrate that our approach consistently outperforms most existing methods.