Image-text retrieval plays a key role in linking visual and language. In recent years, fine-grained semantic alignment has become the mainstream technology in the field of image-text matching. Most current approaches rely on attention mechanisms to model the relationship between images and text, with triplet loss being commonly used to optimize the models. However, when selecting positive samples, models typically choose pairs with the highest similarity. This means that some false negative pairs, which have low similarity but still contain useful semantic information, may be ignored. If these are treated as negative samples, important semantic information could be lost. To solve this problem, we propose a Fine-grained semantic image-text retrieval model based on dual-branch attention (BAFR). The model uses a pure Transformer architecture and combines positive and negative attention mechanisms to better explore “false negative pairs” between images and texts. This allows the BAFR model to learn valuable information from these pairs, improving retrieval performance. Extensive experiments on the Flickr30k and MS-COCO datasets show that the proposed model is effective.