Image-text retrieval aims to retrieve semantically related samples in a different modality given text/image as input. Previous works focus on developing cross-model interaction for the local relationship of the image-text pair to determine the matching. We argue that the paradigm overlooks the global relationship with other candidate texts. The limited information of a single sentence may fails to link sentences and images, and the lack of reference and comparison in globalness leads to the modeling being dominated by the local false signal. We propose to include contextual texts and images to enhance the semantic modeling of an image-text pair. Motivated by the success of graph model for neighbourhood information modeling, we build a cross-modal graph and formulate image-text retrieval task as link prediction on the graph. We add textual concepts beyond image and text in the cross-modal graph to better clarify the global relationship. In the link prediction for matching determination, we propose PAth based Transformer (PAT) or the simultaneous modeling of both semantics and structure. In experiments, we verify the effectiveness of our model on MS-COCO and Flickr30K compared with current competitive pre-trained models.