To narrow down the gap between the image and text modalities, computing their similarity is of the utmost importance. This paper employs a novel, comprehensive system for determining the similarity between text and image using a contrastive learning approach. The integration of state-of-the-art models (YOLO, Vision Transformers, and BERT) offers a promising utilization of the proposed system in areas such as automatic captioning, recommendation systems, etc. BERT is used to generate text data embeddings, while Vision Transformer creates image data embeddings of the detected objects by YOLO. This work investigated the similarities between the text and image data using a Siamese network and measured the similarity using the cosine similarity metric. The proposed model applies an underlying methodology which assumes that in the shared embedding space, similar image text pairs are closer, and dissimilar ones are far apart. The accuracy for this model turned out to be 0.74 for the similarity check at a threshold of 0.5 and indicate the potential of the proposed system for multimodal data matching in several applications where good matching is required. This also shows betterment compared to other state-of-the-art works.