Contrastive learning techniques have achieved significant success and have been widely applied in both general and medical domains. However, there is a data difference between the general domain and the medical domain about negative mentions, which almost never appear in general domain but almost always in medical domain. We find that most existing medical contrastive learning methods do not effectively utilize or even overlook the numerous negative mentions present in the data during training, resulting in deficient multimodal feature alignment capabilities. To address this issue, we propose the Visual Entailment Based Contrastive Learning (VECL) method. By introducing a ternary visual entailment contrast relationship of entailment, neutral, and contradiction, our method effectively utilizes both positive and negative mentions for modeling fine-grained sample relationships, enhancing the model’s multimodal feature alignment capabilities. The experiment results show that we achieves SOTA performance on classification, grounding and report generation tasks. Resources are maintained at https://github.com/WVeLong/VECL .