Exploring Vision Language Pretraining with Knowledge Enhancement via Large Language Model

The integration of Vision-Language Pretraining (VLP) models in the medical field represents a significant advancement in the development of AI-driven diagnostic tools. These models, which learn to understand and generate descriptions of visual content, have shown great promise in enhancing the interpretability and accuracy of medical image analysis. However, the application of VLP models in healthcare poses unique challenges, including the scarcity of labeled data and fine-grained nature of medical imaging. Our contributions include the development of a Medical Visual Language Pre-training (MVLP) model that leverages domain-specific knowledge to improve the alignment between medical images and radiology reports. By utilizing a triplet extraction method and encoding the medical entities with detailed descriptions by MedPALM 2, we simplify language complexity and exploit the rich domain knowledge learned in Large Language Model, and implicitly build relationships between medical entities in the language embedding space. Our model demonstrates significant improvements in disease classification tasks, achieving competitive Area Under the Curve scores on benchmark datasets such as RSNA Pneumonia and ChestX-ray14.