Two-Stage Medical Image-Text Transfer with Supervised Contrastive Learning

Cross-modal translation enables automatic information transformation across different modalities, including images, text, and speech, enabling various applications in the medical domain, such as medical image description and Medical Q&A. Some previous research on cross-modal transfer uses contrastive learning to match paired image and text features without labels while separating others to realize tasks like classification and generation. However, due to label ignorance or absence, previous approaches overlook semantic similarity or dissimilarity between unpaired data from different modalities. Additionally, prior cross-modal models employ all-in-one training methods that cannot decouple each block to reduce the model size during each training iteration. In this paper, we propose a two-stage model for bi-directional translation using contrastive learning as an application for adjunctive medical diagnosis, which comprises two encoders for image and text modalities along with two projections. In the first stage, two encoders are trained using inner-modal contrastive learning with labels to extract features that can identify the same/different classes as explicitly as possible. In the second stage, supervised learning is introduced between the features output by the two encoders to train the two projections. This model is applied to a medical dataset that consists of paired medical images and reports with labels, enabling automatic diagnosis generation and cross-image-text retrieval.