Correlate-MobileCap: A Framework for Diagnostic Report Generation Using Lesion and Disease Information
摘要
The integration of multimodal learning approaches has emerged as a promising strategy for enhancing the accuracy of medical report generation. In this paper, we propose the Correlate-MobileCap framework to improve diagnostic report generation for chest X-ray images by addressing the limitations of traditional models in capturing complex dependencies between local lesion features and their corresponding diagnostic terms. The framework incorporates attention mechanisms to facilitate effective interactions between suspected lesion regions and diagnostic information extracted from medical images. Two variants of the framework are introduced: Correlate-MobileCap-SAttn and Correlate-MobileCap-CAttn. The former employs a self-attention-based transformer encoder to model intricate interactions across diverse image regions and their associated disease terms, thereby enhancing contextual understanding. In contrast, the latter utilizes a cross-attention-based transformer encoder to align specific lesion regions with relevant diagnostic terms, thereby improving both interpretability and diagnostic accuracy. We evaluate the performance of the proposed methods using the MIMIC-CXR and Open-I datasets. The experimental results demonstrate that our methods outperform models that rely solely on entire images, producing more accurate and interpretable diagnostic reports.