IFNet: An Image-Enhanced Cross-Modal Fusion Network for Radiology Report Generation

Radiology Report Generation (RRG) tasks aim to automatically generate descriptive textual reports for medical images through computer-assisted technologies, which can alleviate the workload of radiologists, reduce the probability of misdiagnoses, and mitigate the strain on medical resources. However, previous studies explore rarely improving low-quality images in datasets, integrating cross-modal information, and optimizing network latency. To address its existing challenges, we propose an Image-enhanced cross-modal Fusion Network (IFNet) for the automated generation of radiology reports. IFNet comprises three core modules. First is an image enhancement module for augmenting X-ray images’ fine-grained normal and abnormal structural representations, increasing the probability of successful detection. Second is a cross-modal fusion network capable of capturing the interactive relations of cross-modal features comprehensively and efficiently. Third is a Transformer report generation module with linear time complexity, aimed at efficiently producing radiology reports with reduced network latency and operability on resource-constrained devices. Experiments on the public dataset IU-Xray demonstrate significant achievements of IFNet, surpassing the performance of the current state-of-the-art methods. The code is available at https://github.com/Hood0602/IFNet .