FgKF: Fine-Grained Knowledge Fusion for Radiology Report Generation

Radiology imaging examination is an important basis for disease diagnosis and treatment. Based on existing radiology images and reports, automated generation of image-to-report can effectively relieve pressure on physicians. The generation of radiology reports utilizes the terminology and expertise inherent to the field of radiology. The integration of this specialized knowledge into automated report generation not only enhances the precision of disease findings descriptions, but also significantly elevates the quality of the reports produced. In this paper, we propose a fine-grained knowledge fusion model for radiology report generation that reduces the gap between visual and textual features by fusing image features with fine-grained radiographic knowledge. Specifically, the image-text cross-modal retrieval model, CLIP, is utilized to retrieve report from the dataset that are similar to the current image. The feature representations of the image and the fine-grained knowledge which are extracted from the similar report, are aligned by an Entities-Enhanced Multi-Head Attention mechanism. Then the fused features are decoded by a Transformer decoder with a semantic information fusion module to generate the radiology report. Experimental results on IU X-Ray and MIMIC-CXR show that the fusion of fine-grained knowledge guides the model to produce higher quality radiology reports.