Recurrent Mutual Attention Network with Transformer and LSTM Integration for Radiology Report Generation
摘要
Generating radiology reports by analyzing medical imaging data is becoming increasingly important not only in the research field but also in clinical practice. The automatic generation of accurate and coherent reports is of paramount importance to enhancing the efficiency and quality of medical diagnosis. Due to the rapid development of deep learning, automatic radiology report generation has evolved from simple feature transformation to cross-modal interactions between images and text. However, the correspondence between image regions and words remains unclear. Therefore, this paper proposes a recurrent mutual attention network that queries image features through text and then queries corresponding text features through image features. These features are then input into the report generation module. To enrich the semantic information, the long short-term memory (LSTM) is connected with the Transformer through gate control, enabling the automatic generation of radiology reports. The experimental results show that, compared with the other published methods, our proposed method has significantly improved the diagnostic report generation in quality, accuracy, and natural language fluency. Although our method has only been tested in radiology, it has shown potential usability in a wider range of different types of medical imaging analysis, due to its design principles and versatility.