Improving the Efficiency of Multimodal Approach for Chest X-Ray

Medical data analysis with limited data has been a challenging problem in machine learning. Privacy regulations limit the sharing and use of patient data, making it challenging to access and aggregate large datasets for research purposes. To address this challenge, we propose a deep learning-based Multimodal network that incorporates data from multiple data sources: images, and text in the learning process. When faced with limited data of one type, incorporating data from multiple sources can enhance the model’s performance and generalization. Our approach integrates the text sub-model and image sub-model with deep network of Dense 512 using the transfer learning process. Specifically, the goal is to explore the impact of hyperparameter adjustments on the performance of the model with different learning rates. Evaluation of our proposed approach on the Indiana University Chest X-ray dataset that consists of textual (radiological reports) and visual (X-ray images) show that our approach significantly outperforms several baseline methods in classification accuracy.