A Study on DL for Pulmonary Embolism Prediction Harnessing Multimodal Data

Pulmonary Embolism (PE) is a critical medical condition with high mortality, demanding prompt diagnosis. Current Deep Learning (DL) approaches for detection rely on a single modality (image), resulting in suboptimal predictions. Fusing image data with relevant clinical data enhances the prediction accuracy. Among the existing fusion techniques viz. early, late, and joint fusion, early fusion is a simple strategy wherein the intermodality correlations are utilized by fusing the features at the input stage itself. Attention-based fusion has been used to enhance the model performance by focusing on the most relevant modalities for PE detection. Hence this work aims to study the performance of LSTM and GRU and their variants like Bi-LSTM and Bi-GRU in PE detection by early fusing CTPA and EMR data. The effect of adding attention to LSTM (AttnLSTM) and GRU (AttnGRU) has also been studied. Using DL-based multimodal early fusion features from Computed Tomography of Pulmonary Angiography (CTPA) images and six Electronic Medical Record (EMR) modalities are concatenated at the input level. The performance of the six DL models using the RadFusion dataset, a manually labeled and processed dataset from Stanford University Medical Centre has been studied. The result analysis reveals that the AttnLSTM outperforms the other models with an accuracy of 86%.