MSD-HAM-Net: A Multi-modality Fusion Network of PET/CT Images for the Prognosis of DLBCL Patients

18F-FDG PET/CT images have been proven promising for the prognosis of Diffuse Large B-cell Lymphoma (DLBCL) patients. However, the implicit drawbacks of images constrain their wide applications. In this paper, we propose a fusion solution which complements PET and CT images using a convolutional neural network (CNN), called MSD-HAM-Net. In this solution, non-delineated CT images are aligned to the corresponding delineated PET images during the registration procedure to synchronize PET and CT images, and lessen the manual delineations on CT images. The aligned PET and CT images are fed into MSD-HAM-Net which leverages multi-scale decomposition (MSD) and hybrid attention mechanism (HAM) for feature-level image fusion. After that the radiomic features are extracted from the fused images, and feature selection is performed to achieve the prognosis of DLBCL patients. In addition, to avoid the problem of poor generalization of traditional single machine learning model, we introduce the automated machine learning (AutoML) technique to predict the prognosis. Analysis results demonstrate that our solution can extract richer information from the fused images compared to the methods based on single modality images, i.e. PET/CT, which might be helpful for the prognosis of DLBCL patients.