Deep adaptive fusion with cross-modality feature transition and modality quaternion learning for medical image fusion

In today's rapidly advancing medical landscape, the integration of information from multiple imaging modalities, known as medical fusion, stands at the forefront of diagnostic innovation. This approach combines the strengths of diverse techniques such as magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), and single-photon emission computed tomography (SPECT) to offer a more comprehensive view of a patient's condition. Issues such as data heterogeneity, where varied resolutions and contrasts must be harmonized, complicate the seamless integration of imaging data. The complexity of interpreting fused images demands specialized training for clinicians and raises concerns about potential diagnostic errors. This work presents the deep adaptive fusion (Deep-AF) model for image fusion in multimodal biomedical scans includes MRI, CT, PET, and SPECT. This Deep-AF model integrates convolutional neural network (CNN)-based decision maps, deep sparse coding, cross-modality feature transition, and fusion techniques. Three pre-processing steps, including intensity normalization, noise reduction, and spatial registration, are initially applied to enhance alignment and quality in fused images. Non-subsampled contourlet thresholding (NSCTT) is employed to address challenges related to intensity, resolution, and contrast differences among modalities, facilitating multi-scale and multidirectional representation. Despite challenges in spatial alignment, interpretation across modalities, and model generalization, the proposed gradient-weighted class activation mapping with CNN (GradCAM-CNN) enhances interpretability by visualizing crucial regions for CNN predictions. Deep sparse coding fusion (DSCF) overcomes challenges through the adaptive learning of complex features, capturing high-level features while enforcing sparsity. The cross-modality feature transition mechanism (CMFTM) addresses variations in modality characteristics. The attention weighted averaging network (AtWANet) addresses challenges in multimodal feature fusion by dynamically assigning weights based on relevance, providing a flexible approach despite misalignment and scale variations. AtWANet's model training optimizes the fusion process by dynamically assigning attention weights to each modality, ensuring effective integration of varied representations. Simulation results obtains that the proposed Deep-AF model obtains robust fusion results in terms of statistical and accuracy metrics.