Incomplete Multimodal Learning with Modality-Aware Feature Interaction for Brain Tumor Segmentation

Brain tumor segmentation in multimodal medical imaging poses challenges due to tumor heterogeneity and variability in imaging modalities. However, in clinical practice, the frequent absence of one or more modalities often renders multimodal glioma segmentation models either inapplicable or with poor segmentation accuracy. To address this challenge, we propose an Incomplete Multimodal Learning framework with Modality-Aware Feature Interaction (IML-MAFI) for brain tumor segmentation. Specifically, we devise a modality-aware feature interaction (MAFI) mechanism incorporating a modality missing status code to dynamically facilitate feature interaction across modalities in diverse modality absence scenarios. MAFI is a straightforward yet potent module, leveraging graph structure and attention mechanisms to acquire and interact with complementary features among graph nodes. Concurrently, the novel modality missing status code, which signals the presence or absence of each modality, steers MAFI towards learning adaptive complementary information among nodes in various missing modality scenarios. By integrating MAFI with the modality missing status code into the encoder stage of a U-shaped architecture, we have devised a unified yet stable convolutional network to effectively segment tumors under various potential missing modality scenarios. Experimental findings demonstrate that our approach achieves state-of-the-art performance in brain tumor segmentation task involving missing modalities.