MagNET: Modality-Agnostic Network for Brain Tumor Segmentation and Characterization with Missing Modalities

Multiple modalities provide complementary information in medical image segmentation tasks. However, in practice, not all modalities are available during inference. Missing modalities may affect the performance of segmentation and other downstream tasks like genomic biomarker prediction. Previous approaches either attempt a naive fusion of multi-modal features or synthesize missing modalities in the image or feature space. We propose an end-to-end modality-agnostic segmentation network (MagNET) to handle heterogeneous modality combinations, which is also utilized for radiogenomics classification. An attention-based fusion module is designed to generate a modality-agnostic tumor-aware representation. We design an adversarial training strategy to improve the quality of the representation. A missing-modality detector is used as a discriminator to push the encoded feature representation to mimic a full-modality setting. In addition, we introduce a loss function to maximize inter-modal correlations; this helps generate the modality-agnostic representation. MagNET significantly outperforms state-of-the-art segmentation and methylation status prediction methods under missing modality scenarios, as demonstrated on brain tumor datasets.