Multimodal Fusion of Imaging and Multi-omics Data for Enhanced Breast Cancer Detection
摘要
Breast cancer, a complex adaptive system of interacting genetic, epigenetic, and environmental factors, exhibits profound radiologic and molecular heterogeneity that challenges single-modality AI models. We present a biologically structured deep learning framework that unifies eight complementary modalities, including MRI, dual-resolution histopathology (20 × /40 ×), and five omics streams (RNA-Seq, CNV, DNA methylation, miRNA, and proteomics) to uncover cross-scale determinants of tumor behavior within the TCGA-BRCA cohort. By integrating macro-scale anatomy, micro-scale cellular architecture, and molecular regulation, the framework provides a holistic biological perspective linking imaging phenotypes to underlying gene activity and pathway dysregulation. Modality-specific encoders (ResNet/ViT for imaging, 1D-CNN/BiLSTM for omics) generate biologically anchored prototypes through supervised contrastive learning and graph-based compression. A memory-efficient FlashAttention Transformer performs token-level cross-attention between imaging and omics representations, revealing how spatial and molecular patterns jointly shape cancer phenotypes. On the 60-patient training/validation set, the framework achieved 93.19% accuracy and an AUC of 0.9208; on the 24-patient held-out test set, 92% accuracy and an AUC of 0.912, with 0% false positives under cross-patient modality-swap tests. SHAP analyses reveal consistent alignment between MRI-derived tumor-core necrosis patterns and BRCA1/2-related molecular deficiencies, demonstrating that the model learns biologically interpretable mechanisms rather than functioning as a black-box classifier. With sub-second inference and biologically grounded attention maps, the framework establishes a path toward interpretable, real-time, multimodal precision oncology.