Integrating multi-modal insights with transfer learning for detecting metastatic breast cancer (MBC-stage IV) prognostics

Accurate breast cancer prognosis prediction is crucial for efficient treatment planning and improving patient outcomes. While significant progress has been made in treating primary breast cancer, the development of robust predictive models remains a critical challenge. This study introduces multi-modal prognostic approach for enhanced Metastatic Breast Cancer prediction. Recognizing the limitations of relying solely on uni-modal data, our approach leverages the power of multi-modal datasets. The images are taken from breast ultrasound scans dataset and mammograms are resized into 224 × 224, and finally, data pre-processing steps are used. This study presents a two-stage model: first, a convolutional neural network is used to extract salient features from the multi-modal data. These extracted features are further concatenated for feature-fusion and served as an input for a dual transfer learning hybrid model in the second stage by combining EfficientNetB4 and InceptionV3, thus enabling robust and accurate prognosis prediction. The outputs are flattened and concatenated followed by implementation of Deep_dense layers to learn the weights of the combined models. The proposed hybrid approach effectively categorizes medical images for the prognostics of breast-cancer. Evaluations demonstrate the superior performance of our proposed model, achieving an accuracy of 99.12%, precision of 95%, recall of 93%, and F1 score of 98.12%.