<p>Medical imaging has been an essential part of the prior detection and classification of tumors formed in the various parts of the human body. It is known that medical images are of low resolution which hinders the classification or identification of tumors. Usually, tumors will vary widely in their characteristics, behavior, initial position of tumors and prognosis, requiring accurate and timely identification to initiate optimal patient treatment. Till date, breast cancer continues to be the most common cancer diagnosed in women globally, and brain tumors pose significant prognostic difficulties because of their complicated nature and quick progression. A significant amount of research focuses on classifying tumors of specific organs only, but less research contributions in framing a generic framework that can identify and classify different tumors. Hence, in this paper, a generic customized multi-modal approach is devised to identify and classify Breast and Tumors in a single framework, which comprises ResNet-50, VGG-16 and EfficientNet-B0 Deep Learning (DL) architectures. The experimentation work is performed on the MIAS and Br35H datasets for the classification of breast and brain tumors, respectively. To mitigate overfitting and validate robustness, early stopping was employed. Class imbalance in the datasets was addressed through data augmentation. A uniform Hyperparameter setting is applied to all three Deep Learning architectures. EfficientNet-B0 demonstrated superior performance when trained using the Stochastic Gradient Descent (SGD) optimizer along with a learning rate of 0.01, achieving 98.61% accuracy on the MIAS dataset and 98.33% accuracy on the Br35H dataset. While the study focuses on two modalities (digital mammograms and MRI), its adaptability to other tumor types remains a scope for future work.</p>