Breast cancer (BC) remains a significant global health concern, necessitating accurate and efficient diagnostic approaches. In this study, we propose a comprehensive framework that integrates feature extraction, selection, and classification using Support Vector Machines (SVM) along with hyperparameter optimization. Additionally, we employ multi-level Vision Transformers (ViTs) for patch classification, aiming to capture both local and global information from histopathological slides. Moreover, we introduce a Fusion Probability Network (FPN) to combine the outputs of SVM and ViTs. Through this multi-faceted approach, we aim to improve diagnostic performance and contribute to more effective BC diagnosis. Sensitivity analyses and ablation studies across various sample sizes confirm the framework’s effectiveness. Results show high accuracy (up to 96.50%), precision (up to 93.33%), recall (up to 93%), specificity (up to 97.67%), F1 score (up to 92.99%), and balanced accuracy (up to 95.33%). Ablation studies highlight the significance of the feature extraction pipeline in enhancing the framework’s effectiveness and robustness, as well as its adaptability to diverse patch morphologies. Overall, our study offers promising avenues for improving BC grading, with potential implications for enhancing clinical decision-making and patient outcomes.