Breast cancer (BC) remains a significant global health challenge, impacting millions of lives annually. Traditional histopathological analysis, while essential, can be subjective and time-consuming, potentially leading to diagnostic inaccuracies. This study proposes a novel Computer-Aided Diagnosis (CAD) framework utilizing Vision Transformers (ViTs) for BC diagnosis from histopathology slides. ViTs excel in capturing global dependencies within images, offering enhanced diagnostic accuracy compared to conventional methods. The framework integrates ViTs with advanced decision-making techniques like 2-tier majority fusion and SHapley Additive exPlanations (SHAP) for improved interpretability. Experimental results on a dataset of post-neoadjuvant therapy breast cancer samples demonstrate the efficacy of the proposed approach, achieving high performance metrics and providing insights into model predictions. The proposed approach achieves state-of-the-art performance with an accuracy exceeding 97% surpassing existing methods both on the utilized dataset and an external benchmark, specifically the Breast Cancer Histopathological Database (BreakHis). Time complexity analysis suggests that the proposed framework offers computational efficiency, with the dominant factors influencing overall complexity being the number of patches, sequence length, and number of layers in the ViT model. This study contributes a robust methodology towards enhancing BC diagnostic precision and efficiency through cutting-edge AI technologies.