<p>Breast cancer has emerged as the most significant health issue for women, making it the leading contributor to mortality worldwide. Hence, the early prediction of breast cancer plays a prominent place in improving the patient’s life against the intolerable lifestyles. Mammography procedures are predominantly used to detect early-stage cancer cells but still suffer from high false rates, discomfort in handling the patients, and radiation side effects. With the recent advancement of Artificial Intelligence (AI) techniques in the medical field, early recognition methods of breast cancer have reached new heights, offering renewed hope to patients. Nevertheless, these intelligent methods are challenged by the poor performance and interoperability issues. In order to address these concerns, this study suggests a Hybrid Pyramidal Swin- Linformer Networks, in which it leverages Pyramidal Pooling-based attention mechanisms to enhance the performance of classification. The Proposed framework further utilizes a hybrid CLAHE and K-means SMOTE algorithm to overcome overfitting and solve the class imbalance. In addition, poly-loss entropy-based Feed-Forward Neural Networks are applied to classify images accurately. To make the results interpretable, the Explainable AI techniques are utilized, especially GRAD-CAM, to visualize and assess the impact of the essential features on the classification performance. The effectiveness of the model was verified using the benchmark mammogram datasets such as MIAS and DDSM, with the performance measures like precision, accuracy, recall, AUC-ROC, and F1-score. Comparison with the state-of-the-art techniques demonstrates that the suggested framework is superior in terms of accuracy (1.00), precision (0.998), recall (0.998), and F1-score (1.00). The findings of the research provide valuable solutions to the current issues in AI integration for advancing breast cancer recognition, which could improve patient health under existing healthcare restrictions.</p>