Prognosis and treatment choices are largely based on the accurate staging of breast cancer. Traditional deep learning approaches often find it hard to capture the detailed and spatial features seen in histopathological images, particularly when magnification changes. In addition, most WSI staging models are computationally complex and hard to interpret, making them tough to use in real-world clinical situations that require fast and early diagnosis. In response to these limitations, we present HMST-Lite, a lightweight transformer-based architecture that makes use of multi-scale histopathological data for early breast cancer detection. Compare to the other regular models, HMST-Lite takes advantage of visual tokens at two magnification levels (10× and 20×) to recognize fine details from tissue structure efficiently. This multi-scale mechanism captures the context within each scale is used, then the fusion layer combines the multi-scale features. The model’s performance is measured on a specially selected part of the TCGA-BRCA cohort dataset, after preprocessing to highlight tissue regions important for early detection. The model’s performance is evaluated with Accuracy, Macro F1-Score, and QWK to guarantee reliability in class balance and simulated inter-rater agreement. The HMST-Lite framework provides the basic structure for the future HMST model, which will include cross-cohort fusion and prognostic learning.