A hybrid DeiT-CNN model with spatial attention gating for invasive breast cancer detection in H&E-stained images

If detected in the last stages, breast cancer can cause mortality in the worldwide female population. Early detection is key to cancer survival and a good quality of life for almost all cancer types. Accurate differentiation between invasive and non-invasive lesions in hematoxylin and eosin (H&E)-stained histopathological slides is crucial for accurate diagnosis and effective treatment planning. In this paper, we present a hybrid framework that integrates a convolutional neural network (CNN) and a Vision Transformer (ViT) module. Using a combination of an institutional dataset collected from a hospital in Pune, India, and the subset of publicly available Breast Carcinoma Subtyping (BRACS) dataset, we developed a balanced H&E-stained patch-level dataset with strict slide-level splitting. We evaluated five CNN baseline models (ResNet50, MobileNet V3, Inception V3, Xception, and EfficientNet B0) and vanilla vision transformers (ViT-B16 and Deit-S) on the combined dataset. The performance of all baseline CNNs and ViTs was compared with the performance of the proposed DeiT+CNN approach. Inception V3, which provides spatial gating, was combined with the DeiT model. This resulted in the best test accuracy of 90.49% with an area under the receiver operating characteristic of 95.63%. The independent and completely unseen test set consisted of invasive and noninvasive patches from the publicly available Warwick dataset. This model yielded an accuracy of 87.26% on the Warwick test dataset, with an area under the receiver operating characteristic of 94.13%. The results demonstrate that the proposed hybrid architecture outperforms conventional CNNs and ViT architectures, highlighting its potential for robust and efficient histopathological diagnosis.