Vision transformer based convolutional neural network for breast cancer histopathological images classification

Breast cancer (BC) is a widespread and lethal cancer affecting women world- wide. Early diagnosis plays a pivotal role in ensuring survival, as late detection can result in a fatal outcome. Convolutional neural networks (CNNs) have made significant contributions to the task of medical imaging modalities and have dis- played promise in addressing this challenge. Recently, the success of the vision transformer (ViT) architecture has encouraged the use of the attention mecha- nism in computer-aided diagnosis (CAD) tasks. However, the ViT is known for its data-intensive nature and a substantial number of parameters and needs power- ful computer resources when training, which often leads to the same performance compared to CNNs. These challenges are particularly evident in tasks involving medical image datasets with complex images and limited data. This problem- atic situation led to the suggestion three of low-weight parameter systems based on convolution and attention techniques: vision transformer base model (ViT), compact convolution transformers (CCT), and lightweight mobile vision trans- formers (MVIT). These systems are developed by using the BreakHis dataset, which includes images captured at different magnification levels (40x, 100x, 200x, 400x), for both binary and multi classification of breast cancer subtypes. These low-weight hybrid ViT-CNN networks operate directly on input patches and convolution layers, to improve feature extraction and attention layers to train patches in all networks. This approach results in lower training time and fewer parameters while achieving accurate breast tumors classification. The proposed method is based on splitting the input image into patches and then focusing them on the area of cancerous lumps, providing a sequence of linear embedding of these patches as input. Second, we applied a convolution layer directly to the histopathology input patches, with the fewest possible modifications. Finally, we train patches in all transformer encoder layers to evaluate the performance of the classification of breast subtypes. The performance accuracies of our suggested models are 98.64% for VIT, 96.99% for CCT and 97.52% for MVIT. Moreover, the proposed models were compared with state-of-the-art models using the same dataset. Our study demonstrates how convolution and attention mechanisms can minimize computational training resources and decision time, to develop high- performing computer-aided analyses for breast cancer diagnosis. The source codes are accessible at https://github.com/abimouloud/ViT-CNN.