Multiclass classification of breast cancer histopathological images is crucial for clinical applications, and it is a challenging task due to the intricate similarities within classes, the extensive variability in appearance across high-resolution images, and the considerable inhomogeneity in color distribution. Despite the noteworthy advancements of convolutional neural networks (CNNs) in histopathological imaging, they often struggle to comprehensively capture the intricate details in histopathological images. On the other hand, vision transformers (ViTs) show promise in learning complex visual patterns but have limited capability in exploring local contextual information. In addition, they face challenges when applied to medical image analysis tasks due to high data requirements and computational costs. To address these challenges, this paper presents a novel cascaded deformable transformer network named CaDT-Net for effective multiclass breast cancer classification through histopathological images. Specifically, we propose a cascaded deformable transformer layer (CDTL), which is placed after a pre-trained convolutional ViT model (MaxViT) to enable modeling global-local feature interactions, allowing it to learn fine-grained feature representations. Further, the proposed CDTL offers the advantage of using deformable convolution, which enhances the model’s ability to adapt suitably to complex and diverse lesion patterns. Extensive experiments on a benchmark BreaKHis dataset and a comparative analysis with state-of-the-art methods exhibit the superior performance of CaDT-Net. Notably, it achieves an accuracy of 97.32%, 97.75%, 98.67%, and 97.25% for 40 \(\times \) , 100 \(\times \) , 200 \(\times \) , and 400 \(\times \) magnifications, respectively. Our code is available at: https://github.com/skb999/CaDT-Net .