Metastasis is the primary cause of death from breast cancer (BC), which is still a major public health problem. Visual indicators and morphological features of stained membrane areas are used to score immunohistochemistry (IHC) slides for breast cancer. Recent years have seen a rise in the use of histology whole slide images (WSIs) in digital pathology algorithms used for computer-assisted evaluations. Manual evaluation of microscopy pictures stained with human epidermal growth factor receptor 2 (HER2) is difficult, time-consuming, and prone to errors. The large, nonhomogeneous slides, overlapping sections, and varying staining methods are to blame for this. The challenging aspects of the photos, such as the atypical cell structure and tissue coloration, can only be captured by classifying HER2 images based on the selection of basic criteria. The first step of this study is to use Swin Transformer’s (ST) robust feature learning capabilities to extract features from input data. Golden Jackal Optimization Algorithm (GJOA) is used to fine-tune the ST parameters, which enhances the classification accuracy. Incorporating a backbone network with modules to produce feature maps for different target sizes, the study used the YOLOv4 algorithm for breast cancer diagnosis. The proposed model outperforms the current state-of-the-art approaches, according to results from the HER2GAN and HER2SC datasets. On the HER2SC dataset, for example, it improved accuracy, precision, recall, besides F1-score by approximately 8–10%.