Reinforcement learning (RL)-based semantic segmentation and attention based backpropagation convolutional neural network (ABB-CNN) for breast cancer identification and classification using mammogram images

Breast cancer poses a threat to women’s health and contributes to an increase in mortality rates. Mammography has proven to be an effective tool for the early detection of breast cancer. However, it faces many challenges in early breast cancer detection due to poor image quality, traditional segmentation, and feature extraction. Therefore, this work addresses these issues and proposes an attention-based backpropagation convolutional neural network (ABB-CNN) to detect breast cancer from mammogram images more accurately. The proposed work includes image enhancement, reinforcement learning-based semantic segmentation (RLSS), and multiview feature extraction and classification. The image enhancement is performed by removing noise and artefacts through a hybrid filter (HF), image scaling through a pixel-based bilinear interpolation (PBI), and contrast enhancement through an election-based optimization (EO) algorithm. In addition, the RLSS introduces intelligent segmentation by utilizing a deep Q network (DQN) to segment the region of interest (ROI) strategically. Moreover, the proposed ABB-CNN facilitates multiview feature extraction from the segmented region to classify the mammograms into normal, malignant, and benign classes. The proposed framework is evaluated on the collected and the digital database for screening mammography (DDSM) datasets. The proposed framework provides better outcomes in terms of accuracy, sensitivity, specificity, precision, f-measure, false-negative rate (FNR) and area under the curve (AUC). This work achieved (99.20%, 99.35%), (99.56%, 99.66%), (98.96%, 98.99%), (99.05%, 99.12%), (0.44%, 0.34%), (99.31%, 99.39%) and (99.27%, 99.32%) of accuracy, sensitivity, specificity, precision, FNR, f-measure and AUC on (collected, DDSM datasets), respectively. This research addresses the prevalent challenges in breast cancer identification and offers a robust and highly accurate solution by integrating advanced deep-learning techniques. The evaluated results reveal the proposed framework’s potential in early breast cancer detection.