Breast cancer, the most prevalent cancer in women worldwide, brings out the need for early detection and accurate lesion segmentation. Deep learning models show promise in breast cancer classification and segmentation. However, the existing models alone are not able to effectively capture the spatial and temporal features present in the images that are crucial for segmentation and classification. This study introduces an innovative deep learning model that effectively integrates spatial attention and recurrent neural networks for enhanced breast cancer diagnosis. Spatial attention mechanisms, such as the Squeeze-and-Excitation block, are integrated into a Convolutional Neural Network (CNN) to identify crucial regions within breast medical images, thereby aiding segmentation. Bidirectional recurrent neural networks are combined with the CNN to extract temporal features for a comprehensive understanding of cancerous tissues within breast lesions for distinguishing benign from malignant tumors. An experimental evaluation on the standard CBIS-DDSM dataset demonstrates a substantial increase in lesion segmentation with mDSC (0.9919) and mIoU (0.854) and lesion classification with accuracy (95.93%) compared to the state-of-the-art deep learning methods.