The most common disease to be diagnosed is breast cancer, which is one of the top killers of women. In the United States, 1 in 8 females (or roughly 13%) may eventually have invasive breast cancer. Early detection of this fatal disease is now feasible thanks to advancements in radiographic imaging, including computed tomography (CT), magnetic resonance imaging (MRI), 3D mammograms, and histopathology imaging (HI). Machine learning (ML) techniques and computer vision systems have come a long way in the last ten years. Among them, ML’s deep learning (DL) branch has shown remarkable achievements in several industries, most notably biomedicine. Large-scale data management and automatic feature extraction are strengths of DL techniques. Using radiographic and histological pictures, the potential of DL models has been efficiently applied and evaluated in the diagnosis and prognosis of breast cancer. The purpose of this study is to critically review the literature on the identification and categorization of breast cancer utilizing a range of imaging modalities, such as mammography, histology, ultrasound, PET/CT, MRI, and thermography. To detect and classify breast cancer, we thoroughly analyze earlier research using deep learning, machine learning, and reinforcement learning. To aid in future studies, we also examine publically available datasets for various imaging modalities.