Early detection is still important because breast cancer is already the leading cause of death for women worldwide. Traditional imaging modalities including mammography, ultrasound, magnetic resonance imaging (MRI) and histopathology, have been used for screening over many years, but are dependent on radiologist interpretation and tend to be highly variable in diagnostic quality. Recent successes in deep learning have demonstrated potential to complement traditional diagnostic methods with computer aided diagnostic (CAD) systems capable of more precise, automated and consistent analysis over multiple imaging modalities. In this paper, we review the state of the DL applications for diagnoses of breast cancer based on imaging modalities with emphasis on DL architectures, performance metrics, available datasets, and a comparison of techniques aimed at improving the accuracy of diagnoses. Ethical issues, impediments, and suggestions for future research in DL implementation in clinical settings are also stressed.