Adversarial data can lead to malfunction of deep learning applications. It is essential to develop deep learning models that are resilient to adversarial data while accurate on standard, clean data. In this study, we focus on building safe breast cancer diagnosis models against mammographic adversarial samples. We proposed a novel adversarially robust feature learning (ARFL) method to facilitate adversarial training using both standard data and adversarial data, where a feature correlation measure is incorporated as an objective function to encourage learning of robust features and restrain spurious features. To show the efficacy of ARFL for robust breast cancer diagnosis, we built and evaluated deep learning diagnosis models using two independent clinically collected breast imaging datasets, comprising a total of 9,548 mammogram images. We performed extensive experiments showing that the ARFL method outperformed several state-of-the-art methods. ARFL can serve as an effective method to enhance adversarial training, towards building safe breast cancer diagnosis against adversarial attacks in clinical settings. The code repository of this study is publicly available at GitHub: https://github.com/usernamesafeai/ARFL .