Breast cancer, the most prevalent one among women, has emerged as a global socioeconomic concern. Detection in early stages can significantly improve overall well-being of patients and reduce mortality rate. In this work, a deep learning and meta-heuristics algorithm-guided breast cancer diagnosis from mammogram has been examined. The proposed method has three main stages. Firstly, three attention-assisted deep learning models, i.e., VGG16, VGG19, and MobileNetV2 have been applied to obtain deep intrinsic features from dataset. Next, the features are fused together using the horizontal concatenation technique. Subsequently, in order to identify the features that highly characterize breast cancer, gravitational search algorithm followed by a random forest classifier is applied. This fusion approach has revealed improved performance over standalone deep learning models. It yields 92.03% classification accuracy and 97.65% feature reduction on DDSM mammography dataset, which demonstrates its prowess over other methods examined in this work as well as some related state-of-the-art methods.