Breast cancer is one of the common cancers worldwide, and early identification plays a crucial role in its subsequent diagnosis and treatment. However, centralized learning that collects data from distributed institutions may lead to medical privacy breaches, and whether the diagnostic results of the model can be trusted by doctors remains a concern. Therefore, we propose a Federated Learning Adaptive Knowledge Distillation Aggregation (FedAKDA) for breast cancer assisted diagnosis. First, Gaussian Blur Histogram Equalization (GBHE) is used for image preprocessing to address the issue of indistinct tumor regions in mammograms. The FedAKDA method consists of two modules: Adaptive Knowledge Distillation (AKD) and Adaptive Local Aggregation (ALA). These modules are designed to protect patient privacy, compress the model to reduce training time, and maintain the personalization of each client model. Finally, Gradient-weighted Class Activation Mapping (Grad-CAM) is applied to visualize breast cancer lesion regions, enhancing the model’s interpretability and aiding doctors in making better judgments. Experimental results show that compared to other methods, the proposed approach achieves superior diagnostic performance for breast cancer.