Current methods for breast cancer diagnosis often face limitations in integrating prior knowledge, fusing multimodal data, and providing interpretability. To this end, we propose a novel knowledge-augmented multimodal learning framework. Based on a clinically guided breast cancer knowledge graph, our approach enhances patient clinical representations using Graph Attention Networks (GAT) and extracts pathological image features through weak supervision learning. A bidirectional cross-attention fusion mechanism enables interactive alignment and integration of multimodal heterogeneous information in the feature space. Extensive experiments on a real-world PathologicalEMR dataset demonstrate that our method achieves outstanding performance, with an AUC of 0.9963 in distinguishing benign from malignant cases. Further ablation studies validate the contributions of various multimodal feature combinations and knowledge enhancement strategies, underscoring the framework's effectiveness and interpretability. The proposed solution significantly advances the capability and trustworthiness of multi-source information integration for complex medical diagnostics, offering a promising tool for clinical decision support.