Contrast-enhanced mammography (CEM) offers improved breast cancer diagnosis by enhancing vascular contrast uptake. However, the development of reliable deep learning-based computer-aided detection (CAD) systems for CEM is hindered by limited data availability. This paper introduces ELK (Enhanced Learning through cross-modal Knowledge transfer), a deep learning pipeline designed to adapt large pre-trained models into a target limited data-volume population by leveraging synthetic data augmentation. Specifically, we adapt a detection model pretrained on digital breast tomosynthesis (DBT) and digital mammography data into a target CEM population using diffusion models to generate high-resolution, realistic synthetic lesions, preserving the visual integrity of CEM images. To assess the efficacy of our synthetic lesions, we compare the detection performance of a pretrained Faster R-CNN detector fine-tuned using only real images, synthetic images, and a combination of both. Our approach improves mean sensitivity by 4% on a test sample from the same population and by 7% on a newly collected out-of-domain CEM dataset. Our code and synthetic datasets are available at https://github.com/Likalto4/CEM-Detect .