Distribution shifts are common in medical imaging datasets due to variations in scanner types, imaging protocols, and patient populations. Such shifts between source and target datasets/domains can significantly degrade the performance of deep learning models trained solely on source data. There is a strong need on developing methods that can effectively account for data shifts and heterogeneity for data harmonization. In this study, we propose Bias-Resilient Feature Learning (BRFL), a supervised domain adaptation approach designed to address this challenge. BRFL adopts a “trust but verify” strategy: the model is first pretrained on source data, then identifies potentially domain-intrinsic features by evaluating feature-label correlations on the target dataset, and is subsequently fine-tuned by swapping those dataset-dependent features. We evaluate BRFL on breast cancer diagnosis using two real-world mammography datasets. Extensive experiments show that BRFL outperforms conventional fine-tuning and dataset-merging baselines, achieving superior performance on target datasets with distribution shifts. Code is available at https://github.com/usernameoliver/BRFL .