Breast ultrasound (BUS) offers a low-cost, radiation-free imaging alternative for breast cancer diagnostics, particularly suitable for point-of-care use. Despite promising results from deep learning (DL) models for BUS lesion classification, most models show significant performance drops on external datasets, suggesting overfitting to dataset-specific features. This lack of generalizability is concerning, especially given disparities in breast cancer outcomes across demographic groups and the diversity of ultrasound acquisition conditions. In this study, we investigate the impact of training dataset diversity on the robustness of DL models for BUS lesion classification. We compare three model architectures: ResNet50 (CNN), MViTv2 (Vision Transformer), and MambaOut (Vision Mamba), using 8403 B-mode BUS images from ten publicly available datasets originating from seven countries. Models were evaluated under three scenarios: single-dataset training, leave-one-dataset-out (LODO), and limited-data all-source training. Our results show that performance strongly depends on training set composition, with certain datasets consistently yielding better model performance, and substantial variability in cross-dataset generalization. This study provides new insights into the design of fair and generalizable DL systems for breast cancer diagnostics.